提高去除每行重复值和R中的移位值的效率

问题描述

我有一个庞大的数据集(> 250万).一小部分看起来像这样(可复制代码)

I have a huge dataset ( > 2.5 Million). A small subset looks like this (code reproducible)

temp <- data.frame(list(col1 = c("424", "560", "557"), 
                        col2 = c("276", "427", "V46"), 
                        col3 = c("780", "V45", "584"), 
                        col4 = c("276", "V45", "995"), 
                        col5 = c("428", "799", "427")))

> temp
  col1 col2 col3 col4 col5
1  424  276  780  276  428
2  560  427  V45  V45  799
3  557  V46  584  995  427

我正在尝试使用此代码删除每行重复项，并将值左移

I am trying to remove duplicates per row, and shifting values left, using this code

library(plyr)
temp <- apply(temp,1,function(x) unique(unlist(x)))
temp <- ldply(temp, rbind)

> temp
      1   2   3   4    5
  1 424 276 780 428 <NA>
  2 560 427 V45 799 <NA>
  3 557 V46 584 995  427

我这样做很成功，但是当我将上面的代码扩展到我原来的庞大数据集时，我遇到了性能问题.因为我使用的是apply，所以代码需要花费很多时间来执行

I am successfull in doing this, however when I extend the above code to my original huge dataset, I am facing performance issues.because I am using apply, the code takes lot of time to execute

我可以改善这一点吗?

推荐答案

如果只有字符串，则应该使用矩阵而不是数据框.也许移调它也会很有用.

If you have only strings, you should really use a matrix rather than a data frame.Maybe transposing it would be useful too.

temp <- data.frame(list(col1 = c("424", "560", "557"), 
                        col2 = c("276", "427", "V46"), 
                        col3 = c("780", "V45", "584"), 
                        col4 = c("276", "V45", "995"), 
                        col5 = c("428", "799", "427")),
                   stringsAsFactors = FALSE)

p <- ncol(temp)

myf <- compiler::cmpfun(
  function(x) {
    un <- unique(x)
    d <- p - length(un)
    if (d > 0) {
      un <- c(un, rep(NA_character_, d))
    }
    un
  }
)

microbenchmark::microbenchmark(
  privefl = as.data.frame(t(apply(t(temp), 2, myf))),
  OP = plyr::ldply(apply(temp, 1, function(x) unique(unlist(x))), rbind)
)

小尺寸结果:

Unit: microseconds
    expr     min       lq      mean   median       uq       max neval
 privefl 278.775 301.7855  376.2803 320.8235 409.0580  1705.428   100
      OP 567.152 619.7950 1027.1277 658.2010 792.6225 29558.777   100

具有100,000个观察值(temp <- temp[sample(nrow(temp), size = 1e5, replace = TRUE), ]):

With 100,000 observations (temp <- temp[sample(nrow(temp), size = 1e5, replace = TRUE), ]):

Unit: milliseconds
    expr       min        lq      mean    median       uq      max neval
 privefl  975.1688  975.1688  988.2184  988.2184 1001.268 1001.268     2
      OP 9196.5199 9196.5199 9518.3922 9518.3922 9840.264 9840.264     2

这篇关于提高去除每行重复值和R中的移位值的效率的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！