使用并行的 parLapply:无法访问并行代码中的变量

问题描述

我最近得到了一台具有多个内核的计算机，并且正在学习使用并行计算.我对 lapply 相当精通，并被告知 parLapply 的工作原理非常相似.我没有正确操作它.似乎我必须明确地将所有内容都放在 parLapply 中才能使其工作(即要使用的函数、变量等).使用 lapply 它从父环境中读取，而 parLapply 似乎没有这样做.所以在我下面的例子中，我可以通过将所有信息放在 parLapply 中来使一切正常，但是如果我在用户定义的函数中使用它，我不能明确地将 text.var 放在里面parLapply.

I recently got a computer with several cores and am learning to use parallel computing. I'm fairly proficient with lapply and was told parLapply works very similarly. I'm not operating it correctly though. It seems I have to explicitly put everything inside the parLapply to make it work (that is functions to be use, variables etc.). With lapply it reads from the parent environment and parLapply does not seem to do this. So in my example below I could make everything work by placing all info inside parLapply but if I use this inside a user defined function I can't explicitly put text.var inside of parLapply.

library(parallel)
text.var <- rep("I like cake and ice cream so much!", 20)
ntv <- length(text.var)
gc.rate <- 10

pos <-  function(i) {
    paste(sapply(strsplit(tolower(i), " "), nchar), collapse=" | ")
}

lapply(seq_len(ntv), function(i) {
        x <- pos(text.var[i])
        if (i%%gc.rate==0) gc()
        return(x)
    }

)

#doesn't work
cl <- makeCluster(mc <- getOption("cl.cores", 4))
parLapply(cl, seq_len(ntv), function(i) {
        x <- pos(text.var[i])
        if (i%%gc.rate==0) gc()
        return(x)
    }

)

#does work but have to specify all the stuff inside parLapply
cl <- makeCluster(mc <- getOption("cl.cores", 4))
parLapply(cl, seq_len(ntv), function(i) {
        ######stuff I have to put inside parLapply##########
        text.var <- rep("I like cake and ice cream so much!", 20)
        ntv <- length(text.var)
        gc.rate <- 10
        pos <-  function(i) {
            paste(sapply(strsplit(tolower(i), " "), nchar), collapse=" | ")
        }
        ######stuff I have to put inside parLapply##########
        x <- pos(text.var[i])
        if (i%%gc.rate==0) gc()
        return(x)
    }
)

如何将 text.var、ntv、gc.rate 和 pos 传递给 parLapply 没有明确地把它们放进去?(我猜你以某种方式将它们作为列表传递)

How can I pass text.var, ntv, gc.rate, and pos to parLapply without explicitly putting them inside? (I'm guessing you pass them as a list somehow)

PS windows 7 机器所以我需要使用 parLapply 我想

PS windows 7 machine so I need to use parLapply I think

推荐答案

您需要将这些变量导出到集群中的其他 R 进程:

You need to export those variables to the other R processes in the cluster:

cl <- makeCluster(mc <- getOption("cl.cores", 4))
clusterExport(cl=cl, varlist=c("text.var", "ntv", "gc.rate", "pos"))

这篇关于使用并行的 parLapply:无法访问并行代码中的变量的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！