RStudio EC2上的Sparklyr发生调用错误hadoopConfiguration独立群集

问题描述

所以我在EC2上有1个master/2从属独立集群.我从EC2运行rstudio并运行以下代码:

So I have a 1 master/2 slave standalone cluster on EC2. I am running rstudio from EC2 and after I run the following code:

library(aws.s3)
library(sparklyr)
library(tidyverse)
library(RCurl)

Sys.setenv("AWS_ACCESS_KEY_ID" =  "myaccesskeyid",
           "AWS_SECRET_ACCESS_KEY" = "myaccesskey",
           "SPARK_CONF_DIR" = "/home/rstudio/spark/spark-2.1.0-bin-hadoop2.7/bin/",
           "JAVA_HOME" = "/usr/lib/jvm/java-8-oracle" )

ctx <- spark_context(sc)
jsc <- invoke_static(sc, 
                    "org.apache.spark.api.java.JavaSparkContext", 
                    "fromSparkContext", ctx)

hconf <- jsc %>% invoke("hadoopConfiguration")

最后一行是我遇到错误的地方:

The last line is where I encounter an error:

Error in do.call(.f, args, envir = .env) : 
  'what' must be a function or character string

从我的研究中，我知道 invoke 是 sparklyr 处理Java对象的方式，我检查并确认我的Java已安装在主服务器/从属服务器和 JAVA_HOME中已设置.

From my research, I know that invoke is how sparklyr handles Java objects and I checked and confirmed my Java was installed in master/slaves and the JAVA_HOME was set.

推荐答案

调用 library(tidyverse)时，您会看到获取冲突信息，该信息将说明发生的情况:

When you call library(tidyverse) you see get conflicts info, which will explain what is going on:

 ── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
✖ dplyr::filter() masks stats::filter()
✖ purrr::invoke() masks sparklyr::invoke()
✖ dplyr::lag()    masks stats::lag()

您将看到 purrr :: invoke ，它具有错误消息中的确切签名:

As you see purrr::invoke, which has exactly the signature from the error message:

invoke(.f, .x = NULL, ..., .env = NULL)

阴影 sparklyr :: invoke .使用完全限定的名称

shades sparklyr::invoke. Using fully qualified name

jsc %>% sparklyr::invoke("hadoopConfiguration")

应该解决问题.

这篇关于RStudio EC2上的Sparklyr发生调用错误hadoopConfiguration独立群集的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！