我正在尝试在Spark上安装Google Cloud Storage。我已使用-libjars在hadoop类路径中添加了“gcs-connector-latest-hadoop2.jar”。我仍然遇到找不到GoogleCloudStorageFileSystem类错误。
$ hadoop fs -libjars /PATH/gcs-connector-hadoop2-latest.jar -ls /HDFS_PATH
20/02/05 05:41:33 WARN fs.FileSystem: Cannot load filesystem: java.util.ServiceConfigurationError: org.apache.hadoop.fs.FileSystem: Provider com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem could not be instantiated
20/02/05 05:41:33 WARN fs.FileSystem: java.lang.NoClassDefFoundError: com/google/cloud/hadoop/gcsio/GoogleCloudStorageFileSystem
20/02/05 05:41:33 WARN fs.FileSystem: java.lang.ClassNotFoundException: com.google.cloud.hadoop.gcsio.GoogleCloudStorageFileSystem
20/02/05 05:41:33 WARN fs.FileSystem: Cannot load filesystem: java.util.ServiceConfigurationError: org.apache.hadoop.fs.FileSystem: Provider com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem could not be instantiated
20/02/05 05:41:33 WARN fs.FileSystem: java.lang.NoClassDefFoundError: com/google/cloud/hadoop/gcsio/GoogleCloudStorageFileSystem
20/02/05 05:41:33 WARN fs.FileSystem: java.lang.ClassNotFoundException: com.google.cloud.hadoop.gcsio.GoogleCloudStorageFileSystem
我在这里想念什么吗？
最佳答案
如果您有权访问dataproc集群的主节点，则可以在此处添加gcs连接器和符号链接(symbolic link)/usr/lib/hadoop/lib
关于hadoop - 提供程序com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem无法实例化，我们在Stack Overflow上找到一个类似的问题：https://stackoverflow.com/questions/60069694/