我正在尝试在 k8s 上设置一个 Spark 集群。我已经按照这篇文章创建并设置了一个包含三个节点的集群:
https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/
之后，当我尝试在集群上部署 spark 时，它在 spark 提交设置中失败了。
我使用了这个命令:
~/opt/spark/spark-2.3.0-bin-hadoop2.7/bin/spark-submit \
--master k8s://https://206.189.126.172:6443 \
--deploy-mode cluster \
--name word-count \
--class org.apache.spark.examples.SparkPi \
--conf spark.executor.instances=5 \
--conf spark.kubernetes.container.image=docker.io/garfiny/spark:v2.3.0 \
—-conf spark.kubernetes.driver.pod.name=word-count \
local:///opt/spark/examples/jars/spark-examples_2.11-2.3.0.jar
它给了我这个错误:
Exception in thread "main" org.apache.spark.SparkException: The Kubernetes mode does not yet support referencing application dependencies in the local file system.
at org.apache.spark.deploy.k8s.submit.DriverConfigOrchestrator.getAllConfigurationSteps(DriverConfigOrchestrator.scala:122)
at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication$$anonfun$run$5.apply(KubernetesClientApplication.scala:229)
at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication$$anonfun$run$5.apply(KubernetesClientApplication.scala:227)
at org.apache.spark.util.Utils$.tryWithResource(Utils.scala:2585)
at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.run(KubernetesClientApplication.scala:227)
at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.start(KubernetesClientApplication.scala:192)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:879)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:197)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:227)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:136)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
2018-06-04 10:58:24 INFO ShutdownHookManager:54 - 关闭 Hook 被调用
2018-06-04 10:58:24 INFO ShutdownHookManager:54 - 删除目录/private/var/folders/lz/0bb8xlyd247cwc3kvh6pmrz00000gn/T/spark-3967f4ae-e8b3-20c-429dc
注意:我按照这篇文章在 k8s 上安装 spark。
https://spark.apache.org/docs/latest/running-on-kubernetes.html
最佳答案
错误消息来自 commit 5d7c4ba4d73a72f26d591108db3c20b4a6c84f3f 并包含您提到的页面:“Running Spark on Kubernetes ”以及您指出的提及:
// TODO(SPARK-23153): remove once submission client local dependencies are supported.
if (existSubmissionLocalFiles(sparkJars) || existSubmissionLocalFiles(sparkFiles)) {
throw new SparkException("The Kubernetes mode does not yet support referencing application " +
"dependencies in the local file system.")
}
这在 SPARK-18278 中有描述:
这与 kubernetes issue 34377 相关联
issue SPARK-22962 "Kubernetes app fails if local files are used" 提到:
同时，该错误消息是在 PR 20320 中引入的。
它包括评论:
这个问题现在应该已经解决了，OP garfiny 确认了 in the comments :
关于apache-spark - K8s 上的 Spark - 出现错误 : kube mode not support referencing app depenpendcies in local，我们在Stack Overflow上找到一个类似的问题：https://stackoverflow.com/questions/50637250/