spark-submit --packages com.databricks:spark-redshift_2.11:2.0.1 --jars /usr/share/aws/redshift/jdbc/RedshiftJDBC4.jar /home/hadoop/test.py
如何以Apache livy格式指定以上(pyspark)spark-submit命令？
我尝试了以下方法:
curl -X POST --data '{"file": "/home/hadoop/test.py", "conf":
{"com.databricks": "spark-redshift_2.11:2.0.1"}, \
"queue": "my_queue", "name": "Livy Example", "jars" :
"/usr/share/aws/redshift/jdbc/RedshiftJDBC4.jar"}', \
-H "Content-Type: application/json" localhost:8998/batches
Refered the following livy article spark livy rest api
还得到以下错误:
"Unexpected character ('“' (code 8220 / 0x201c)): was expecting double-quote to start field name\n at [Source: (org.eclipse.jetty.server.HttpInputOverHTTP); line: 1, column: 37]
最佳答案
您的命令错误，请使用以下示例构造命令。
spark-submit命令
./bin/spark-submit \
--class org.apache.spark.examples.SparkPi \
--jars a.jar,b.jar \
--pyFiles a.py,b.py \
--files foo.txt,bar.txt \
--archives foo.zip,bar.tar \
--master yarn \
--deploy-mode cluster \
--driver-memory 10G \
--driver-cores 1 \
--executor-memory 20G \
--executor-cores 3 \
--num-executors 50 \
--queue default \
--name test \
--proxy-user foo \
--conf spark.jars.packages=xxx \
/path/to/examples.jar \
1000
Livy REST JSON协议(protocol)
{
“className”: “org.apache.spark.examples.SparkPi”,
“jars”: [“a.jar”, “b.jar”],
“pyFiles”: [“a.py”, “b.py”],
“files”: [“foo.txt”, “bar.txt”],
“archives”: [“foo.zip”, “bar.tar”],
“driverMemory”: “10G”,
“driverCores”: 1,
“executorCores”: 3,
“executorMemory”: “20G”,
“numExecutors”: 50,
“queue”: “default”,
“name”: “test”,
“proxyUser”: “foo”,
“conf”: {“spark.jars.packages”: “xxx”},
“file”: “hdfs:///path/to/examples.jar”,
“args”: [1000],
}
-软件包。使用此命令时，将处理所有传递依赖项。
在Livy中，您需要转到解释器设置页面，然后在livy设置下添加新属性-
和值(value)
com.databricks:spark-redshift_2.11:2.0.1
重新启动解释器，然后重试查询。