查看文档,我读到可用的通用参数列表为:
Generic options supported are
-conf <configuration file> specify an application configuration file
-D <property=value> use value for given property
-fs <local|namenode:port> specify a namenode
-jt <local|jobtracker:port> specify a job tracker
-files <comma separated list of files> specify comma separated files to be copied to the map reduce cluster
-libjars <comma separated list of jars> specify comma separated jar files to include in the classpath.
-archives <comma separated list of archives> specify comma separated archives to be unarchived on the compute machines.
我想在sqoop命令中指定以下两个属性:
mapred.job.queuename=batch
mapred.child.java.opts="\-Djava.security.egd=file:///dev/urandom"
所以我这样写了我的导入sqoop脚本:
sqoop import -Dmapred.job.queuename=batch \
mapred.child.java.opts="\-Djava.security.egd=file:///dev/urandom" \
--connect $connection \
--username $username\
--password $password \
--table $sourceTable \
--columns "$columns"\
--hive-import \
--hive-overwrite \
--hive-table $targetTable \
--compression-codec org.apache.hadoop.io.compress.SnappyCodec \
--hive-delims-replacement "SPECIAL" \
--null-string '\\N' \
--null-non-string '\\N' \
-m 1
但这没用...
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/hdp/2.6.4.0-91/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/2.6.4.0-91/accumulo/lib/slf4j-log4j12.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
18/03/07 08:00:04 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6.2.6.4.0-91
18/03/07 08:00:04 ERROR tool.BaseSqoopTool: Error parsing arguments for import:
18/03/07 08:00:04 ERROR tool.BaseSqoopTool: Unrecognized argument: mapred.child.java.opts=\-Djava.security.egd=file:///dev/urandom
18/03/07 08:00:04 ERROR tool.BaseSqoopTool: Unrecognized argument: --connect
我也考虑过要做以下事情
sqoop import \
-D mapred.job.queuename=batch \
-D mapred.child.java.opts="\-Djava.security.egd=file:///dev/urandom" \
--connect $connection \
--username $username \
--password $password \
--table $sourceTable \
--columns "$columns"\
--hive-import \
--hive-overwrite \
--hive-table $targetTable \
--compression-codec org.apache.hadoop.io.compress.SnappyCodec \
--hive-delims-replacement "SPECIAL" \
--null-string '\\N' \
--null-non-string '\\N' \
-m 1
但是恐怕只有两个-D通用参数之一会被解析。
哪种是正确的方法?
最佳答案
Java D标志在属性后不需要空格
sqoop import \
-Dmapred.child.java.opts="-Djava.security.egd=file:///dev/urandom" \
-Dmapred.job.queue.name=batch \
还修复了您的队列名称属性