我正在尝试在azkaban中进行 hive 作业
我能够上载配置单元作业，但是配置单元作业未正确执行。
ERROR [hive-demo] [Azkaban] Failed to build job executor for job hive-demoJob type 'hive' is unrecognized. Could not construct job[{hive.query.01: drop table words;, hive.query.03: describe words;, hive.query.02: create table words (freq int, word string) row format delimited fields terminated by ' ' stored as textfile;, working.dir: /home/hduser/Technology/azkaban/azkaban-executor/executions/44, azk.hive.action: execute.query, azkaban.job.attempt: 0, type: hive, hive.query.05: select * from words limit 10;, hive.query.04: load data local inpath "res/input" into table words;, user.to.proxy: azkaban, hive.query.06: select freq, count(1) as f2 from words group by freq sort by f2 desc limit 10;, parent = {azkaban.flow.flowid: hive-demo, azkaban.flow.execid: 44, azkaban.flow.start.timezone: America/Los_Angeles, azkaban.flow.start.hour: 22, azkaban.flow.start.second: 17, azkaban.flow.start.year: 2014, azkaban.flow.start.milliseconds: 111, azkaban.flow.start.minute: 54, azkaban.flow.start.timestamp: 2014-01-31T22:54:17.111-08:00, azkaban.flow.start.month: 01, azkaban.flow.projectversion: 2, azkaban.flow.projectid: 4, azkaban.flow.uuid: 29a46293-cc95- 4c66-b2b1-51014205ce3b, azkaban.flow.start.day: 31, parent = {}}}] of type[hive]. 2014/02/01 12:24:17.295 +0530 ERROR [hive-demo] [Azkaban] Failed to build job type, skipping this job 2014/02/01 12:24:17.295 +0530 ERROR [hive-demo] [Azkaban] Job run failed! 2014/02/01 12:24:17.295 +0530 INFO [hive-demo] [Azkaban] Finishing job hive-demo at 1391237657295 2014/02/01 12:24:17.382 +0530 INFO [FlowRunnerManager] [Azkaban] Cleaning recently finished 2014/02/01 12:24:17.383 +0530 INFO [FlowRunnerManager] [Azkaban] Cleaning execution 43 from recently finished flows list. 2014/02/01 12:24:17.408 +0530 INFO [hive-demo] [Azkaban] Job Finished hive-demo with status FAILED 2014/02/01 12:24:17.463 +0530 INFO [hive-demo] [Azkaban] Finishing up flow. Awaiting Termination 2014/02/01 12:24:17.463 +0530 INFO [hive- demo] [Azkaban] Setting flow status to Failed. 2014/02/01 12:24:17.464 +0530 INFO [hive- demo] [Azkaban] Flow is set to FAILED 2014/02/01 12:24:17.464 +0530 INFO [hive-demo] [Azkaban] Setting end time for flow 44 to 1391237657464 2014/02/01 12:24:17.552 +0530 INFO [FlowRunnerManager] [Azkaban] Flow 44 is finished. Adding it to recently finished flows list. 2014/02/01 12:26:17.383 +0530 INFO [FlowRunnerManager] [Azkaban] Cleaning recently finished 2014/02/01 12:26:17.384 +0530 INFO [FlowRunnerManager] [Azkaban] Cleaning execution 44 from recently finished flows list.
========================================================================
这是我的plugin.properties中的内容
job.class=azkaban.jobtype.hiveutils.azkaban.HiveViaAzkaban
hive.home=HIVE_HOME
hive.aux.jars.path=/usr/lib/hive/lib
env.HIVE_HOME=$HIVE_HOME
env.HIVE_AUX_JARS_PATH=/usr/lib/hive/lib
#hive.jvm.args=-Dhive.querylog.location=. -Dhive.exec.scratchdir=/tmp/hive- ${user.to.proxy} -Dhive.aux.jars.path=${hive.aux.jars.path}
jobtype.global.jvm.args=${hive.jvm.args}
这是我的private.properties
jobtype.classpath = $ HADOOP_HOME / conf，$ HADOOP_HOME / lib / *，$ HIVE_HOME / lib / *，$ HIVE_HOME / conf，/ usr / lib / hive / lib
jobtype.class = azkaban.jobtype.HadoopJavaJob
hive.aux.jars.path=/usr/lib/hive/lib
env.HADOOP_HOME=$HADOOP_HOME
env.HIVE_HOME=$HIVE_HOME
env.HIVE_AUX_JARS_PATH=/usr/lib/hive/lib
hive.home=$HIVE_HOME
jobtype.global.classpath=${hive.classpath.items}
hive.classpath.items=./*,${hive.home}/lib/*,${hive.home}/conf/,${hive.aux.jars.path}/*,${had oop.home}/*,${hadoop.home}/lib/*,${hadoop.home}/conf/
我仅在executor文件夹中添加了plugin文件夹。
为什么不检测作业类型。请帮忙。
最佳答案
很难说是怎么回事，因为您既不共享任务本身也不记录日志，但是尝试以下操作并让我们知道是否可行。创建具有以下内容的文件test.job:
type=hive
azk.hive.action=execute.query
hive.query=CREATE TABLE test(id INT, name STRING) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t';
user.to.proxy=hadoop
并按照“creating flow”部分中的说明进行提交。有关开箱即用的Azkaban(2.1)支持的不同类型的作业的更多详细信息，请参见job types/plugin documentation。
关于hadoop - Azkaban-上类失败，我们在Stack Overflow上找到一个类似的问题：https://stackoverflow.com/questions/21495230/