原始SQL:
select a2.ISSUE_CODE as ISSUE_CODE,
a2.FZQDM as FZQDM,
a2.FZQLB as FZQLB,
a2.FJJDM as FJJDM,
a3.FSETCODE as FSETCODE,
a3.FSETID as FSETID,
a2.SRSCD as SRSCD
from (select t1.FSCDM as ISSUE_CODE,--市场代码
t1.FZQDM as FZQDM,
(case when instr(t1.FZQLB, '非银行间') > 0
then '非银行间'
else '银行间'
end) as FZQLB,
t1.FJJDM as FJJDM,
t1.SRSCD as SRSCD
from (select
a1.FZQDM as FZQDM,
a1.FZQLB as FZQLB,
a1.FJJDM as FJJDM,
a1.SRSCD as SRSCD,
a1.FSCDM as FSCDM,
row_number() over(partition by a1.FZQDM,a1.SRSCD order by length(trim(a1.FSCDM)) desc,a1.FSCDM desc) sem
from TMP.CS_DWM_ISSU_SRC2STD_REL_H_05 a1
where a1.FJJDM is not null or a1.FJJDM = ' ') t1
where t1.sem=1
) a2
left join (select distinct concat('A',lpad(t.FSETCODE,3,0)) as FSETCODE, --基金代码
t.FSETID as FSETID, --套账号
t.SRSCD as SRSCD
from TMP.CS_DWM_ISSU_SRC2STD_REL_H_06 t
)a3
on a3.FSETCODE = a2.FJJDM and a2.SRSCD=a3.SRSCD
执行计划：
STAGE DEPENDENCIES:
Stage-1 is a root stage
Stage-5 depends on stages: Stage-1, Stage-3 , consists of Stage-6, Stage-2 /* Stage-5 依赖于流：Stage-1,Stage-3 ,由Stage-6,Stage-2组成*/
Stage-6 has a backup stage: Stage-2 /*Stage-6 有一个备份流：Stage-2 */
Stage-4 depends on stages: Stage-6 /*Stage-4 依赖于Stage-6*/
Stage-2
Stage-3 is a root stage
Stage-0 depends on stages: Stage-4, Stage-2 STAGE PLANS:
Stage: Stage-1
Map Reduce
Map Operator Tree:
TableScan
alias: a1
filterExpr: (fjjdm is not null or (fjjdm = ' ')) (type: boolean)
Statistics: Num rows: 13830 Data size: 580901 Basic stats: COMPLETE Column stats: NONE
Filter Operator
predicate: (fjjdm is not null or (fjjdm = ' ')) (type: boolean)
Statistics: Num rows: 13830 Data size: 580901 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: fzqdm (type: string), srscd (type: string), length(trim(fscdm)) (type: int), fscdm (type: string)
sort order: ++--
Map-reduce partition columns: fzqdm (type: string), srscd (type: string)
Statistics: Num rows: 13830 Data size: 580901 Basic stats: COMPLETE Column stats: NONE
value expressions: fzqlb (type: string), fjjdm (type: string)
Reduce Operator Tree:
Select Operator
expressions: KEY.reducesinkkey3 (type: string), KEY.reducesinkkey0 (type: string), VALUE._col0 (type: string), VALUE._col1 (type: string), KEY.reducesinkkey1 (type: string)
outputColumnNames: _col0, _col1, _col2, _col3, _col4
Statistics: Num rows: 13830 Data size: 580901 Basic stats: COMPLETE Column stats: NONE
PTF Operator
Statistics: Num rows: 13830 Data size: 580901 Basic stats: COMPLETE Column stats: NONE
Filter Operator
predicate: (_wcol0 = 1) (type: boolean)
Statistics: Num rows: 6915 Data size: 290450 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col0 (type: string), _col1 (type: string), CASE WHEN ((instr(_col2, '非银行间') > 0)) THEN ('非银行间') ELSE ('银行间') END (type: string), _col3 (type: string), _col4 (type: string)
outputColumnNames: _col0, _col1, _col2, _col3, _col4
Statistics: Num rows: 6915 Data size: 290450 Basic stats: COMPLETE Column stats: NONE
File Output Operator
compressed: false
table:
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe Stage: Stage-5
Conditional Operator /* 条件运算符 ,是不是不知道什么意思,见Debug信息*/ Stage: Stage-6
Map Reduce Local Work
Alias -> Map Local Tables:
$INTNAME1
Fetch Operator
limit: -1
Alias -> Map Local Operator Tree:
$INTNAME1
TableScan
HashTable Sink Operator
keys:
0 _col3 (type: string), _col4 (type: string)
1 _col0 (type: string), _col2 (type: string) Stage: Stage-4
Map Reduce
Map Operator Tree:
TableScan
Map Join Operator
condition map:
Left Outer Join0 to 1
keys:
0 _col3 (type: string), _col4 (type: string)
1 _col0 (type: string), _col2 (type: string)
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6
Statistics: Num rows: 7606 Data size: 319495 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col0 (type: string), _col1 (type: string), _col2 (type: string), _col3 (type: string), _col5 (type: string), _col6 (type: string), _col4 (type: string)
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6
Statistics: Num rows: 7606 Data size: 319495 Basic stats: COMPLETE Column stats: NONE
File Output Operator
compressed: true
Statistics: Num rows: 7606 Data size: 319495 Basic stats: COMPLETE Column stats: NONE
table:
input format: org.apache.hadoop.mapred.TextInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
Local Work:
Map Reduce Local Work Stage: Stage-2
Map Reduce
Map Operator Tree:
TableScan
Reduce Output Operator
key expressions: _col3 (type: string), _col4 (type: string)
sort order: ++
Map-reduce partition columns: _col3 (type: string), _col4 (type: string)
Statistics: Num rows: 6915 Data size: 290450 Basic stats: COMPLETE Column stats: NONE
value expressions: _col0 (type: string), _col1 (type: string), _col2 (type: string)
TableScan
Reduce Output Operator
key expressions: _col0 (type: string), _col2 (type: string)
sort order: ++
Map-reduce partition columns: _col0 (type: string), _col2 (type: string)
Statistics: Num rows: 318 Data size: 4648 Basic stats: COMPLETE Column stats: NONE
value expressions: _col1 (type: string)
Reduce Operator Tree:
Join Operator
condition map:
Left Outer Join0 to 1
keys:
0 _col3 (type: string), _col4 (type: string)
1 _col0 (type: string), _col2 (type: string)
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6
Statistics: Num rows: 7606 Data size: 319495 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col0 (type: string), _col1 (type: string), _col2 (type: string), _col3 (type: string), _col5 (type: string), _col6 (type: string), _col4 (type: string)
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6
Statistics: Num rows: 7606 Data size: 319495 Basic stats: COMPLETE Column stats: NONE
File Output Operator
compressed: true
Statistics: Num rows: 7606 Data size: 319495 Basic stats: COMPLETE Column stats: NONE
table:
input format: org.apache.hadoop.mapred.TextInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Stage: Stage-3
Map Reduce
Map Operator Tree:
TableScan
alias: t
Statistics: Num rows: 636 Data size: 9296 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: fsetcode (type: string), fsetid (type: string), srscd (type: string)
outputColumnNames: fsetcode, fsetid, srscd
Statistics: Num rows: 636 Data size: 9296 Basic stats: COMPLETE Column stats: NONE
Group By Operator
keys: concat('A', lpad(fsetcode, 3, 0)) (type: string), fsetid (type: string), srscd (type: string)
mode: hash
outputColumnNames: _col0, _col1, _col2
Statistics: Num rows: 636 Data size: 9296 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col0 (type: string), _col1 (type: string), _col2 (type: string)
sort order: +++
Map-reduce partition columns: _col0 (type: string), _col1 (type: string), _col2 (type: string)
Statistics: Num rows: 636 Data size: 9296 Basic stats: COMPLETE Column stats: NONE
Reduce Operator Tree:
Group By Operator
keys: KEY._col0 (type: string), KEY._col1 (type: string), KEY._col2 (type: string)
mode: mergepartial
outputColumnNames: _col0, _col1, _col2
Statistics: Num rows: 318 Data size: 4648 Basic stats: COMPLETE Column stats: NONE
File Output Operator
compressed: false
table:
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe Stage: Stage-0
Fetch Operator
limit: -1
Processor Tree:
ListSink
在命令行里面启用了Hive的debug模式，由于debug模式信息输出比较多，我将信息数据到了文件里面：
hive -hiveconf hive.root.logger=INFO,console -e "
select a2.ISSUE_CODE as ISSUE_CODE,
a2.FZQDM as FZQDM,
a2.FZQLB as FZQLB,
a2.FJJDM as FJJDM,
a3.FSETCODE as FSETCODE,
a3.FSETID as FSETID,
a2.SRSCD as SRSCD
from (select t1.FSCDM as ISSUE_CODE,--市场代码
t1.FZQDM as FZQDM,
(case when instr(t1.FZQLB, '非银行间') > 0
then '非银行间'
else '银行间'
end) as FZQLB,
t1.FJJDM as FJJDM,
t1.SRSCD as SRSCD
from (select
a1.FZQDM as FZQDM,
a1.FZQLB as FZQLB,
a1.FJJDM as FJJDM,
a1.SRSCD as SRSCD,
a1.FSCDM as FSCDM,
row_number() over(partition by a1.FZQDM,a1.SRSCD order by length(trim(a1.FSCDM)) desc,a1.FSCDM desc) sem
from TMP.CS_DWM_ISSU_SRC2STD_REL_H_05 a1
where a1.FJJDM is not null or a1.FJJDM = ' ') t1
where t1.sem=1
) a2
left join (select distinct concat('A',lpad(t.FSETCODE,3,0)) as FSETCODE, --基金代码
t.FSETID as FSETID, --套账号
t.SRSCD as SRSCD
from TMP.CS_DWM_ISSU_SRC2STD_REL_H_06 t
)a3
on a3.FSETCODE = a2.FJJDM and a2.SRSCD=a3.SRSCD" >> hive.query.debug 2>&1
查看 hive.query.debug文件：
Logging initialized using configuration in jar:file:/opt/apache-hive-1.2.1-bin/lib/hive-common-1.2.1.jar!/hive-log4j.properties
16/12/20 22:15:58 [main]: INFO SessionState:
Logging initialized using configuration in jar:file:/opt/apache-hive-1.2.1-bin/lib/hive-common-1.2.1.jar!/hive-log4j.properties
16/12/20 22:15:58 [main]: WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/12/20 22:15:59 [main]: INFO hive.metastore: Trying to connect to metastore with URI thrift://qxy1:9083
16/12/20 22:15:59 [main]: INFO hive.metastore: Connected to metastore.
16/12/20 22:16:00 [main]: INFO session.SessionState: Created local directory: /home/hadoop/tmpdir/adaaea53-aac6-4dd7-b86c-53daed294f17_resources
16/12/20 22:16:00 [main]: INFO session.SessionState: Created HDFS directory: /tmp/hive/hadoop/adaaea53-aac6-4dd7-b86c-53daed294f17
16/12/20 22:16:00 [main]: INFO session.SessionState: Created local directory: /home/hadoop/tmpdir/hive/adaaea53-aac6-4dd7-b86c-53daed294f17
16/12/20 22:16:00 [main]: INFO session.SessionState: Created HDFS directory: /tmp/hive/hadoop/adaaea53-aac6-4dd7-b86c-53daed294f17/_tmp_space.db
16/12/20 22:16:00 [main]: INFO log.PerfLogger: <PERFLOG method=Driver.run from=org.apache.hadoop.hive.ql.Driver>
16/12/20 22:16:00 [main]: INFO log.PerfLogger: <PERFLOG method=TimeToSubmit from=org.apache.hadoop.hive.ql.Driver>
16/12/20 22:16:00 [main]: INFO log.PerfLogger: <PERFLOG method=compile from=org.apache.hadoop.hive.ql.Driver>
16/12/20 22:16:00 [main]: INFO log.PerfLogger: <PERFLOG method=parse from=org.apache.hadoop.hive.ql.Driver>
16/12/20 22:16:00 [main]: INFO parse.ParseDriver: Parsing command:
select a2.ISSUE_CODE as ISSUE_CODE,
a2.FZQDM as FZQDM,
a2.FZQLB as FZQLB,
a2.FJJDM as FJJDM,
a3.FSETCODE as FSETCODE,
a3.FSETID as FSETID,
a2.SRSCD as SRSCD
from (select t1.FSCDM as ISSUE_CODE,--市场代码
t1.FZQDM as FZQDM,
(case when instr(t1.FZQLB, '非银行间') > 0
then '非银行间'
else '银行间'
end) as FZQLB,
t1.FJJDM as FJJDM,
t1.SRSCD as SRSCD
from (select
a1.FZQDM as FZQDM,
a1.FZQLB as FZQLB,
a1.FJJDM as FJJDM,
a1.SRSCD as SRSCD,
a1.FSCDM as FSCDM,
row_number() over(partition by a1.FZQDM,a1.SRSCD order by length(trim(a1.FSCDM)) desc,a1.FSCDM desc) sem
from TMP.CS_DWM_ISSU_SRC2STD_REL_H_05 a1
where a1.FJJDM is not null or a1.FJJDM = ' ') t1
where t1.sem=1
) a2
left join (select distinct concat('A',lpad(t.FSETCODE,3,0)) as FSETCODE, --基金代码
t.FSETID as FSETID, --套账号
t.SRSCD as SRSCD
from TMP.CS_DWM_ISSU_SRC2STD_REL_H_06 t
)a3
on a3.FSETCODE = a2.FJJDM and a2.SRSCD=a3.SRSCD
16/12/20 22:16:00 [main]: INFO parse.ParseDriver: Parse Completed
16/12/20 22:16:00 [main]: INFO log.PerfLogger: </PERFLOG method=parse start=1482300960101 end=1482300960807 duration=706 from=org.apache.hadoop.hive.ql.Driver>
16/12/20 22:16:00 [main]: INFO log.PerfLogger: <PERFLOG method=semanticAnalyze from=org.apache.hadoop.hive.ql.Driver>
16/12/20 22:16:00 [main]: INFO parse.CalcitePlanner: Starting Semantic Analysis
16/12/20 22:16:01 [main]: INFO parse.CalcitePlanner: Completed phase 1 of Semantic Analysis
16/12/20 22:16:01 [main]: INFO parse.CalcitePlanner: Get metadata for source tables
16/12/20 22:16:01 [main]: INFO parse.CalcitePlanner: Get metadata for subqueries
16/12/20 22:16:01 [main]: INFO parse.CalcitePlanner: Get metadata for source tables
16/12/20 22:16:01 [main]: INFO parse.CalcitePlanner: Get metadata for subqueries
16/12/20 22:16:01 [main]: INFO parse.CalcitePlanner: Get metadata for source tables
16/12/20 22:16:01 [main]: INFO parse.CalcitePlanner: Get metadata for subqueries
16/12/20 22:16:01 [main]: INFO parse.CalcitePlanner: Get metadata for destination tables
16/12/20 22:16:01 [main]: INFO parse.CalcitePlanner: Get metadata for destination tables
16/12/20 22:16:01 [main]: INFO parse.CalcitePlanner: Get metadata for source tables
16/12/20 22:16:01 [main]: INFO parse.CalcitePlanner: Get metadata for subqueries
16/12/20 22:16:01 [main]: INFO parse.CalcitePlanner: Get metadata for destination tables
16/12/20 22:16:01 [main]: INFO parse.CalcitePlanner: Get metadata for destination tables
16/12/20 22:16:01 [main]: INFO ql.Context: New scratch dir is hdfs://qxy1/tmp/hive/hadoop/adaaea53-aac6-4dd7-b86c-53daed294f17/hive_2016-12-20_22-16-00_099_8319005040929428210-1
16/12/20 22:16:01 [main]: INFO parse.CalcitePlanner: Completed getting MetaData in Semantic Analysis
16/12/20 22:16:01 [main]: INFO parse.BaseSemanticAnalyzer: Not invoking CBO because the statement has too few joins
16/12/20 22:16:01 [main]: INFO common.FileUtils: Creating directory if it doesn't exist: hdfs://qxy1/tmp/hive/hadoop/adaaea53-aac6-4dd7-b86c-53daed294f17/hive_2016-12-20_22-16-00_099_8319005040929428210-1/-mr-10000/.hive-staging_hive_2016-12-20_22-16-00_099_8319005040929428210-1
16/12/20 22:16:02 [main]: INFO parse.CalcitePlanner: Set stats collection dir : hdfs://qxy1/tmp/hive/hadoop/adaaea53-aac6-4dd7-b86c-53daed294f17/hive_2016-12-20_22-16-00_099_8319005040929428210-1/-mr-10000/.hive-staging_hive_2016-12-20_22-16-00_099_8319005040929428210-1/-ext-10002
16/12/20 22:16:02 [main]: INFO ppd.OpProcFactory: Processing for FS(19)
16/12/20 22:16:02 [main]: INFO ppd.OpProcFactory: Processing for SEL(18)
16/12/20 22:16:02 [main]: INFO ppd.OpProcFactory: Processing for JOIN(17)
16/12/20 22:16:02 [main]: INFO ppd.OpProcFactory: Processing for RS(15)
16/12/20 22:16:02 [main]: INFO ppd.OpProcFactory: Processing for SEL(8)
16/12/20 22:16:02 [main]: INFO ppd.OpProcFactory: Processing for FIL(7)
16/12/20 22:16:02 [main]: INFO ppd.OpProcFactory: Pushdown Predicates of FIL For Alias : t1
16/12/20 22:16:02 [main]: INFO ppd.OpProcFactory: (_col5 = 1)
16/12/20 22:16:02 [main]: INFO ppd.OpProcFactory: Processing for SEL(6)
16/12/20 22:16:02 [main]: INFO ppd.OpProcFactory: Pushdown Predicates of SEL For Alias : t1
16/12/20 22:16:02 [main]: INFO ppd.OpProcFactory: (row_number_window_0 = 1)
16/12/20 22:16:02 [main]: INFO ppd.OpProcFactory: Processing for SEL(5)
16/12/20 22:16:02 [main]: INFO ppd.OpProcFactory: Pushdown Predicates of SEL For Alias :
16/12/20 22:16:02 [main]: INFO ppd.OpProcFactory: (row_number_window_0 = 1)
16/12/20 22:16:02 [main]: INFO ppd.OpProcFactory: Processing for PTF(4)
16/12/20 22:16:02 [main]: INFO ppd.OpProcFactory: Processing for PTF(4)
16/12/20 22:16:02 [main]: INFO ppd.OpProcFactory: Pushdown Predicates of PTF For Alias :
16/12/20 22:16:02 [main]: INFO ppd.OpProcFactory: (row_number_window_0 = 1)
16/12/20 22:16:02 [main]: INFO ppd.OpProcFactory: Processing for SEL(3)
16/12/20 22:16:02 [main]: INFO ppd.OpProcFactory: Processing for RS(2)
16/12/20 22:16:02 [main]: INFO ppd.OpProcFactory: Processing for FIL(1)
16/12/20 22:16:02 [main]: INFO ppd.OpProcFactory: Pushdown Predicates of FIL For Alias : a1
16/12/20 22:16:02 [main]: INFO ppd.OpProcFactory: (fjjdm is not null or (fjjdm = ' '))
16/12/20 22:16:02 [main]: INFO ppd.OpProcFactory: Processing for TS(0)
16/12/20 22:16:02 [main]: INFO ppd.OpProcFactory: Pushdown Predicates of TS For Alias : a1
16/12/20 22:16:02 [main]: INFO ppd.OpProcFactory: (fjjdm is not null or (fjjdm = ' '))
16/12/20 22:16:02 [main]: INFO ppd.OpProcFactory: Processing for RS(16)
16/12/20 22:16:02 [main]: INFO ppd.OpProcFactory: Processing for SEL(14)
16/12/20 22:16:02 [main]: INFO ppd.OpProcFactory: Processing for GBY(13)
16/12/20 22:16:02 [main]: INFO ppd.OpProcFactory: Processing for RS(12)
16/12/20 22:16:02 [main]: INFO ppd.OpProcFactory: Processing for GBY(11)
16/12/20 22:16:02 [main]: INFO ppd.OpProcFactory: Processing for SEL(10)
16/12/20 22:16:02 [main]: INFO ppd.OpProcFactory: Processing for TS(9)
16/12/20 22:16:02 [main]: INFO log.PerfLogger: <PERFLOG method=partition-retrieving from=org.apache.hadoop.hive.ql.optimizer.ppr.PartitionPruner>
16/12/20 22:16:02 [main]: INFO log.PerfLogger: </PERFLOG method=partition-retrieving start=1482300962166 end=1482300962169 duration=3 from=org.apache.hadoop.hive.ql.optimizer.ppr.PartitionPruner>
16/12/20 22:16:02 [main]: INFO optimizer.ColumnPrunerProcFactory: JOIN 17 oldExprs: {0=[Column[VALUE._col0], Column[VALUE._col1], Column[VALUE._col2], Column[KEY.reducesinkkey0], Column[KEY.reducesinkkey1]], 1=[Column[KEY.reducesinkkey0], Column[VALUE._col0], Column[KEY.reducesinkkey1]]}
16/12/20 22:16:02 [main]: INFO optimizer.ColumnPrunerProcFactory: JOIN 17 newExprs: {0=[Column[VALUE._col0], Column[VALUE._col1], Column[VALUE._col2], Column[KEY.reducesinkkey0], Column[KEY.reducesinkkey1]], 1=[Column[KEY.reducesinkkey0], Column[VALUE._col0]]}
16/12/20 22:16:02 [main]: INFO optimizer.ColumnPrunerProcFactory: RS 15 oldColExprMap: {VALUE._col2=Column[_col2], VALUE._col0=Column[_col0], VALUE._col1=Column[_col1], KEY.reducesinkkey0=Column[_col3], KEY.reducesinkkey1=Column[_col4]}
16/12/20 22:16:02 [main]: INFO optimizer.ColumnPrunerProcFactory: RS 15 newColExprMap: {VALUE._col2=Column[_col2], VALUE._col0=Column[_col0], VALUE._col1=Column[_col1], KEY.reducesinkkey0=Column[_col3], KEY.reducesinkkey1=Column[_col4]}
16/12/20 22:16:02 [main]: INFO optimizer.ColumnPrunerProcFactory: RS 16 oldColExprMap: {VALUE._col0=Column[_col1], KEY.reducesinkkey0=Column[_col0], KEY.reducesinkkey1=Column[_col2]}
16/12/20 22:16:02 [main]: INFO optimizer.ColumnPrunerProcFactory: RS 16 newColExprMap: {VALUE._col0=Column[_col1], KEY.reducesinkkey0=Column[_col0], KEY.reducesinkkey1=Column[_col2]}
16/12/20 22:16:02 [main]: INFO optimizer.ColumnPrunerProcFactory: RS 12 oldColExprMap: {KEY._col0=Column[_col0], KEY._col1=Column[_col1], KEY._col2=Column[_col2]}
16/12/20 22:16:02 [main]: INFO optimizer.ColumnPrunerProcFactory: RS 12 newColExprMap: {KEY._col0=Column[_col0], KEY._col1=Column[_col1], KEY._col2=Column[_col2]}
16/12/20 22:16:02 [main]: INFO optimizer.ColumnPrunerProcFactory: RS 2 oldColExprMap: {VALUE._col2=Column[BLOCK__OFFSET__INSIDE__FILE], VALUE._col3=Column[INPUT__FILE__NAME], VALUE._col4=Column[ROW__ID], VALUE._col0=Column[fzqlb], VALUE._col1=Column[fjjdm], KEY.reducesinkkey0=Column[fzqdm], KEY.reducesinkkey1=Column[srscd], KEY.reducesinkkey2=GenericUDFBridge(GenericUDFTrim(Column[fscdm])), KEY.reducesinkkey3=Column[fscdm]}
16/12/20 22:16:02 [main]: INFO optimizer.ColumnPrunerProcFactory: RS 2 newColExprMap: {VALUE._col0=Column[fzqlb], VALUE._col1=Column[fjjdm], KEY.reducesinkkey0=Column[fzqdm], KEY.reducesinkkey1=Column[srscd], KEY.reducesinkkey2=GenericUDFBridge(GenericUDFTrim(Column[fscdm])), KEY.reducesinkkey3=Column[fscdm]}
16/12/20 22:16:02 [main]: INFO ql.Context: New scratch dir is hdfs://qxy1/tmp/hive/hadoop/adaaea53-aac6-4dd7-b86c-53daed294f17/hive_2016-12-20_22-16-00_099_8319005040929428210-1
16/12/20 22:16:02 [main]: INFO log.PerfLogger: <PERFLOG method=partition-retrieving from=org.apache.hadoop.hive.ql.optimizer.ppr.PartitionPruner>
16/12/20 22:16:02 [main]: INFO log.PerfLogger: </PERFLOG method=partition-retrieving start=1482300962235 end=1482300962235 duration=0 from=org.apache.hadoop.hive.ql.optimizer.ppr.PartitionPruner>
16/12/20 22:16:02 [main]: INFO ql.Context: New scratch dir is hdfs://qxy1/tmp/hive/hadoop/adaaea53-aac6-4dd7-b86c-53daed294f17/hive_2016-12-20_22-16-00_099_8319005040929428210-1
16/12/20 22:16:02 [main]: INFO log.PerfLogger: <PERFLOG method=getInputSummary from=org.apache.hadoop.hive.ql.exec.Utilities>
16/12/20 22:16:02 [main]: INFO exec.Utilities: Cannot get size of hdfs://qxy1/tmp/hive/hadoop/adaaea53-aac6-4dd7-b86c-53daed294f17/hive_2016-12-20_22-16-00_099_8319005040929428210-1/-mr-10003. Safely ignored.
16/12/20 22:16:02 [main]: INFO exec.Utilities: Cannot get size of hdfs://qxy1/tmp/hive/hadoop/adaaea53-aac6-4dd7-b86c-53daed294f17/hive_2016-12-20_22-16-00_099_8319005040929428210-1/-mr-10004. Safely ignored.
16/12/20 22:16:02 [main]: INFO log.PerfLogger: </PERFLOG method=getInputSummary start=1482300962243 end=1482300962258 duration=15 from=org.apache.hadoop.hive.ql.exec.Utilities>
16/12/20 22:16:02 [main]: INFO log.PerfLogger: <PERFLOG method=clonePlan from=org.apache.hadoop.hive.ql.exec.Utilities>
16/12/20 22:16:02 [main]: INFO log.PerfLogger: <PERFLOG method=serializePlan from=org.apache.hadoop.hive.ql.exec.Utilities>
16/12/20 22:16:02 [main]: INFO exec.Utilities: Serializing MapredWork via kryo
16/12/20 22:16:02 [main]: INFO log.PerfLogger: </PERFLOG method=serializePlan start=1482300962335 end=1482300962407 duration=72 from=org.apache.hadoop.hive.ql.exec.Utilities>
16/12/20 22:16:02 [main]: INFO log.PerfLogger: <PERFLOG method=deserializePlan from=org.apache.hadoop.hive.ql.exec.Utilities>
16/12/20 22:16:02 [main]: INFO exec.Utilities: Deserializing MapredWork via kryo
16/12/20 22:16:02 [main]: INFO log.PerfLogger: </PERFLOG method=deserializePlan start=1482300962407 end=1482300962435 duration=28 from=org.apache.hadoop.hive.ql.exec.Utilities>
16/12/20 22:16:02 [main]: INFO log.PerfLogger: </PERFLOG method=clonePlan start=1482300962265 end=1482300962435 duration=170 from=org.apache.hadoop.hive.ql.exec.Utilities>
16/12/20 22:16:02 [main]: INFO ql.Context: New scratch dir is hdfs://qxy1/tmp/hive/hadoop/adaaea53-aac6-4dd7-b86c-53daed294f17/hive_2016-12-20_22-16-00_099_8319005040929428210-1
16/12/20 22:16:02 [main]: INFO ql.Context: New scratch dir is hdfs://qxy1/tmp/hive/hadoop/adaaea53-aac6-4dd7-b86c-53daed294f17/hive_2016-12-20_22-16-00_099_8319005040929428210-1
16/12/20 22:16:02 [main]: INFO physical.LocalMapJoinProcFactory: Setting max memory usage to 0.9 for table sink not followed by group by
16/12/20 22:16:02 [main]: INFO physical.NullScanTaskDispatcher: Looking for table scans where optimization is applicable
16/12/20 22:16:02 [main]: INFO physical.NullScanTaskDispatcher: Found 0 null table scans
16/12/20 22:16:02 [main]: INFO physical.NullScanTaskDispatcher: Looking for table scans where optimization is applicable
16/12/20 22:16:02 [main]: INFO physical.NullScanTaskDispatcher: Found 0 null table scans
16/12/20 22:16:02 [main]: INFO physical.NullScanTaskDispatcher: Looking for table scans where optimization is applicable
16/12/20 22:16:02 [main]: INFO physical.NullScanTaskDispatcher: Found 0 null table scans
16/12/20 22:16:02 [main]: INFO physical.NullScanTaskDispatcher: Looking for table scans where optimization is applicable
16/12/20 22:16:02 [main]: INFO physical.NullScanTaskDispatcher: Found 0 null table scans
16/12/20 22:16:02 [main]: INFO physical.NullScanTaskDispatcher: Looking for table scans where optimization is applicable
16/12/20 22:16:02 [main]: INFO physical.NullScanTaskDispatcher: Found 0 null table scans
16/12/20 22:16:02 [main]: INFO physical.NullScanTaskDispatcher: Looking for table scans where optimization is applicable
16/12/20 22:16:02 [main]: INFO physical.NullScanTaskDispatcher: Found 0 null table scans
16/12/20 22:16:02 [main]: INFO parse.CalcitePlanner: Completed plan generation
16/12/20 22:16:02 [main]: INFO ql.Driver: Semantic Analysis Completed
16/12/20 22:16:02 [main]: INFO log.PerfLogger: </PERFLOG method=semanticAnalyze start=1482300960810 end=1482300962455 duration=1645 from=org.apache.hadoop.hive.ql.Driver>
16/12/20 22:16:02 [main]: INFO exec.ListSinkOperator: Initializing operator OP[36]
16/12/20 22:16:02 [main]: INFO exec.ListSinkOperator: Initialization Done 36 OP
16/12/20 22:16:02 [main]: INFO exec.ListSinkOperator: Operator 36 OP initialized
16/12/20 22:16:02 [main]: INFO ql.Driver: Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:issue_code, type:string, comment:null), FieldSchema(name:fzqdm, type:string, comment:null), FieldSchema(name:fzqlb, type:string, comment:null), FieldSchema(name:fjjdm, type:string, comment:null), FieldSchema(name:fsetcode, type:string, comment:null), FieldSchema(name:fsetid, type:string, comment:null), FieldSchema(name:srscd, type:string, comment:null)], properties:null)
16/12/20 22:16:02 [main]: INFO log.PerfLogger: </PERFLOG method=compile start=1482300960067 end=1482300962528 duration=2461 from=org.apache.hadoop.hive.ql.Driver>
16/12/20 22:16:02 [main]: INFO log.PerfLogger: <PERFLOG method=acquireReadWriteLocks from=org.apache.hadoop.hive.ql.Driver>
16/12/20 22:16:02 [main]: INFO lockmgr.DummyTxnManager: Creating lock manager of type org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager
16/12/20 22:16:02 [main]: INFO imps.CuratorFrameworkImpl: Starting
16/12/20 22:16:02 [main]: INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
16/12/20 22:16:02 [main]: INFO zookeeper.ZooKeeper: Client environment:host.name=qxy1
16/12/20 22:16:02 [main]: INFO zookeeper.ZooKeeper: Client environment:java.version=1.8.0_77
16/12/20 22:16:02 [main]: INFO zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation
16/12/20 22:16:02 [main]: INFO zookeeper.ZooKeeper: Client environment:java.home=/opt/jdk1.8.0_77/jre
16/12/20 22:16:02 [main]: INFO zookeeper.ZooKeeper: Client environment:java.class.path=/opt/hadoop-2.6.2/etc/hadoop:/opt/hadoop-2.6.2/share/hadoop/common/lib/xz-1.0.jar:/opt/hadoop-2.6.2/share/hadoop/common/lib/commons-el-1.0.jar:/opt/hadoop-2.6.2/share/hadoop/common/lib/commons-codec-1.4.jar:/opt/hadoop-2.6.2/share/hadoop/common/lib/commons-net-3.1.jar:/opt/hadoop-2.6.2/share/hadoop/common/lib/stax-api-1.0-2.jar:/opt/hadoop-2.6.2/share/hadoop/common/lib/commons-math3-3.1.1.jar:/opt/hadoop-2.6.2/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/opt/hadoop-2.6.2/share/hadoop/common/lib/jasper-runtime-5.5.23.jar:/opt/hadoop-2.6.2/share/hadoop/common/lib/htrace-core-3.0.4.jar:/opt/hadoop-2.6.2/share/hadoop/common/lib/xmlenc-0.52.jar:/opt/hadoop-2.6.2/share/hadoop/common/lib/curator-recipes-2.6.0.jar:/opt/hadoop-2.6.2/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/opt/hadoop-2.6.2/share/hadoop/common/lib/hadoop-annotations-2.6.2.jar:/opt/hadoop-2.6.2/share/hadoop/common/lib/commons-collections-3.2.1.jar:/opt/hadoop-2.6.2/share/hadoop/common/lib/jersey-core-1.9.jar:/opt/hadoop-2.6.2/share/hadoop/common/lib/guava-11.0.2.jar:/opt/hadoop-2.6.2/share/hadoop/common/lib/curator-client-2.6.0.jar:/opt/hadoop-2.6.2/share/hadoop/common/lib/commons-lang-2.6.jar:/opt/hadoop-2.6.2/share/hadoop/common/lib/slf4j-api-1.7.5.jar:/opt/hadoop-2.6.2/share/hadoop/common/lib/jetty-6.1.26.jar:/opt/hadoop-2.6.2/share/hadoop/common/lib/commons-digester-1.8.jar:/opt/hadoop-2.6.2/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/opt/hadoop-2.6.2/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/opt/hadoop-2.6.2/share/hadoop/common/lib/paranamer-2.3.jar:/opt/hadoop-2.6.2/share/hadoop/common/lib/activation-1.1.jar:/opt/hadoop-2.6.2/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/opt/hadoop-2.6.2/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/opt/hadoop-2.6.2/share/hadoop/common/lib/jets3t-0.9.0.jar:/opt/hadoop-2.6.2/share/hadoop/common/lib/commons-logging-1.1.3.jar:/opt/hadoop-2.6.2/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/opt/hadoop-2.6.2/share/hadoop/common/lib/httpclient-4.2.5.jar:/opt/hadoop-2.6.2/share/hadoop/common/lib/hamcrest-core-1.3.jar:/opt/hadoop-2.6.2/share/hadoop/common/lib/jetty-util-6.1.26.jar:/opt/hadoop-2.6.2/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/opt/hadoop-2.6.2/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/opt/hadoop-2.6.2/share/hadoop/common/lib/jersey-server-1.9.jar:/opt/hadoop-2.6.2/share/hadoop/common/lib/hadoop-auth-2.6.2.jar:/opt/hadoop-2.6.2/share/hadoop/common/lib/commons-io-2.4.jar:/opt/hadoop-2.6.2/share/hadoop/common/lib/netty-3.6.2.Final.jar:/opt/hadoop-2.6.2/share/hadoop/common/lib/jsr305-1.3.9.jar:/opt/hadoop-2.6.2/share/hadoop/common/lib/jsch-0.1.42.jar:/opt/hadoop-2.6.2/share/hadoop/common/lib/commons-configuration-1.6.jar:/opt/hadoop-2.6.2/share/hadoop/common/lib/zookeeper-3.4.6.jar:/opt/hadoop-2.6.2/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/opt/hadoop-2.6.2/share/hadoop/common/lib/jasper-compiler-5.5.23.jar:/opt/hadoop-2.6.2/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/opt/hadoop-2.6.2/share/hadoop/common/lib/commons-cli-1.2.jar:/opt/hadoop-2.6.2/share/hadoop/common/lib/jersey-json-1.9.jar:/opt/hadoop-2.6.2/share/hadoop/common/lib/hadoop-lzo-0.4.21-SNAPSHOT.jar:/opt/hadoop-2.6.2/share/hadoop/common/lib/junit-4.11.jar:/opt/hadoop-2.6.2/share/hadoop/common/lib/servlet-api-2.5.jar:/opt/hadoop-2.6.2/share/hadoop/common/lib/httpcore-4.2.5.jar:/opt/hadoop-2.6.2/share/hadoop/common/lib/commons-compress-1.4.1.jar:/opt/hadoop-2.6.2/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/opt/hadoop-2.6.2/share/hadoop/common/lib/asm-3.2.jar:/opt/hadoop-2.6.2/share/hadoop/common/lib/gson-2.2.4.jar:/opt/hadoop-2.6.2/share/hadoop/common/lib/commons-httpclient-3.1.jar:/opt/hadoop-2.6.2/share/hadoop/common/lib/curator-framework-2.6.0.jar:/opt/hadoop-2.6.2/share/hadoop/common/lib/mockito-all-1.8.5.jar:/opt/hadoop-2.6.2/share/hadoop/common/lib/avro-1.7.4.jar:/opt/hadoop-2.6.2/share/hadoop/common/lib/jettison-1.1.jar:/opt/hadoop-2.6.2/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/opt/hadoop-2.6.2/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/opt/hadoop-2.6.2/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/opt/hadoop-2.6.2/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar:/opt/hadoop-2.6.2/share/hadoop/common/lib/jsp-api-2.1.jar:/opt/hadoop-2.6.2/share/hadoop/common/lib/log4j-1.2.17.jar:/opt/hadoop-2.6.2/share/hadoop/common/hadoop-common-2.6.2-tests.jar:/opt/hadoop-2.6.2/share/hadoop/common/hadoop-common-2.6.2.jar:/opt/hadoop-2.6.2/share/hadoop/common/hadoop-nfs-2.6.2.jar:/opt/hadoop-2.6.2/share/hadoop/hdfs:/opt/hadoop-2.6.2/share/hadoop/hdfs/lib/commons-el-1.0.jar:/opt/hadoop-2.6.2/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/opt/hadoop-2.6.2/share/hadoop/hdfs/lib/jasper-runtime-5.5.23.jar:/opt/hadoop-2.6.2/share/hadoop/hdfs/lib/htrace-core-3.0.4.jar:/opt/hadoop-2.6.2/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/opt/hadoop-2.6.2/share/hadoop/hdfs/lib/xercesImpl-2.9.1.jar:/opt/hadoop-2.6.2/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/opt/hadoop-2.6.2/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/opt/hadoop-2.6.2/share/hadoop/hdfs/lib/guava-11.0.2.jar:/opt/hadoop-2.6.2/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/opt/hadoop-2.6.2/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/opt/hadoop-2.6.2/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/opt/hadoop-2.6.2/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/opt/hadoop-2.6.2/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/opt/hadoop-2.6.2/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/opt/hadoop-2.6.2/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/opt/hadoop-2.6.2/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/opt/hadoop-2.6.2/share/hadoop/hdfs/lib/commons-io-2.4.jar:/opt/hadoop-2.6.2/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/opt/hadoop-2.6.2/share/hadoop/hdfs/lib/jsr305-1.3.9.jar:/opt/hadoop-2.6.2/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/opt/hadoop-2.6.2/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/opt/hadoop-2.6.2/share/hadoop/hdfs/lib/asm-3.2.jar:/opt/hadoop-2.6.2/share/hadoop/hdfs/lib/xml-apis-1.3.04.jar:/opt/hadoop-2.6.2/share/hadoop/hdfs/lib/jsp-api-2.1.jar:/opt/hadoop-2.6.2/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/opt/hadoop-2.6.2/share/hadoop/hdfs/hadoop-hdfs-2.6.2.jar:/opt/hadoop-2.6.2/share/hadoop/hdfs/hadoop-hdfs-nfs-2.6.2.jar:/opt/hadoop-2.6.2/share/hadoop/hdfs/hadoop-hdfs-2.6.2-tests.jar:/opt/hadoop-2.6.2/share/hadoop/yarn/lib/xz-1.0.jar:/opt/hadoop-2.6.2/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/opt/hadoop-2.6.2/share/hadoop/yarn/lib/commons-codec-1.4.jar:/opt/hadoop-2.6.2/share/hadoop/yarn/lib/stax-api-1.0-2.jar:/opt/hadoop-2.6.2/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/opt/hadoop-2.6.2/share/hadoop/yarn/lib/commons-collections-3.2.1.jar:/opt/hadoop-2.6.2/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/opt/hadoop-2.6.2/share/hadoop/yarn/lib/jersey-core-1.9.jar:/opt/hadoop-2.6.2/share/hadoop/yarn/lib/guava-11.0.2.jar:/opt/hadoop-2.6.2/share/hadoop/yarn/lib/commons-lang-2.6.jar:/opt/hadoop-2.6.2/share/hadoop/yarn/lib/jetty-6.1.26.jar:/opt/hadoop-2.6.2/share/hadoop/yarn/lib/jackson-mapper-asl-1.9.13.jar:/opt/hadoop-2.6.2/share/hadoop/yarn/lib/jackson-core-asl-1.9.13.jar:/opt/hadoop-2.6.2/share/hadoop/yarn/lib/aopalliance-1.0.jar:/opt/hadoop-2.6.2/share/hadoop/yarn/lib/activation-1.1.jar:/opt/hadoop-2.6.2/share/hadoop/yarn/lib/commons-logging-1.1.3.jar:/opt/hadoop-2.6.2/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/opt/hadoop-2.6.2/share/hadoop/yarn/lib/guice-3.0.jar:/opt/hadoop-2.6.2/share/hadoop/yarn/lib/jetty-util-6.1.26.jar:/opt/hadoop-2.6.2/share/hadoop/yarn/lib/jersey-server-1.9.jar:/opt/hadoop-2.6.2/share/hadoop/yarn/lib/commons-io-2.4.jar:/opt/hadoop-2.6.2/share/hadoop/yarn/lib/netty-3.6.2.Final.jar:/opt/hadoop-2.6.2/share/hadoop/yarn/lib/jsr305-1.3.9.jar:/opt/hadoop-2.6.2/share/hadoop/yarn/lib/zookeeper-3.4.6.jar:/opt/hadoop-2.6.2/share/hadoop/yarn/lib/jackson-xc-1.9.13.jar:/opt/hadoop-2.6.2/share/hadoop/yarn/lib/commons-cli-1.2.jar:/opt/hadoop-2.6.2/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar:/opt/hadoop-2.6.2/share/hadoop/yarn/lib/jersey-json-1.9.jar:/opt/hadoop-2.6.2/share/hadoop/yarn/lib/servlet-api-2.5.jar:/opt/hadoop-2.6.2/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/opt/hadoop-2.6.2/share/hadoop/yarn/lib/asm-3.2.jar:/opt/hadoop-2.6.2/share/hadoop/yarn/lib/commons-httpclient-3.1.jar:/opt/hadoop-2.6.2/share/hadoop/yarn/lib/javax.inject-1.jar:/opt/hadoop-2.6.2/share/hadoop/yarn/lib/jersey-client-1.9.jar:/opt/hadoop-2.6.2/share/hadoop/yarn/lib/jettison-1.1.jar:/opt/hadoop-2.6.2/share/hadoop/yarn/lib/jackson-jaxrs-1.9.13.jar:/opt/hadoop-2.6.2/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar:/opt/hadoop-2.6.2/share/hadoop/yarn/lib/log4j-1.2.17.jar:/opt/hadoop-2.6.2/share/hadoop/yarn/hadoop-yarn-api-2.6.2.jar:/opt/hadoop-2.6.2/share/hadoop/yarn/hadoop-yarn-server-common-2.6.2.jar:/opt/hadoop-2.6.2/share/hadoop/yarn/hadoop-yarn-client-2.6.2.jar:/opt/hadoop-2.6.2/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.6.2.jar:/opt/hadoop-2.6.2/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.6.2.jar:/opt/hadoop-2.6.2/share/hadoop/yarn/hadoop-yarn-common-2.6.2.jar:/opt/hadoop-2.6.2/share/hadoop/yarn/hadoop-yarn-server-tests-2.6.2.jar:/opt/hadoop-2.6.2/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.6.2.jar:/opt/hadoop-2.6.2/share/hadoop/yarn/hadoop-yarn-registry-2.6.2.jar:/opt/hadoop-2.6.2/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.6.2.jar:/opt/hadoop-2.6.2/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.6.2.jar:/opt/hadoop-2.6.2/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.6.2.jar:/opt/hadoop-2.6.2/share/hadoop/mapreduce/lib/xz-1.0.jar:/opt/hadoop-2.6.2/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/opt/hadoop-2.6.2/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar:/opt/hadoop-2.6.2/share/hadoop/mapreduce/lib/hadoop-annotations-2.6.2.jar:/opt/hadoop-2.6.2/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/opt/hadoop-2.6.2/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/opt/hadoop-2.6.2/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/opt/hadoop-2.6.2/share/hadoop/mapreduce/lib/jackson-core-asl-1.9.13.jar:/opt/hadoop-2.6.2/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/opt/hadoop-2.6.2/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/opt/hadoop-2.6.2/share/hadoop/mapreduce/lib/guice-3.0.jar:/opt/hadoop-2.6.2/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar:/opt/hadoop-2.6.2/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/opt/hadoop-2.6.2/share/hadoop/mapreduce/lib/commons-io-2.4.jar:/opt/hadoop-2.6.2/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar:/opt/hadoop-2.6.2/share/hadoop/mapreduce/lib/leveldbjni-all-1.8.jar:/opt/hadoop-2.6.2/share/hadoop/mapreduce/lib/junit-4.11.jar:/opt/hadoop-2.6.2/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/opt/hadoop-2.6.2/share/hadoop/mapreduce/lib/asm-3.2.jar:/opt/hadoop-2.6.2/share/hadoop/mapreduce/lib/javax.inject-1.jar:/opt/hadoop-2.6.2/share/hadoop/mapreduce/lib/avro-1.7.4.jar:/opt/hadoop-2.6.2/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/opt/hadoop-2.6.2/share/hadoop/mapreduce/lib/log4j-1.2.17.jar:/opt/hadoop-2.6.2/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.6.2.jar:/opt/hadoop-2.6.2/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.2.jar:/opt/hadoop-2.6.2/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.6.2.jar:/opt/hadoop-2.6.2/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.6.2.jar:/opt/hadoop-2.6.2/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.6.2.jar:/opt/hadoop-2.6.2/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.6.2.jar:/opt/hadoop-2.6.2/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.6.2.jar:/opt/hadoop-2.6.2/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.6.2-tests.jar:/opt/hadoop-2.6.2/share/hadoop/mapreduce/wc.jar:/opt/hadoop-2.6.2/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.6.2.jar:/opt/apache-hive-1.2.1-bin/conf:/opt/apache-hive-1.2.1-bin/lib/accumulo-core-1.6.0.jar:/opt/apache-hive-1.2.1-bin/lib/accumulo-fate-1.6.0.jar:/opt/apache-hive-1.2.1-bin/lib/accumulo-start-1.6.0.jar:/opt/apache-hive-1.2.1-bin/lib/accumulo-trace-1.6.0.jar:/opt/apache-hive-1.2.1-bin/lib/activation-1.1.jar:/opt/apache-hive-1.2.1-bin/lib/ant-1.9.1.jar:/opt/apache-hive-1.2.1-bin/lib/ant-launcher-1.9.1.jar:/opt/apache-hive-1.2.1-bin/lib/antlr-2.7.7.jar:/opt/apache-hive-1.2.1-bin/lib/antlr-runtime-3.4.jar:/opt/apache-hive-1.2.1-bin/lib/apache-log4j-extras-1.2.17.jar:/opt/apache-hive-1.2.1-bin/lib/asm-commons-3.1.jar:/opt/apache-hive-1.2.1-bin/lib/asm-tree-3.1.jar:/opt/apache-hive-1.2.1-bin/lib/avro-1.7.5.jar:/opt/apache-hive-1.2.1-bin/lib/bonecp-0.8.0.RELEASE.jar:/opt/apache-hive-1.2.1-bin/lib/calcite-avatica-1.2.0-incubating.jar:/opt/apache-hive-1.2.1-bin/lib/calcite-core-1.2.0-incubating.jar:/opt/apache-hive-1.2.1-bin/lib/calcite-linq4j-1.2.0-incubating.jar:/opt/apache-hive-1.2.1-bin/lib/commons-beanutils-1.7.0.jar:/opt/apache-hive-1.2.1-bin/lib/commons-beanutils-core-1.8.0.jar:/opt/apache-hive-1.2.1-bin/lib/commons-cli-1.2.jar:/opt/apache-hive-1.2.1-bin/lib/commons-codec-1.4.jar:/opt/apache-hive-1.2.1-bin/lib/commons-collections-3.2.1.jar:/opt/apache-hive-1.2.1-bin/lib/commons-compiler-2.7.6.jar:/opt/apache-hive-1.2.1-bin/lib/commons-compress-1.4.1.jar:/opt/apache-hive-1.2.1-bin/lib/commons-configuration-1.6.jar:/opt/apache-hive-1.2.1-bin/lib/commons-dbcp-1.4.jar:/opt/apache-hive-1.2.1-bin/lib/commons-digester-1.8.jar:/opt/apache-hive-1.2.1-bin/lib/commons-httpclient-3.0.1.jar:/opt/apache-hive-1.2.1-bin/lib/commons-io-2.4.jar:/opt/apache-hive-1.2.1-bin/lib/commons-lang-2.6.jar:/opt/apache-hive-1.2.1-bin/lib/commons-logging-1.1.3.jar:/opt/apache-hive-1.2.1-bin/lib/commons-math-2.1.jar:/opt/apache-hive-1.2.1-bin/lib/commons-pool-1.5.4.jar:/opt/apache-hive-1.2.1-bin/lib/commons-vfs2-2.0.jar:/opt/apache-hive-1.2.1-bin/lib/curator-client-2.6.0.jar:/opt/apache-hive-1.2.1-bin/lib/curator-framework-2.6.0.jar:/opt/apache-hive-1.2.1-bin/lib/curator-recipes-2.6.0.jar:/opt/apache-hive-1.2.1-bin/lib/datanucleus-api-jdo-3.2.6.jar:/opt/apache-hive-1.2.1-bin/lib/datanucleus-core-3.2.10.jar:/opt/apache-hive-1.2.1-bin/lib/datanucleus-rdbms-3.2.9.jar:/opt/apache-hive-1.2.1-bin/lib/derby-10.10.2.0.jar:/opt/apache-hive-1.2.1-bin/lib/eigenbase-properties-1.1.5.jar:/opt/apache-hive-1.2.1-bin/lib/geronimo-annotation_1.0_spec-1.1.1.jar:/opt/apache-hive-1.2.1-bin/lib/geronimo-jaspic_1.0_spec-1.0.jar:/opt/apache-hive-1.2.1-bin/lib/geronimo-jta_1.1_spec-1.1.1.jar:/opt/apache-hive-1.2.1-bin/lib/groovy-all-2.1.6.jar:/opt/apache-hive-1.2.1-bin/lib/guava-14.0.1.jar:/opt/apache-hive-1.2.1-bin/lib/hamcrest-core-1.1.jar:/opt/apache-hive-1.2.1-bin/lib/hive-accumulo-handler-1.2.1.jar:/opt/apache-hive-1.2.1-bin/lib/hive-ant-1.2.1.jar:/opt/apache-hive-1.2.1-bin/lib/hive-beeline-1.2.1.jar:/opt/apache-hive-1.2.1-bin/lib/hive-cli-1.2.1.jar:/opt/apache-hive-1.2.1-bin/lib/hive-common-1.2.1.jar:/opt/apache-hive-1.2.1-bin/lib/hive-contrib-1.2.1.jar:/opt/apache-hive-1.2.1-bin/lib/hive-exec-1.2.1.jar:/opt/apache-hive-1.2.1-bin/lib/hive-hbase-handler-1.2.1.jar:/opt/apache-hive-1.2.1-bin/lib/hive-hwi-1.2.1.jar:/opt/apache-hive-1.2.1-bin/lib/hive-jdbc-1.2.1.jar:/opt/apache-hive-1.2.1-bin/lib/hive-jdbc-1.2.1-standalone.jar:/opt/apache-hive-1.2.1-bin/lib/hive-metastore-1.2.1.jar:/opt/apache-hive-1.2.1-bin/lib/hive-serde-1.2.1.jar:/opt/apache-hive-1.2.1-bin/lib/hive-service-1.2.1.jar:/opt/apache-hive-1.2.1-bin/lib/hive-shims-0.20S-1.2.1.jar:/opt/apache-hive-1.2.1-bin/lib/hive-shims-0.23-1.2.1.jar:/opt/apache-hive-1.2.1-bin/lib/hive-shims-1.2.1.jar:/opt/apache-hive-1.2.1-bin/lib/hive-shims-common-1.2.1.jar:/opt/apache-hive-1.2.1-bin/lib/hive-shims-scheduler-1.2.1.jar:/opt/apache-hive-1.2.1-bin/lib/hive-testutils-1.2.1.jar:/opt/apache-hive-1.2.1-bin/lib/httpclient-4.4.jar:/opt/apache-hive-1.2.1-bin/lib/httpcore-4.4.jar:/opt/apache-hive-1.2.1-bin/lib/ivy-2.4.0.jar:/opt/apache-hive-1.2.1-bin/lib/janino-2.7.6.jar:/opt/apache-hive-1.2.1-bin/lib/jcommander-1.32.jar:/opt/apache-hive-1.2.1-bin/lib/jdo-api-3.0.1.jar:/opt/apache-hive-1.2.1-bin/lib/jetty-all-7.6.0.v20120127.jar:/opt/apache-hive-1.2.1-bin/lib/jetty-all-server-7.6.0.v20120127.jar:/opt/apache-hive-1.2.1-bin/lib/jline-2.12.jar:/opt/apache-hive-1.2.1-bin/lib/joda-time-2.5.jar:/opt/apache-hive-1.2.1-bin/lib/jpam-1.1.jar:/opt/apache-hive-1.2.1-bin/lib/json-20090211.jar:/opt/apache-hive-1.2.1-bin/lib/jsr305-3.0.0.jar:/opt/apache-hive-1.2.1-bin/lib/jta-1.1.jar:/opt/apache-hive-1.2.1-bin/lib/junit-4.11.jar:/opt/apache-hive-1.2.1-bin/lib/libfb303-0.9.2.jar:/opt/apache-hive-1.2.1-bin/lib/libthrift-0.9.2.jar:/opt/apache-hive-1.2.1-bin/lib/log4j-1.2.16.jar:/opt/apache-hive-1.2.1-bin/lib/mail-1.4.1.jar:/opt/apache-hive-1.2.1-bin/lib/maven-scm-api-1.4.jar:/opt/apache-hive-1.2.1-bin/lib/maven-scm-provider-svn-commons-1.4.jar:/opt/apache-hive-1.2.1-bin/lib/maven-scm-provider-svnexe-1.4.jar:/opt/apache-hive-1.2.1-bin/lib/mysql-connector-java-5.1.39.jar:/opt/apache-hive-1.2.1-bin/lib/netty-3.7.0.Final.jar:/opt/apache-hive-1.2.1-bin/lib/opencsv-2.3.jar:/opt/apache-hive-1.2.1-bin/lib/oro-2.0.8.jar:/opt/apache-hive-1.2.1-bin/lib/paranamer-2.3.jar:/opt/apache-hive-1.2.1-bin/lib/parquet-hadoop-bundle-1.6.0.jar:/opt/apache-hive-1.2.1-bin/lib/pentaho-aggdesigner-algorithm-5.1.5-jhyde.jar:/opt/apache-hive-1.2.1-bin/lib/plexus-utils-1.5.6.jar:/opt/apache-hive-1.2.1-bin/lib/regexp-1.3.jar:/opt/apache-hive-1.2.1-bin/lib/servlet-api-2.5.jar:/opt/apache-hive-1.2.1-bin/lib/snappy-java-1.0.5.jar:/opt/apache-hive-1.2.1-bin/lib/ST4-4.0.4.jar:/opt/apache-hive-1.2.1-bin/lib/stax-api-1.0.1.jar:/opt/apache-hive-1.2.1-bin/lib/stringtemplate-3.2.1.jar:/opt/apache-hive-1.2.1-bin/lib/super-csv-2.2.0.jar:/opt/apache-hive-1.2.1-bin/lib/tempus-fugit-1.1.jar:/opt/apache-hive-1.2.1-bin/lib/velocity-1.5.jar:/opt/apache-hive-1.2.1-bin/lib/xz-1.0.jar:/opt/apache-hive-1.2.1-bin/lib/zookeeper-3.4.6.jar::/opt/hadoop-2.6.2//contrib/capacity-scheduler/*.jar
16/12/20 22:16:02 [main]: INFO zookeeper.ZooKeeper: Client environment:java.library.path=/opt/hadoop-2.6.2//lib/native:/opt/hadoop-2.6.2/lib/native
16/12/20 22:16:02 [main]: INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
16/12/20 22:16:02 [main]: INFO zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
16/12/20 22:16:02 [main]: INFO zookeeper.ZooKeeper: Client environment:os.name=Linux
16/12/20 22:16:02 [main]: INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64
16/12/20 22:16:02 [main]: INFO zookeeper.ZooKeeper: Client environment:os.version=2.6.32-573.el6.x86_64
16/12/20 22:16:02 [main]: INFO zookeeper.ZooKeeper: Client environment:user.name=hadoop
16/12/20 22:16:02 [main]: INFO zookeeper.ZooKeeper: Client environment:user.home=/home/hadoop
16/12/20 22:16:02 [main]: INFO zookeeper.ZooKeeper: Client environment:user.dir=/home/hadoop
16/12/20 22:16:02 [main]: INFO zookeeper.ZooKeeper: Initiating client connection, connectString=qxy1:2181,qxy2:2181,qxy3:2181 sessionTimeout=1200000 watcher=org.apache.curator.ConnectionState@7d904ff1
16/12/20 22:16:02 [main-SendThread(qxy2:2181)]: INFO zookeeper.ClientCnxn: Opening socket connection to server qxy2/192.168.233.160:2181. Will not attempt to authenticate using SASL (unknown error)
16/12/20 22:16:02 [main-SendThread(qxy2:2181)]: INFO zookeeper.ClientCnxn: Socket connection established to qxy2/192.168.233.160:2181, initiating session
16/12/20 22:16:02 [main-SendThread(qxy2:2181)]: INFO zookeeper.ClientCnxn: Session establishment complete on server qxy2/192.168.233.160:2181, sessionid = 0x2591f661ea30000, negotiated timeout = 40000
16/12/20 22:16:02 [main-EventThread]: INFO state.ConnectionStateManager: State change: CONNECTED
16/12/20 22:16:02 [main]: INFO log.PerfLogger: </PERFLOG method=acquireReadWriteLocks start=1482300962528 end=1482300962777 duration=249 from=org.apache.hadoop.hive.ql.Driver>
16/12/20 22:16:02 [main]: INFO log.PerfLogger: <PERFLOG method=Driver.execute from=org.apache.hadoop.hive.ql.Driver>
16/12/20 22:16:02 [main]: INFO ql.Driver: Starting command(queryId=hadoop_20161220221600_6290d359-2941-4748-84f6-75f6fc444d71):
select a2.ISSUE_CODE as ISSUE_CODE,
a2.FZQDM as FZQDM,
a2.FZQLB as FZQLB,
a2.FJJDM as FJJDM,
a3.FSETCODE as FSETCODE,
a3.FSETID as FSETID,
a2.SRSCD as SRSCD
from (select t1.FSCDM as ISSUE_CODE,--市场代码
t1.FZQDM as FZQDM,
(case when instr(t1.FZQLB, '非银行间') > 0
then '非银行间'
else '银行间'
end) as FZQLB,
t1.FJJDM as FJJDM,
t1.SRSCD as SRSCD
from (select
a1.FZQDM as FZQDM,
a1.FZQLB as FZQLB,
a1.FJJDM as FJJDM,
a1.SRSCD as SRSCD,
a1.FSCDM as FSCDM,
row_number() over(partition by a1.FZQDM,a1.SRSCD order by length(trim(a1.FSCDM)) desc,a1.FSCDM desc) sem
from TMP.CS_DWM_ISSU_SRC2STD_REL_H_05 a1
where a1.FJJDM is not null or a1.FJJDM = ' ') t1
where t1.sem=1
) a2
left join (select distinct concat('A',lpad(t.FSETCODE,3,0)) as FSETCODE, --基金代码
t.FSETID as FSETID, --套账号
t.SRSCD as SRSCD
from TMP.CS_DWM_ISSU_SRC2STD_REL_H_06 t
)a3
on a3.FSETCODE = a2.FJJDM and a2.SRSCD=a3.SRSCD
Query ID = hadoop_20161220221600_6290d359-2941-4748-84f6-75f6fc444d71
16/12/20 22:16:02 [main]: INFO ql.Driver: Query ID = hadoop_20161220221600_6290d359-2941-4748-84f6-75f6fc444d71
Total jobs = 4
16/12/20 22:16:02 [main]: INFO ql.Driver: Total jobs = 4
16/12/20 22:16:02 [main]: INFO log.PerfLogger: </PERFLOG method=TimeToSubmit start=1482300960067 end=1482300962800 duration=2733 from=org.apache.hadoop.hive.ql.Driver>
16/12/20 22:16:02 [main]: INFO log.PerfLogger: <PERFLOG method=runTasks from=org.apache.hadoop.hive.ql.Driver>
16/12/20 22:16:02 [main]: INFO log.PerfLogger: <PERFLOG method=task.MAPRED.Stage-1 from=org.apache.hadoop.hive.ql.Driver>
Launching Job 1 out of 4
16/12/20 22:16:02 [main]: INFO ql.Driver: Launching Job 1 out of 4
16/12/20 22:16:02 [main]: INFO ql.Driver: Starting task [Stage-1:MAPRED] in serial mode
16/12/20 22:16:02 [main]: INFO log.PerfLogger: <PERFLOG method=getInputSummary from=org.apache.hadoop.hive.ql.exec.Utilities>
16/12/20 22:16:02 [main]: INFO lzo.GPLNativeCodeLoader: Loaded native gpl library from the embedded binaries
16/12/20 22:16:02 [main]: INFO lzo.LzoCodec: Successfully loaded & initialized native-lzo library [hadoop-lzo rev b67c96f080d1ca98525a61000186eaf3fba959d3]
16/12/20 22:16:02 [main]: INFO exec.Utilities: Cache Content Summary for hdfs://qxy1/user/hive/warehouse/tmp.db/cs_dwm_issu_src2std_rel_h_05 length: 2016043 file count: 1 directory count: 1
16/12/20 22:16:02 [main]: INFO log.PerfLogger: </PERFLOG method=getInputSummary start=1482300962817 end=1482300962907 duration=90 from=org.apache.hadoop.hive.ql.exec.Utilities>
16/12/20 22:16:02 [main]: INFO exec.Utilities: BytesPerReducer=256000000 maxReducers=1009 totalInputFileSize=2016043
Number of reduce tasks not specified. Estimated from input data size: 1
16/12/20 22:16:02 [main]: INFO exec.Task: Number of reduce tasks not specified. Estimated from input data size: 1
In order to change the average load for a reducer (in bytes):
16/12/20 22:16:02 [main]: INFO exec.Task: In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
16/12/20 22:16:02 [main]: INFO exec.Task: set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
16/12/20 22:16:02 [main]: INFO exec.Task: In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
16/12/20 22:16:02 [main]: INFO exec.Task: set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
16/12/20 22:16:02 [main]: INFO exec.Task: In order to set a constant number of reducers:
set mapreduce.job.reduces=<number>
16/12/20 22:16:02 [main]: INFO exec.Task: set mapreduce.job.reduces=<number>
16/12/20 22:16:02 [main]: INFO ql.Context: New scratch dir is hdfs://qxy1/tmp/hive/hadoop/adaaea53-aac6-4dd7-b86c-53daed294f17/hive_2016-12-20_22-16-00_099_8319005040929428210-1
16/12/20 22:16:02 [main]: INFO mr.ExecDriver: Using org.apache.hadoop.hive.ql.io.CombineHiveInputFormat
16/12/20 22:16:02 [main]: INFO exec.Utilities: Processing alias a2:t1:a1
16/12/20 22:16:02 [main]: INFO exec.Utilities: Adding input file hdfs://qxy1/user/hive/warehouse/tmp.db/cs_dwm_issu_src2std_rel_h_05
16/12/20 22:16:02 [main]: INFO exec.Utilities: Content Summary hdfs://qxy1/user/hive/warehouse/tmp.db/cs_dwm_issu_src2std_rel_h_05length: 2016043 num files: 1 num directories: 1
16/12/20 22:16:02 [main]: INFO ql.Context: New scratch dir is hdfs://qxy1/tmp/hive/hadoop/adaaea53-aac6-4dd7-b86c-53daed294f17/hive_2016-12-20_22-16-00_099_8319005040929428210-1
16/12/20 22:16:02 [main]: INFO log.PerfLogger: <PERFLOG method=serializePlan from=org.apache.hadoop.hive.ql.exec.Utilities>
16/12/20 22:16:02 [main]: INFO exec.Utilities: Serializing MapWork via kryo
16/12/20 22:16:03 [main]: INFO log.PerfLogger: </PERFLOG method=serializePlan start=1482300962994 end=1482300963242 duration=248 from=org.apache.hadoop.hive.ql.exec.Utilities>
16/12/20 22:16:03 [main]: INFO Configuration.deprecation: mapred.submit.replication is deprecated. Instead, use mapreduce.client.submit.file.replication
16/12/20 22:16:03 [main]: INFO log.PerfLogger: <PERFLOG method=serializePlan from=org.apache.hadoop.hive.ql.exec.Utilities>
16/12/20 22:16:03 [main]: INFO exec.Utilities: Serializing ReduceWork via kryo
16/12/20 22:16:03 [main]: INFO log.PerfLogger: </PERFLOG method=serializePlan start=1482300963253 end=1482300963292 duration=39 from=org.apache.hadoop.hive.ql.exec.Utilities>
16/12/20 22:16:03 [main]: ERROR mr.ExecDriver: yarn
16/12/20 22:16:03 [main]: INFO client.RMProxy: Connecting to ResourceManager at qxy1/192.168.233.159:8032
16/12/20 22:16:03 [main]: INFO client.RMProxy: Connecting to ResourceManager at qxy1/192.168.233.159:8032
16/12/20 22:16:03 [main]: INFO exec.Utilities: PLAN PATH = hdfs://qxy1/tmp/hive/hadoop/adaaea53-aac6-4dd7-b86c-53daed294f17/hive_2016-12-20_22-16-00_099_8319005040929428210-1/-mr-10008/20552a25-93da-4705-9630-9664220b1407/map.xml
16/12/20 22:16:03 [main]: INFO exec.Utilities: PLAN PATH = hdfs://qxy1/tmp/hive/hadoop/adaaea53-aac6-4dd7-b86c-53daed294f17/hive_2016-12-20_22-16-00_099_8319005040929428210-1/-mr-10008/20552a25-93da-4705-9630-9664220b1407/reduce.xml
16/12/20 22:16:04 [main]: WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
16/12/20 22:16:04 [main]: INFO log.PerfLogger: <PERFLOG method=getSplits from=org.apache.hadoop.hive.ql.io.CombineHiveInputFormat>
16/12/20 22:16:04 [main]: INFO exec.Utilities: PLAN PATH = hdfs://qxy1/tmp/hive/hadoop/adaaea53-aac6-4dd7-b86c-53daed294f17/hive_2016-12-20_22-16-00_099_8319005040929428210-1/-mr-10008/20552a25-93da-4705-9630-9664220b1407/map.xml
16/12/20 22:16:04 [main]: INFO io.CombineHiveInputFormat: Total number of paths: 1, launching 1 threads to check non-combinable ones.
16/12/20 22:16:04 [main]: INFO io.CombineHiveInputFormat: CombineHiveInputSplit creating pool for hdfs://qxy1/user/hive/warehouse/tmp.db/cs_dwm_issu_src2std_rel_h_05; using filter path hdfs://qxy1/user/hive/warehouse/tmp.db/cs_dwm_issu_src2std_rel_h_05
16/12/20 22:16:04 [main]: INFO input.FileInputFormat: Total input paths to process : 1
16/12/20 22:16:04 [main]: INFO input.CombineFileInputFormat: DEBUG: Terminated node allocation with : CompletedNodes: 2, size left: 0
16/12/20 22:16:04 [main]: INFO io.CombineHiveInputFormat: number of splits 1
16/12/20 22:16:04 [main]: INFO io.CombineHiveInputFormat: Number of all splits 1
16/12/20 22:16:04 [main]: INFO log.PerfLogger: </PERFLOG method=getSplits start=1482300964426 end=1482300964483 duration=57 from=org.apache.hadoop.hive.ql.io.CombineHiveInputFormat>
16/12/20 22:16:04 [main]: INFO mapreduce.JobSubmitter: number of splits:1
16/12/20 22:16:04 [main]: INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1482284999685_0020
16/12/20 22:16:05 [main]: INFO impl.YarnClientImpl: Submitted application application_1482284999685_0020
16/12/20 22:16:05 [main]: INFO mapreduce.Job: The url to track the job: http://qxy1:8088/proxy/application_1482284999685_0020/
Starting Job = job_1482284999685_0020, Tracking URL = http://qxy1:8088/proxy/application_1482284999685_0020/
16/12/20 22:16:05 [main]: INFO exec.Task: Starting Job = job_1482284999685_0020, Tracking URL = http://qxy1:8088/proxy/application_1482284999685_0020/
Kill Command = /opt/hadoop-2.6.2//bin/hadoop job -kill job_1482284999685_0020
16/12/20 22:16:05 [main]: INFO exec.Task: Kill Command = /opt/hadoop-2.6.2//bin/hadoop job -kill job_1482284999685_0020
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1
16/12/20 22:16:09 [main]: INFO exec.Task: Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1
16/12/20 22:16:09 [main]: WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
2016-12-20 22:16:09,470 Stage-1 map = 0%, reduce = 0%
16/12/20 22:16:09 [main]: INFO exec.Task: 2016-12-20 22:16:09,470 Stage-1 map = 0%, reduce = 0%
2016-12-20 22:16:15,801 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 2.31 sec
16/12/20 22:16:15 [main]: INFO exec.Task: 2016-12-20 22:16:15,801 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 2.31 sec
2016-12-20 22:16:22,059 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 4.71 sec
16/12/20 22:16:22 [main]: INFO exec.Task: 2016-12-20 22:16:22,059 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 4.71 sec
MapReduce Total cumulative CPU time: 4 seconds 710 msec
16/12/20 22:16:23 [main]: INFO exec.Task: MapReduce Total cumulative CPU time: 4 seconds 710 msec
Ended Job = job_1482284999685_0020
16/12/20 22:16:23 [main]: INFO exec.Task: Ended Job = job_1482284999685_0020
16/12/20 22:16:23 [main]: INFO exec.FileSinkOperator: Moving tmp dir: hdfs://qxy1/tmp/hive/hadoop/adaaea53-aac6-4dd7-b86c-53daed294f17/hive_2016-12-20_22-16-00_099_8319005040929428210-1/_tmp.-mr-10003 to: hdfs://qxy1/tmp/hive/hadoop/adaaea53-aac6-4dd7-b86c-53daed294f17/hive_2016-12-20_22-16-00_099_8319005040929428210-1/-mr-10003
16/12/20 22:16:23 [main]: INFO log.PerfLogger: <PERFLOG method=task.MAPRED.Stage-3 from=org.apache.hadoop.hive.ql.Driver>
Launching Job 2 out of 4
16/12/20 22:16:23 [main]: INFO ql.Driver: Launching Job 2 out of 4
16/12/20 22:16:23 [main]: INFO ql.Driver: Starting task [Stage-3:MAPRED] in serial mode
16/12/20 22:16:23 [main]: INFO log.PerfLogger: <PERFLOG method=getInputSummary from=org.apache.hadoop.hive.ql.exec.Utilities>
16/12/20 22:16:23 [main]: INFO exec.Utilities: Cache Content Summary for hdfs://qxy1/user/hive/warehouse/tmp.db/cs_dwm_issu_src2std_rel_h_06 length: 8533 file count: 1 directory count: 1
16/12/20 22:16:23 [main]: INFO log.PerfLogger: </PERFLOG method=getInputSummary start=1482300983165 end=1482300983175 duration=10 from=org.apache.hadoop.hive.ql.exec.Utilities>
16/12/20 22:16:23 [main]: INFO exec.Utilities: BytesPerReducer=256000000 maxReducers=1009 totalInputFileSize=8533
Number of reduce tasks not specified. Estimated from input data size: 1
16/12/20 22:16:23 [main]: INFO exec.Task: Number of reduce tasks not specified. Estimated from input data size: 1
In order to change the average load for a reducer (in bytes):
16/12/20 22:16:23 [main]: INFO exec.Task: In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
16/12/20 22:16:23 [main]: INFO exec.Task: set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
16/12/20 22:16:23 [main]: INFO exec.Task: In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
16/12/20 22:16:23 [main]: INFO exec.Task: set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
16/12/20 22:16:23 [main]: INFO exec.Task: In order to set a constant number of reducers:
set mapreduce.job.reduces=<number>
16/12/20 22:16:23 [main]: INFO exec.Task: set mapreduce.job.reduces=<number>
16/12/20 22:16:23 [main]: INFO ql.Context: New scratch dir is hdfs://qxy1/tmp/hive/hadoop/adaaea53-aac6-4dd7-b86c-53daed294f17/hive_2016-12-20_22-16-00_099_8319005040929428210-1
16/12/20 22:16:23 [main]: INFO mr.ExecDriver: Using org.apache.hadoop.hive.ql.io.CombineHiveInputFormat
16/12/20 22:16:23 [main]: INFO exec.Utilities: Processing alias a3:t
16/12/20 22:16:23 [main]: INFO exec.Utilities: Adding input file hdfs://qxy1/user/hive/warehouse/tmp.db/cs_dwm_issu_src2std_rel_h_06
16/12/20 22:16:23 [main]: INFO exec.Utilities: Content Summary hdfs://qxy1/user/hive/warehouse/tmp.db/cs_dwm_issu_src2std_rel_h_06length: 8533 num files: 1 num directories: 1
16/12/20 22:16:23 [main]: INFO ql.Context: New scratch dir is hdfs://qxy1/tmp/hive/hadoop/adaaea53-aac6-4dd7-b86c-53daed294f17/hive_2016-12-20_22-16-00_099_8319005040929428210-1
16/12/20 22:16:23 [main]: INFO log.PerfLogger: <PERFLOG method=serializePlan from=org.apache.hadoop.hive.ql.exec.Utilities>
16/12/20 22:16:23 [main]: INFO exec.Utilities: Serializing MapWork via kryo
16/12/20 22:16:23 [main]: INFO log.PerfLogger: </PERFLOG method=serializePlan start=1482300983183 end=1482300983207 duration=24 from=org.apache.hadoop.hive.ql.exec.Utilities>
16/12/20 22:16:23 [main]: INFO log.PerfLogger: <PERFLOG method=serializePlan from=org.apache.hadoop.hive.ql.exec.Utilities>
16/12/20 22:16:23 [main]: INFO exec.Utilities: Serializing ReduceWork via kryo
16/12/20 22:16:23 [main]: INFO log.PerfLogger: </PERFLOG method=serializePlan start=1482300983210 end=1482300983223 duration=13 from=org.apache.hadoop.hive.ql.exec.Utilities>
16/12/20 22:16:23 [main]: ERROR mr.ExecDriver: yarn
16/12/20 22:16:23 [main]: INFO client.RMProxy: Connecting to ResourceManager at qxy1/192.168.233.159:8032
16/12/20 22:16:23 [main]: INFO client.RMProxy: Connecting to ResourceManager at qxy1/192.168.233.159:8032
16/12/20 22:16:23 [main]: INFO exec.Utilities: PLAN PATH = hdfs://qxy1/tmp/hive/hadoop/adaaea53-aac6-4dd7-b86c-53daed294f17/hive_2016-12-20_22-16-00_099_8319005040929428210-1/-mr-10010/d4f8779f-af54-4a23-9cb7-90d74fe31387/map.xml
16/12/20 22:16:23 [main]: INFO exec.Utilities: PLAN PATH = hdfs://qxy1/tmp/hive/hadoop/adaaea53-aac6-4dd7-b86c-53daed294f17/hive_2016-12-20_22-16-00_099_8319005040929428210-1/-mr-10010/d4f8779f-af54-4a23-9cb7-90d74fe31387/reduce.xml
16/12/20 22:16:23 [main]: WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
16/12/20 22:16:23 [main]: INFO log.PerfLogger: <PERFLOG method=getSplits from=org.apache.hadoop.hive.ql.io.CombineHiveInputFormat>
16/12/20 22:16:23 [main]: INFO exec.Utilities: PLAN PATH = hdfs://qxy1/tmp/hive/hadoop/adaaea53-aac6-4dd7-b86c-53daed294f17/hive_2016-12-20_22-16-00_099_8319005040929428210-1/-mr-10010/d4f8779f-af54-4a23-9cb7-90d74fe31387/map.xml
16/12/20 22:16:23 [main]: INFO io.CombineHiveInputFormat: Total number of paths: 1, launching 1 threads to check non-combinable ones.
16/12/20 22:16:23 [main]: INFO io.CombineHiveInputFormat: CombineHiveInputSplit creating pool for hdfs://qxy1/user/hive/warehouse/tmp.db/cs_dwm_issu_src2std_rel_h_06; using filter path hdfs://qxy1/user/hive/warehouse/tmp.db/cs_dwm_issu_src2std_rel_h_06
16/12/20 22:16:23 [main]: INFO input.FileInputFormat: Total input paths to process : 1
16/12/20 22:16:23 [main]: INFO input.CombineFileInputFormat: DEBUG: Terminated node allocation with : CompletedNodes: 2, size left: 0
16/12/20 22:16:23 [main]: INFO io.CombineHiveInputFormat: number of splits 1
16/12/20 22:16:23 [main]: INFO io.CombineHiveInputFormat: Number of all splits 1
16/12/20 22:16:23 [main]: INFO log.PerfLogger: </PERFLOG method=getSplits start=1482300983508 end=1482300983515 duration=7 from=org.apache.hadoop.hive.ql.io.CombineHiveInputFormat>
16/12/20 22:16:23 [main]: INFO mapreduce.JobSubmitter: number of splits:1
16/12/20 22:16:23 [main]: INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1482284999685_0021
16/12/20 22:16:23 [main]: INFO impl.YarnClientImpl: Submitted application application_1482284999685_0021
16/12/20 22:16:23 [main]: INFO mapreduce.Job: The url to track the job: http://qxy1:8088/proxy/application_1482284999685_0021/
Starting Job = job_1482284999685_0021, Tracking URL = http://qxy1:8088/proxy/application_1482284999685_0021/
16/12/20 22:16:23 [main]: INFO exec.Task: Starting Job = job_1482284999685_0021, Tracking URL = http://qxy1:8088/proxy/application_1482284999685_0021/
Kill Command = /opt/hadoop-2.6.2//bin/hadoop job -kill job_1482284999685_0021
16/12/20 22:16:23 [main]: INFO exec.Task: Kill Command = /opt/hadoop-2.6.2//bin/hadoop job -kill job_1482284999685_0021
Hadoop job information for Stage-3: number of mappers: 1; number of reducers: 1
16/12/20 22:16:31 [main]: INFO exec.Task: Hadoop job information for Stage-3: number of mappers: 1; number of reducers: 1
16/12/20 22:16:31 [main]: WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
2016-12-20 22:16:31,935 Stage-3 map = 0%, reduce = 0%
16/12/20 22:16:31 [main]: INFO exec.Task: 2016-12-20 22:16:31,935 Stage-3 map = 0%, reduce = 0%
2016-12-20 22:16:37,134 Stage-3 map = 100%, reduce = 0%, Cumulative CPU 1.79 sec
16/12/20 22:16:37 [main]: INFO exec.Task: 2016-12-20 22:16:37,134 Stage-3 map = 100%, reduce = 0%, Cumulative CPU 1.79 sec
2016-12-20 22:16:43,394 Stage-3 map = 100%, reduce = 100%, Cumulative CPU 3.11 sec
16/12/20 22:16:43 [main]: INFO exec.Task: 2016-12-20 22:16:43,394 Stage-3 map = 100%, reduce = 100%, Cumulative CPU 3.11 sec
MapReduce Total cumulative CPU time: 3 seconds 110 msec
16/12/20 22:16:44 [main]: INFO exec.Task: MapReduce Total cumulative CPU time: 3 seconds 110 msec
Ended Job = job_1482284999685_0021
16/12/20 22:16:44 [main]: INFO exec.Task: Ended Job = job_1482284999685_0021
16/12/20 22:16:44 [main]: INFO exec.FileSinkOperator: Moving tmp dir: hdfs://qxy1/tmp/hive/hadoop/adaaea53-aac6-4dd7-b86c-53daed294f17/hive_2016-12-20_22-16-00_099_8319005040929428210-1/_tmp.-mr-10004 to: hdfs://qxy1/tmp/hive/hadoop/adaaea53-aac6-4dd7-b86c-53daed294f17/hive_2016-12-20_22-16-00_099_8319005040929428210-1/-mr-10004
16/12/20 22:16:44 [main]: INFO log.PerfLogger: <PERFLOG method=task.CONDITION.Stage-5 from=org.apache.hadoop.hive.ql.Driver>
16/12/20 22:16:44 [main]: INFO ql.Driver: Starting task [Stage-5:CONDITIONAL] in serial mode
16/12/20 22:16:44 [main]: INFO plan.ConditionalResolverCommonJoin: Driver alias is [$INTNAME] with size 447899 (total size of others : 18054, threshold : 25000000)
Stage-6 is selected by condition resolver.
16/12/20 22:16:44 [main]: INFO exec.Task: Stage-6 is selected by condition resolver.
Stage-2 is filtered out by condition resolver. /* 此处解释了,Stage-2 被编译器过滤了*/
16/12/20 22:16:44 [main]: INFO exec.Task: Stage-2 is filtered out by condition resolver.
16/12/20 22:16:44 [main]: INFO log.PerfLogger: <PERFLOG method=task.MAPREDLOCAL.Stage-6 from=org.apache.hadoop.hive.ql.Driver>
16/12/20 22:16:44 [main]: INFO ql.Driver: Starting task [Stage-6:MAPREDLOCAL] in serial mode
16/12/20 22:16:44 [main]: INFO mr.MapredLocalTask: Generating plan file file:/home/hadoop/tmpdir/hive/adaaea53-aac6-4dd7-b86c-53daed294f17/hive_2016-12-20_22-16-00_099_8319005040929428210-1/-local-10011/plan.xml
16/12/20 22:16:44 [main]: INFO log.PerfLogger: <PERFLOG method=serializePlan from=org.apache.hadoop.hive.ql.exec.Utilities>
16/12/20 22:16:44 [main]: INFO exec.Utilities: Serializing MapredLocalWork via kryo
16/12/20 22:16:44 [main]: INFO log.PerfLogger: </PERFLOG method=serializePlan start=1482301004506 end=1482301004516 duration=10 from=org.apache.hadoop.hive.ql.exec.Utilities>
16/12/20 22:16:44 [main]: INFO mr.MapredLocalTask: Executing: /opt/hadoop-2.6.2//bin/hadoop jar /opt/apache-hive-1.2.1-bin/lib/hive-exec-1.2.1.jar org.apache.hadoop.hive.ql.exec.mr.ExecDriver -localtask -plan file:/home/hadoop/tmpdir/hive/adaaea53-aac6-4dd7-b86c-53daed294f17/hive_2016-12-20_22-16-00_099_8319005040929428210-1/-local-10011/plan.xml -jobconffile file:/home/hadoop/tmpdir/hive/adaaea53-aac6-4dd7-b86c-53daed294f17/hive_2016-12-20_22-16-00_099_8319005040929428210-1/-local-10012/jobconf.xml
16/12/20 22:16:50 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Execution log at: /tmp/hadoop/hadoop_20161220221600_6290d359-2941-4748-84f6-75f6fc444d71.log
2016-12-20 22:16:55 Starting to launch local task to process map join; maximum memory = 477626368
2016-12-20 22:16:57 Dump the side-table for tag: 1 with group count: 64 into file: file:/home/hadoop/tmpdir/hive/adaaea53-aac6-4dd7-b86c-53daed294f17/hive_2016-12-20_22-16-00_099_8319005040929428210-1/-local-10005/HashTable-Stage-4/MapJoin-mapfile01--.hashtable
2016-12-20 22:16:57 Uploaded 1 File to: file:/home/hadoop/tmpdir/hive/adaaea53-aac6-4dd7-b86c-53daed294f17/hive_2016-12-20_22-16-00_099_8319005040929428210-1/-local-10005/HashTable-Stage-4/MapJoin-mapfile01--.hashtable (8059 bytes)
2016-12-20 22:16:57 End of local task; Time Taken: 2.836 sec.
Execution completed successfully
16/12/20 22:16:58 [main]: INFO exec.Task: Execution completed successfully
MapredLocal task succeeded
16/12/20 22:16:58 [main]: INFO exec.Task: MapredLocal task succeeded
16/12/20 22:16:58 [main]: INFO mr.MapredLocalTask: Execution completed successfully
16/12/20 22:16:58 [main]: INFO log.PerfLogger: <PERFLOG method=task.MAPRED.Stage-4 from=org.apache.hadoop.hive.ql.Driver>
Launching Job 4 out of 4
16/12/20 22:16:58 [main]: INFO ql.Driver: Launching Job 4 out of 4
16/12/20 22:16:58 [main]: INFO ql.Driver: Starting task [Stage-4:MAPRED] in serial mode
Number of reduce tasks is set to 0 since there's no reduce operator
16/12/20 22:16:58 [main]: INFO exec.Task: Number of reduce tasks is set to 0 since there's no reduce operator
16/12/20 22:16:58 [main]: INFO ql.Context: New scratch dir is hdfs://qxy1/tmp/hive/hadoop/adaaea53-aac6-4dd7-b86c-53daed294f17/hive_2016-12-20_22-16-00_099_8319005040929428210-1
16/12/20 22:16:58 [main]: INFO mr.ExecDriver: Using org.apache.hadoop.hive.ql.io.CombineHiveInputFormat
16/12/20 22:16:58 [main]: INFO mr.ExecDriver: Archive 1 hash table files to file:/home/hadoop/tmpdir/hive/adaaea53-aac6-4dd7-b86c-53daed294f17/hive_2016-12-20_22-16-00_099_8319005040929428210-1/-local-10005/HashTable-Stage-4/Stage-4.tar.gz
16/12/20 22:16:58 [main]: INFO mr.ExecDriver: Upload 1 archive file fromfile:/home/hadoop/tmpdir/hive/adaaea53-aac6-4dd7-b86c-53daed294f17/hive_2016-12-20_22-16-00_099_8319005040929428210-1/-local-10005/HashTable-Stage-4/Stage-4.tar.gz to: hdfs://qxy1/tmp/hive/hadoop/adaaea53-aac6-4dd7-b86c-53daed294f17/hive_2016-12-20_22-16-00_099_8319005040929428210-1/-mr-10006/HashTable-Stage-4/Stage-4.tar.gz
16/12/20 22:16:58 [main]: INFO mr.ExecDriver: Add 1 archive file to distributed cache. Archive file: hdfs://qxy1/tmp/hive/hadoop/adaaea53-aac6-4dd7-b86c-53daed294f17/hive_2016-12-20_22-16-00_099_8319005040929428210-1/-mr-10006/HashTable-Stage-4/Stage-4.tar.gz
16/12/20 22:16:58 [main]: INFO exec.Utilities: Processing alias $INTNAME
16/12/20 22:16:58 [main]: INFO exec.Utilities: Adding input file hdfs://qxy1/tmp/hive/hadoop/adaaea53-aac6-4dd7-b86c-53daed294f17/hive_2016-12-20_22-16-00_099_8319005040929428210-1/-mr-10003
16/12/20 22:16:58 [main]: INFO exec.Utilities: Content Summary not cached for hdfs://qxy1/tmp/hive/hadoop/adaaea53-aac6-4dd7-b86c-53daed294f17/hive_2016-12-20_22-16-00_099_8319005040929428210-1/-mr-10003
16/12/20 22:16:58 [main]: INFO ql.Context: New scratch dir is hdfs://qxy1/tmp/hive/hadoop/adaaea53-aac6-4dd7-b86c-53daed294f17/hive_2016-12-20_22-16-00_099_8319005040929428210-1
16/12/20 22:16:58 [main]: INFO log.PerfLogger: <PERFLOG method=serializePlan from=org.apache.hadoop.hive.ql.exec.Utilities>
16/12/20 22:16:58 [main]: INFO exec.Utilities: Serializing MapWork via kryo
16/12/20 22:16:58 [main]: INFO log.PerfLogger: </PERFLOG method=serializePlan start=1482301018690 end=1482301018717 duration=27 from=org.apache.hadoop.hive.ql.exec.Utilities>
16/12/20 22:16:58 [main]: ERROR mr.ExecDriver: yarn
16/12/20 22:16:58 [main]: INFO client.RMProxy: Connecting to ResourceManager at qxy1/192.168.233.159:8032
16/12/20 22:16:58 [main]: INFO client.RMProxy: Connecting to ResourceManager at qxy1/192.168.233.159:8032
16/12/20 22:16:58 [main]: INFO exec.Utilities: PLAN PATH = hdfs://qxy1/tmp/hive/hadoop/adaaea53-aac6-4dd7-b86c-53daed294f17/hive_2016-12-20_22-16-00_099_8319005040929428210-1/-mr-10014/dd707887-351d-4106-a1c0-fbcee0a27432/map.xml
16/12/20 22:16:58 [main]: INFO exec.Utilities: PLAN PATH = hdfs://qxy1/tmp/hive/hadoop/adaaea53-aac6-4dd7-b86c-53daed294f17/hive_2016-12-20_22-16-00_099_8319005040929428210-1/-mr-10014/dd707887-351d-4106-a1c0-fbcee0a27432/reduce.xml
16/12/20 22:16:58 [main]: INFO exec.Utilities: ***************non-local mode***************
16/12/20 22:16:58 [main]: INFO exec.Utilities: local path = hdfs://qxy1/tmp/hive/hadoop/adaaea53-aac6-4dd7-b86c-53daed294f17/hive_2016-12-20_22-16-00_099_8319005040929428210-1/-mr-10014/dd707887-351d-4106-a1c0-fbcee0a27432/reduce.xml
16/12/20 22:16:58 [main]: INFO exec.Utilities: Open file to read in plan: hdfs://qxy1/tmp/hive/hadoop/adaaea53-aac6-4dd7-b86c-53daed294f17/hive_2016-12-20_22-16-00_099_8319005040929428210-1/-mr-10014/dd707887-351d-4106-a1c0-fbcee0a27432/reduce.xml
16/12/20 22:16:58 [main]: INFO exec.Utilities: File not found: File does not exist: /tmp/hive/hadoop/adaaea53-aac6-4dd7-b86c-53daed294f17/hive_2016-12-20_22-16-00_099_8319005040929428210-1/-mr-10014/dd707887-351d-4106-a1c0-fbcee0a27432/reduce.xml
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66)
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1893)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1834)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1814)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1786)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:552)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:362)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2040)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2036)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2034) 16/12/20 22:16:58 [main]: INFO exec.Utilities: No plan file found: hdfs://qxy1/tmp/hive/hadoop/adaaea53-aac6-4dd7-b86c-53daed294f17/hive_2016-12-20_22-16-00_099_8319005040929428210-1/-mr-10014/dd707887-351d-4106-a1c0-fbcee0a27432/reduce.xml
16/12/20 22:16:58 [main]: WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
16/12/20 22:16:59 [main]: INFO log.PerfLogger: <PERFLOG method=getSplits from=org.apache.hadoop.hive.ql.io.CombineHiveInputFormat>
16/12/20 22:16:59 [main]: INFO exec.Utilities: PLAN PATH = hdfs://qxy1/tmp/hive/hadoop/adaaea53-aac6-4dd7-b86c-53daed294f17/hive_2016-12-20_22-16-00_099_8319005040929428210-1/-mr-10014/dd707887-351d-4106-a1c0-fbcee0a27432/map.xml
16/12/20 22:16:59 [main]: INFO io.CombineHiveInputFormat: Total number of paths: 1, launching 1 threads to check non-combinable ones.
16/12/20 22:16:59 [main]: INFO io.CombineHiveInputFormat: CombineHiveInputSplit creating pool for hdfs://qxy1/tmp/hive/hadoop/adaaea53-aac6-4dd7-b86c-53daed294f17/hive_2016-12-20_22-16-00_099_8319005040929428210-1/-mr-10003; using filter path hdfs://qxy1/tmp/hive/hadoop/adaaea53-aac6-4dd7-b86c-53daed294f17/hive_2016-12-20_22-16-00_099_8319005040929428210-1/-mr-10003
16/12/20 22:16:59 [main]: INFO input.FileInputFormat: Total input paths to process : 1
16/12/20 22:16:59 [main]: INFO input.CombineFileInputFormat: DEBUG: Terminated node allocation with : CompletedNodes: 2, size left: 0
16/12/20 22:16:59 [main]: INFO io.CombineHiveInputFormat: number of splits 1
16/12/20 22:16:59 [main]: INFO io.CombineHiveInputFormat: Number of all splits 1
16/12/20 22:16:59 [main]: INFO log.PerfLogger: </PERFLOG method=getSplits start=1482301019140 end=1482301019148 duration=8 from=org.apache.hadoop.hive.ql.io.CombineHiveInputFormat>
16/12/20 22:16:59 [main]: INFO mapreduce.JobSubmitter: number of splits:1
16/12/20 22:16:59 [main]: INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1482284999685_0022
16/12/20 22:16:59 [main]: INFO impl.YarnClientImpl: Submitted application application_1482284999685_0022
16/12/20 22:16:59 [main]: INFO mapreduce.Job: The url to track the job: http://qxy1:8088/proxy/application_1482284999685_0022/
Starting Job = job_1482284999685_0022, Tracking URL = http://qxy1:8088/proxy/application_1482284999685_0022/
16/12/20 22:16:59 [main]: INFO exec.Task: Starting Job = job_1482284999685_0022, Tracking URL = http://qxy1:8088/proxy/application_1482284999685_0022/
Kill Command = /opt/hadoop-2.6.2//bin/hadoop job -kill job_1482284999685_0022
16/12/20 22:16:59 [main]: INFO exec.Task: Kill Command = /opt/hadoop-2.6.2//bin/hadoop job -kill job_1482284999685_0022
Hadoop job information for Stage-4: number of mappers: 1; number of reducers: 0
16/12/20 22:17:03 [main]: INFO exec.Task: Hadoop job information for Stage-4: number of mappers: 1; number of reducers: 0
16/12/20 22:17:03 [main]: WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
2016-12-20 22:17:03,473 Stage-4 map = 0%, reduce = 0%
16/12/20 22:17:03 [main]: INFO exec.Task: 2016-12-20 22:17:03,473 Stage-4 map = 0%, reduce = 0%
2016-12-20 22:17:09,752 Stage-4 map = 100%, reduce = 0%, Cumulative CPU 1.57 sec
16/12/20 22:17:09 [main]: INFO exec.Task: 2016-12-20 22:17:09,752 Stage-4 map = 100%, reduce = 0%, Cumulative CPU 1.57 sec
MapReduce Total cumulative CPU time: 1 seconds 570 msec
16/12/20 22:17:10 [main]: INFO exec.Task: MapReduce Total cumulative CPU time: 1 seconds 570 msec
Ended Job = job_1482284999685_0022
16/12/20 22:17:10 [main]: INFO exec.Task: Ended Job = job_1482284999685_0022
16/12/20 22:17:10 [main]: INFO exec.FileSinkOperator: Moving tmp dir: hdfs://qxy1/tmp/hive/hadoop/adaaea53-aac6-4dd7-b86c-53daed294f17/hive_2016-12-20_22-16-00_099_8319005040929428210-1/-mr-10000/.hive-staging_hive_2016-12-20_22-16-00_099_8319005040929428210-1/_tmp.-ext-10001 to: hdfs://qxy1/tmp/hive/hadoop/adaaea53-aac6-4dd7-b86c-53daed294f17/hive_2016-12-20_22-16-00_099_8319005040929428210-1/-mr-10000/.hive-staging_hive_2016-12-20_22-16-00_099_8319005040929428210-1/-ext-10001
16/12/20 22:17:10 [main]: INFO log.PerfLogger: </PERFLOG method=runTasks start=1482300962800 end=1482301030833 duration=68033 from=org.apache.hadoop.hive.ql.Driver>
16/12/20 22:17:10 [main]: INFO log.PerfLogger: </PERFLOG method=Driver.execute start=1482300962778 end=1482301030834 duration=68056 from=org.apache.hadoop.hive.ql.Driver>
MapReduce Jobs Launched:
16/12/20 22:17:10 [main]: INFO ql.Driver: MapReduce Jobs Launched:
Stage-Stage-1: Map: 1 Reduce: 1 Cumulative CPU: 4.71 sec HDFS Read: 2027044 HDFS Write: 447899 SUCCESS
16/12/20 22:17:10 [main]: INFO ql.Driver: Stage-Stage-1: Map: 1 Reduce: 1 Cumulative CPU: 4.71 sec HDFS Read: 2027044 HDFS Write: 447899 SUCCESS
Stage-Stage-3: Map: 1 Reduce: 1 Cumulative CPU: 3.11 sec HDFS Read: 16290 HDFS Write: 18054 SUCCESS
16/12/20 22:17:10 [main]: INFO ql.Driver: Stage-Stage-3: Map: 1 Reduce: 1 Cumulative CPU: 3.11 sec HDFS Read: 16290 HDFS Write: 18054 SUCCESS
Stage-Stage-4: Map: 1 Cumulative CPU: 1.57 sec HDFS Read: 453508 HDFS Write: 432499 SUCCESS
16/12/20 22:17:10 [main]: INFO ql.Driver: Stage-Stage-4: Map: 1 Cumulative CPU: 1.57 sec HDFS Read: 453508 HDFS Write: 432499 SUCCESS
Total MapReduce CPU Time Spent: 9 seconds 390 msec
16/12/20 22:17:10 [main]: INFO ql.Driver: Total MapReduce CPU Time Spent: 9 seconds 390 msec
OK
16/12/20 22:17:10 [main]: INFO ql.Driver: OK
16/12/20 22:17:10 [main]: INFO log.PerfLogger: <PERFLOG method=releaseLocks from=org.apache.hadoop.hive.ql.Driver>
16/12/20 22:17:10 [main]: INFO ZooKeeperHiveLockManager: about to release lock for tmp/cs_dwm_issu_src2std_rel_h_06
16/12/20 22:17:10 [main]: INFO ZooKeeperHiveLockManager: about to release lock for tmp/cs_dwm_issu_src2std_rel_h_05
16/12/20 22:17:10 [main]: INFO ZooKeeperHiveLockManager: about to release lock for tmp
16/12/20 22:17:10 [main]: INFO ZooKeeperHiveLockManager: about to release lock for default
16/12/20 22:17:10 [main]: INFO log.PerfLogger: </PERFLOG method=releaseLocks start=1482301030835 end=1482301030908 duration=73 from=org.apache.hadoop.hive.ql.Driver>
16/12/20 22:17:10 [main]: INFO log.PerfLogger: </PERFLOG method=Driver.run start=1482300960066 end=1482301030908 duration=70842 from=org.apache.hadoop.hive.ql.Driver>
16/12/20 22:17:10 [main]: INFO Configuration.deprecation: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
16/12/20 22:17:10 [main]: INFO mapred.FileInputFormat: Total input paths to process : 1 16/12/20 22:17:11 [main]: INFO exec.ListSinkOperator: 36 finished. closing...
16/12/20 22:17:11 [main]: INFO exec.ListSinkOperator: 36 Close done
Time taken: 70.845 seconds, Fetched: 8990 row(s)
16/12/20 22:17:11 [main]: INFO CliDriver: Time taken: 70.845 seconds, Fetched: 8990 row(s)
16/12/20 22:17:11 [main]: INFO log.PerfLogger: <PERFLOG method=releaseLocks from=org.apache.hadoop.hive.ql.Driver>
16/12/20 22:17:11 [main]: INFO log.PerfLogger: </PERFLOG method=releaseLocks start=1482301031156 end=1482301031156 duration=0 from=org.apache.hadoop.hive.ql.Driver>
16/12/20 22:17:11 [main-EventThread]: INFO zookeeper.ClientCnxn: EventThread shut down
16/12/20 22:17:11 [Thread-8]: INFO zookeeper.ZooKeeper: Session: 0x2591f661ea30000 closed
16/12/20 22:17:11 [Thread-8]: INFO CuratorFrameworkSingleton: Closing ZooKeeper client.
所以，执行计划的顺序为:
Stage-1->Stage-3->Stage-5{Stage-6,Stage-2}->Stage-6->Stage-4->Stage-6