将数据导入hive，再将hive表导入hbase

将数据到入hive的无分区表，再将无分区表导入hive的有分区表：

--备份

create table tds_package_secinfobk as select * from tds_package_secinfo;

--导入分区表

tds_package_secinfo

CREATE TABLE  tds_package_secinfo

(package_name string,program_name string,keyhash string)

PARTITIONED BY(risk_label string)

ROW FORMAT delimited fields terminated by '\t'

LOCATION '/user/hive/warehouse/tds_package_secinfo'

LOAD DATA INPATH '/user/hive/warehouse/邪教app.txt' OVERWRITE INTO TABLE tds_package_secinfo PARTITION(risk_label='邪教app');

--无分区表导有分区表

set hive.exec.max.dynamic.partitions= ;

set hive.exec.max.dynamic.partitions.pernode= ;

set hive.exec.dynamic.partition.mode=nonstrict;

set hive.support.concurrency=false;

set mapreduce.job.reduce.slowstart.completedmaps=;

insert into table tds_package_secinfo partition(risk_label)

select  package_name ,'' ,'' ,risk_label from  hue_package_secinfo;

hbase_package_secinfo

查询该hive表的包名和标签列：

select package_name,risk_label from tds_package_secinfo

hbase表需要先建表，才能插入数据

建表：

CREATE TABLE hbase_package_secinfo

(package_name string,risk_label string)

STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'

WITH SERDEPROPERTIES

("hbase.columns.mapping" = ":key,info:risk_lable") TBLPROPERTIES ("hbase.table.name"="hbase_package_secinfo");

插入数据：

insert into hbase_package_secinfo

select package_name,risk_label from tds_package_secinfo 

-------------------------------------------------建表完成-----------------------------------------------------------

查询语句：

select count(*),dt from MDS_ENGINE_VIR_URL group by dt order by dt desc;

select count(*),dt from MDS_ENGINE_WARESOURCE_INFO_NEW group by dt order by dt desc;

select count(*),dt from MDS_VICTIM_INFO group by dt order by dt desc;   

--------------------------------------------

异常：

Error: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"package_name":"","program_name":"","keyhash":"","risk_label":"__HIVE_DEFAULT_PARTITION__"}

    at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:)

    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:)

    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:)

    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:)

    at org.apache.hadoop.mapred.YarnChild$.run(YarnChild.java:)

    at java.security.AccessController.doPrivileged(Native Method)

    at javax.security.auth.Subject.doAs(Subject.java:)

    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:)

    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:)

Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"package_name":"","program_name":"","keyhash":"","risk_label":"__HIVE_DEFAULT_PARTITION__"}

    at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:)

    at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:)

    ...  more

Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.hadoop.hive.serde2.SerDeException: java.lang.IllegalArgumentException: Row length is

    at org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:)

    at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:)

    at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:)

    at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:)

    at org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:)

    at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:)

    at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:)

    ...  more

Caused by: org.apache.hadoop.hive.serde2.SerDeException: java.lang.IllegalArgumentException: Row length is

    at org.apache.hadoop.hive.hbase.HBaseSerDe.serialize(HBaseSerDe.java:)

    at org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:)

    ...  more

Caused by: java.lang.IllegalArgumentException: Row length is

    at org.apache.hadoop.hbase.client.Mutation.checkRow(Mutation.java:)

    at org.apache.hadoop.hbase.client.Put.<init>(Put.java:)

    at org.apache.hadoop.hbase.client.Put.<init>(Put.java:)

    at org.apache.hadoop.hbase.client.Put.<init>(Put.java:)

    at org.apache.hadoop.hive.hbase.HBaseRowSerializer.serialize(HBaseRowSerializer.java:)

    at org.apache.hadoop.hive.hbase.HBaseSerDe.serialize(HBaseSerDe.java:)

    ...  more

原因：表中的内容为空，所以插不进去

解决方法：加上条件：

where risk_label!='__HIVE_DEFAULT_PARTITION__'

所以插入语句应该为：

insert into hbase_package_secinfo

select package_name,risk_label from tds_package_secinfo where risk_label!='__HIVE_DEFAULT_PARTITION__'; 

最后删除risk_label":"__HIVE_DEFAULT_PARTITION__的那个分区

ALTER TABLE tds_package_secinfo DROP IF EXISTS PARTITION(risk_label='__HIVE_DEFAULT_PARTITION__' );