相比于Map-Reduce,Hive对数据的处理相对简单,但是Hive本身提供的函数,对于处理复杂的字符串问题,就显得不是很方便,此时,可以借助transform,引入外界的Python程序对字符串进行处理。
transform
transform的基本用法为:
transform的基本语法为:
select transform(intput columns)
using 'python *.py'
as (output columns)
实例
假设目前我们有如下的一些数据:
需要取出以分号“;”分隔的倒数第二位。实际的代码如下:
- Hive的代码:
function create_table(){
sql_create_table_1="drop table if exists ${table_name_deal};
create table if not exists ${table_name_deal}(
deal string
)
row format delimited fields terminated by '\t'
lines terminated by '\n'
stored as rcfile
location '${table_path}/${table_name_deal}';"
hive -e"${sql_create_table_1}"
}
function data_deal(){
deal_sql="add file deal.py;
insert overwrite table ${table_name_deal}
select a.deal
from
(select transform(match_id)
using 'python deal.py'
as (deal)
from ${table_name_sel}
) a;"
hive -e"${deal_sql}"
}
- python脚本
#!/usr/bin/python
#coding:UTF-8
import sys
for line in sys.stdin:
lines = line.strip().split(";")
if len(lines) < 10:
continue
deal = lines[-2]
print deal