相比于Map-Reduce,Hive对数据的处理相对简单,但是Hive本身提供的函数,对于处理复杂的字符串问题,就显得不是很方便,此时,可以借助transform,引入外界的Python程序对字符串进行处理。

transform

transform的基本用法为:

transform的基本语法为:

select transform(intput columns)
using 'python *.py'
as (output columns)

实例

假设目前我们有如下的一些数据:

Hive——巧用transform处理复杂的字符串问题-LMLPHP

需要取出以分号“;”分隔的倒数第二位。实际的代码如下:

  • Hive的代码:
function create_table(){
    sql_create_table_1="drop table if exists ${table_name_deal};
    create table if not exists ${table_name_deal}(
    deal string
    )
    row format delimited fields terminated by '\t'
    lines terminated by '\n'
    stored as rcfile
    location '${table_path}/${table_name_deal}';"
    hive -e"${sql_create_table_1}"

}

function data_deal(){
    deal_sql="add file deal.py;
    insert overwrite table ${table_name_deal}
    select a.deal
    from
    (select transform(match_id)
                using 'python deal.py'
                as (deal)
                from ${table_name_sel}
            ) a;"

    hive -e"${deal_sql}"
}
  • python脚本
#!/usr/bin/python
#coding:UTF-8

import sys

for line in sys.stdin:
    lines = line.strip().split(";")
    if len(lines) < 10:
        continue

    deal = lines[-2]
    print deal
05-11 21:42