如何在来自列的 Pig 中解码 JSON?

问题描述

有许多示例说明如何使用 JsonLoader() 从文件中加载带有架构的 JSON 数据，而不是从任何类型的其他输出中加载.

There are numerous examples of how to use JsonLoader() to load JSON data with a schema from a file, but not from any sort of other output.

推荐答案

您正在寻找 Elephant Bird 中提供的 JsonStringToMap UDF:https://github.com/kevinweil/elephant-bird/search?q=JsonStringToMap&ref=cmdform

You are looking for the JsonStringToMap UDF provided in Elephant Bird: https://github.com/kevinweil/elephant-bird/search?q=JsonStringToMap&ref=cmdform

示例文件:

foo     bar     {"version":1, "type":"an event", "count": 1}
foo     bar     {"version":1, "type":"another event", "count": 1}

猪脚本:

REGISTER /path/to/elephant-bird.jar;
DEFINE JsonStringToMap com.twitter.elephantbird.pig.piggybank.JsonStringToMap();
raw = LOAD '/tmp/file.tsv' USING PigStorage('\t') AS (col1:chararray,col2:chararray,json_string:chararray);
parsed = FOREACH raw GENERATE col1,col2,JsonStringToMap(json_string);
ILLUSTRATE parsed; -- Just to show the output

预处理(JSON 作为字符数组/字符串):

Pre-processing (JSON as chararray/string):

-------------------------------------------------------------------------------------------------------
| raw     | col1:chararray    | col2:chararray    | json_string:chararray                             |
-------------------------------------------------------------------------------------------------------
|         | foo               | bar               | {"version":1, "type":"another event", "count": 1} |

后处理(JSON 作为地图):

Post-processing (JSON as map):

-------------------------------------------------------------------------------------------------
| parsed     | col1:chararray    | col2:chararray    | json:map(:chararray)                     |
-------------------------------------------------------------------------------------------------
|            | foo               | bar               | {count=1, type=another event, version=1} |
-------------------------------------------------------------------------------------------------

这个问题与How使用 Pig

这篇关于如何在来自列的 Pig 中解码 JSON?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！