问题描述
标题有点奇怪,因为我很难缩小问题的范围.我在 Hadoop 2.0.0-cdh4.4.0 和 hive 0.10 上使用了我的解决方案,没有问题.
The title is a bit weird as I'm having difficulties narrowing down the problem. I used my solution on Hadoop 2.0.0-cdh4.4.0 and hive 0.10 without issues.
- 我无法使用此 SerDe 创建表:https://github.com/rcongiu/Hive-JSON-Serde
第一次尝试:
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. org.apache.hadoop.hive.serde2.objectinspector.primitive.AbstractPrimitiveJavaObjectInspector.<init>(Lorg/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorUtils$PrimitiveTypeEntry;)V
第二次尝试:
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. Could not initialize class org.openx.data.jsonserde.objectinspector.JsonObjectInspectorFactory
我可以用这个 SerDe 创建一个表:https://github.com/cloudera/cdh-twitter-example
我用来自flume的推文创建了一个外部表.我不能做SELECT * FROM tweets;"
I create an external table with tweets from flume. I can't do "SELECT * FROM tweets;"
失败:运行时异常 org.apache.hadoop.hive.ql.metadata.HiveException:因异常 java.lang.ClassNotFoundException 失败:com.cloudera.hive.serde.JSONSerDejava.lang.RuntimeException:java.lang.ClassNotFoundException:com.cloudera.hive.serde.JSONSerDe
FAILED: RuntimeException org.apache.hadoop.hive.ql.metadata.HiveException: Failed with exception java.lang.ClassNotFoundException: com.cloudera.hive.serde.JSONSerDejava.lang.RuntimeException: java.lang.ClassNotFoundException: com.cloudera.hive.serde.JSONSerDe
我可以做 SELECT id, text FROM tweets;
I can do SELECT id, text FROM tweets;
我无法自行加入此表:
执行日志位于:/tmp/jochen.debie/jochen.debie_20140311121313_164611a9-b0d8-4e53-9bda-f9f7ac342aaf.log2014-03-11 12:13:30 开始启动本地任务处理地图加入;最大内存 = 257294336执行失败,退出状态:2获取错误信息
Execution log at: /tmp/jochen.debie/jochen.debie_20140311121313_164611a9-b0d8-4e53-9bda-f9f7ac342aaf.log2014-03-11 12:13:30 Starting to launch local task to process map join; maximum memory = 257294336Execution failed with exit status: 2Obtaining error information
任务失败!任务编号:第五阶段
Task failed!Task ID: Stage-5
提到的执行日志:
2014-03-11 12:13:30,331 ERROR mr.MapredLocalTask (MapredLocalTask.java:executeFromChildJVM(324)) - Hive Runtime Error: Map local work failed org.apache.hadoop.hive.ql.metadata.HiveException: Failed with exception java.lang.ClassNotFoundException: com.cloudera.hive.serde.JSONSerDejava.lang.RuntimeException: java.lang.ClassNotFoundException: com.cloudera.hive.serde.JSONSerDe
有谁知道如何解决这个问题,或者至少告诉我问题出在哪里?
Does anyone know how to fix this or at least show me where the problem is?
我在 Hadoop 2.0.0-cdh4.4.0 和 hive 0.10 上构建 serde 会不会有问题?
Can it be a problem that I built the serde on a Hadoop 2.0.0-cdh4.4.0 and hive 0.10?
推荐答案
据我所知,Hive-.11+ 在加入自定义 SerDe 时存在错误.
From what I've seen, Hive-.11+ has a bug in join with custom SerDe.
https://github.com/Esri/gis-tools-for-hadoop/issues/9
您可以尝试将包含 SerDe 类的 JAR 文件复制到 $HIVE_HOME/lib 的解决方法.
You might try the workaround of copying the JAR file containing the SerDe class, to $HIVE_HOME/lib .
(我在你的问题中看到你在 join 和其他情况下都遇到了 ClassNotFoundException;到目前为止,我遇到的都是 join.)
(I see in your question you got ClassNotFoundException both in join and in other cases; so far the times I have encountered such were all with join.)
另一个解决方法是使用 HADOOP_CLASSPATH:
Another workaround is to use HADOOP_CLASSPATH:
env HADOOP_CLASSPATH=some.jar:other.jar hive ...
[Edit] 变通方法适用于 Hive 版本 0.11 和 0.12;然后 0.13 及更高版本包含对 HIVE-6670 的修复.
The work around applies to Hive versions 0.11 and 0.12; then 0.13 and above contain the fix for HIVE-6670.
这篇关于Hive 0.12 和 Hadoop 2.2.0-cdh5.0.0-beta2 的 SerDe 问题的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持!