级联+ libjars = ClassNotFoundException.有时

问题描述

我正在运行将DistributedCache用于相关jar的层叠(实际上是缩放)hadoop作业.

I am running Cascading (actually Scalding) hadoop job that uses DistributedCache for dependent jars.

第一时间工作正常(这意味着类路径已正确设置)，但随后由于ClassNotFoundException而开始失败:

Fist time it works fine (meaning that the classpath is set up correctly) but then it starts failing with ClassNotFoundException:

java.io.IOException: Split class cascading.tap.hadoop.io.MultiInputSplit not found
    at org.apache.hadoop.mapred.MapTask.getSplitDetails(MapTask.java:387)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:412)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at  org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
    at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.lang.ClassNotFoundException: cascading.tap.hadoop.io.MultiInputSplit
    at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:247)
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:820)
    at org.apache.hadoop.mapred.MapTask.getSplitDetails(MapTask.java:385)
    ...

其他人在DistributedCache中使用Cascading和jars是否成功

Did anybody else have success with Cascading and jars in the DistributedCache

此消息似乎在暗示Cascading对分布式缓存jar进行了一些内部处理.您对此有何见解?

This message seems to imply that Cascading has some internal handling of the distributed cache jars. Any light you can shed on this?

编辑:我正在Hadoop 1.0.3上使用Cascading 2.1.6

Edit: I am using Cascading 2.1.6 on Hadoop 1.0.3

推荐答案

Chris K Wensel ，作者邮件列表上的级联已回复分布式缓存.

Chris K Wensel, the author of Cascading responded on the mailing list that Cascading does not do anything with DistributedCache.

我进一步看了看，这是我的代码中的一个问题-我没有将这些文件正确地添加到DistributedCache中.

I looked further and it was a problem in my code -- I did not add these files to the DistributedCache properly.

这篇关于级联+ libjars = ClassNotFoundException.有时的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！