将外部罐子设置为hadoop classpath

问题描述

我试图将外部罐子设置为hadoop classpath，但目前为止还没有运气。

我有以下设置

Classpath

我能够看到上面的HADOOP_CLASSPATH已经被hadoop拾起

命令

我试过还有-libjars选项以及

lockquote
$ sudo hadoop jar FlightsByCarrier.jar FlightsByCarrier / user / root / 1987。 csv / user / root / result -libjars /home/tom/workspace/libs/opencsv-2.3.jar

堆栈跟踪

任何帮助都非常感谢。

解决方案

运行地图的节点上缺少外部jar。您必须将其添加到缓存以使其可用。尝试：

  DistributedCache.addFileToClassPath（new Path（pathToJar），conf）; 
  

不确定版本 DistributedCache 是否被弃用，但从Hadoop 2.2.0开始，您可以使用：

  job.addFileToClassPath（new Path（pathToJar））; 
  

I am trying to set external jars to hadoop classpath but no luck so far.

I have the following setup

Classpath

I am able see the above HADOOP_CLASSPATH has been picked up by hadoop

Command

I tried with -libjars option as well

The stacktrace

Any help is highly appreciated.

解决方案

Your external jar is missing on the node running maps. You have to add it to the cache to make it available. Try :

DistributedCache.addFileToClassPath(new Path("pathToJar"), conf);

Not sure in which version DistributedCache was deprecated, but from Hadoop 2.2.0 onward you can use :

job.addFileToClassPath(new Path("pathToJar"));

这篇关于将外部罐子设置为hadoop classpath的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！