问题描述
我使用Flume 1.4.0和Hadoop 2.2.0。
当我启动Flume并写入HDFS时,出现以下异常:
$ b $ $ $ $ $ $ $ $ $ $ $ $ $ $ $(SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:460)] process failed
java.lang.VerifyError:class org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos $ RenewLeaseRequestProto覆盖最终方法getUnknownFields。()Lcom / google / protobuf / UnknownFieldSet;在java.lang.ClassLoader.defineClass1处使用
(Native Method)$在java.lang.ClassLoader.defineClass中使用
(ClassLoader.java:791)在java.security.SecureClassLoader.defineClass处使用
(SecureClassLoader。在java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
在java.net.URLClassLoader.access $ b $(URLClassLoader.java:71)$ b $在java。 net.URLClassLoader $ 1.run(URLClassLoader.java:361)在java.net.URLClassLoader上
$ 1.run(URLClassLoader.java:355)$ java.util.AccessController.doPrivileged中的b $ b(Native方法)
。在在java.lang.ClassLoader.loadClass(ClassLoader.java:423)
java.net.URLClassLoader.findClass(URLClassLoader.java:354)
在sun.misc.Launcher $ AppClassLoader。 loadClass(Launcher.java:308)$ b $ java.util.ClassLoader.loadClass(ClassLoader.java:356)
at java.lang.Class.getDeclaredMethods0(Native Method)
at java。 lang.Class.privateGetDec laredMethods(Class.java:2442)
在java.lang.Class.privateGetPublicMethods(Class.java:2562)
在java.lang.Class.privateGetPublicMethods(Class.java:2572)
在java.lang.Class.getMethods(Class.java:1427)
在sun.misc.ProxyGenerator.generateClassFile(ProxyGenerator.java:426)
在sun.misc.ProxyGenerator.generateProxyClass(ProxyGenerator.java :323)
at java.lang.reflect.Proxy.getProxyClass(Proxy.java:521)
at java.lang.reflect.Proxy.newProxyInstance(Proxy.java:601)
at在org.apache.hadoop.ipc.RPC.getProtocolProxy org.apache.hadoop.ipc.ProtobufRpcEngine.getProxy(ProtobufRpcEngine.java:92)
(RPC.java:537)
。在org.apache。 hadoop.hdfs.NameNodeProxies.createNNProxyWithClientProtocol(NameNodeProxies.java:328)
at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:235)
at org.apache.hadoop.hdfs。 NameNodeProxies.createProxy(NameNodePro xies.java:139)
at org.apache.hadoop.hdfs.DFSClient。< init>(DFSClient.java:510)
at org.apache.hadoop.hdfs.DFSClient。< init> ;(DFSClient.java:453)
at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:136)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem。
at org.apache.hadoop.fs.FileSystem.access $ 200(FileSystem.java:88)
at org.apache.hadoop.fs.FileSystem $ Cache.getInternal(FileSystem.java :2467)
at org.apache.hadoop.fs.FileSystem $ Cache.get(FileSystem.java:2449)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:367 )
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:287)
at org.apache.flume.sink.hdfs.BucketWriter.doOpen(BucketWriter.java:207)
at org.apache.flume.sink.hdfs.BucketWriter.access $ 000(BucketWriter.java:53)
at org.apache.flume.sink.hdfs.BucketWriter $ 1.run(Bu cketWriter.java:172)
at org.apache.flume.sink.hdfs.BucketWriter $ 1.run(BucketWriter.java:170)
at org.apache.flume.sink.hdfs.BucketWriter.runPrivileged (BucketWriter.java:143)
at org.apache.flume.sink.hdfs.BucketWriter.open(BucketWriter.java:170)
at org.apache.flume.sink.hdfs.BucketWriter.append (BucketWriter.java:364)
。在org.apache.flume.sink.hdfs.HDFSEventSink $ 2.call(HDFSEventSink.java:729)
。在org.apache.flume.sink.hdfs.HDFSEventSink $ 2 .call(HDFSEventSink.java:727)
at java.util.concurrent.FutureTask $ Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java :166)在java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110
)
在java.util.concurrent.ThreadPoolExecutor中的$ Worker.run(ThreadPoolExecutor.java:603)
在java.lang.Thread.run(Thread.java:722)
我的hdfs的一部分在flume.conf中的-sink看起来像这样:
定义输出到hdfs的接收器
agent.sinks.hdfs-sink.channel =内存通道
agent.sinks.hdfs-sink.type = hdfs
agent.sinks.hdfs-sink.hdfs。 path = hdfs:// localhost:8020 / flume
agent.sinks.hdfs-sink.hdfs.fileType = DataStream
agent.sinks.hdfs-sink.hdfs.writeFormat = Text
agent .sinks.hdfs-sink.hdfs.rollCount = 10
agent.sinks.hdfs-sink.hdfs.batchSize = 10
agent.sinks.hdfs-sink.hdfs.rollSize = 0
我希望任何人都可以帮助我。
如果有人遇到同样的问题,可以使用以下解决方案:从flume / lib目录中替换所有较旧的jar, hadoop。
I'm using Flume 1.4.0 and Hadoop 2.2.0.When I'm starting Flume and writing to HDFS I get following Exception:
(SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:460)] process failed java.lang.VerifyError: class org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$RenewLeaseRequestProto overrides final method getUnknownFields.()Lcom/google/protobuf/UnknownFieldSet; at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:791) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:449) at java.net.URLClassLoader.access$100(URLClassLoader.java:71) at java.net.URLClassLoader$1.run(URLClassLoader.java:361) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:423) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:356) at java.lang.Class.getDeclaredMethods0(Native Method) at java.lang.Class.privateGetDeclaredMethods(Class.java:2442) at java.lang.Class.privateGetPublicMethods(Class.java:2562) at java.lang.Class.privateGetPublicMethods(Class.java:2572) at java.lang.Class.getMethods(Class.java:1427) at sun.misc.ProxyGenerator.generateClassFile(ProxyGenerator.java:426) at sun.misc.ProxyGenerator.generateProxyClass(ProxyGenerator.java:323) at java.lang.reflect.Proxy.getProxyClass(Proxy.java:521) at java.lang.reflect.Proxy.newProxyInstance(Proxy.java:601) at org.apache.hadoop.ipc.ProtobufRpcEngine.getProxy(ProtobufRpcEngine.java:92) at org.apache.hadoop.ipc.RPC.getProtocolProxy(RPC.java:537) at org.apache.hadoop.hdfs.NameNodeProxies.createNNProxyWithClientProtocol(NameNodeProxies.java:328) at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:235) at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:139) at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:510) at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:453) at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:136) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2433) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:88) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2467) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2449) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:367) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:287) at org.apache.flume.sink.hdfs.BucketWriter.doOpen(BucketWriter.java:207) at org.apache.flume.sink.hdfs.BucketWriter.access$000(BucketWriter.java:53) at org.apache.flume.sink.hdfs.BucketWriter$1.run(BucketWriter.java:172) at org.apache.flume.sink.hdfs.BucketWriter$1.run(BucketWriter.java:170) at org.apache.flume.sink.hdfs.BucketWriter.runPrivileged(BucketWriter.java:143) at org.apache.flume.sink.hdfs.BucketWriter.open(BucketWriter.java:170) at org.apache.flume.sink.hdfs.BucketWriter.append(BucketWriter.java:364) at org.apache.flume.sink.hdfs.HDFSEventSink$2.call(HDFSEventSink.java:729) at org.apache.flume.sink.hdfs.HDFSEventSink$2.call(HDFSEventSink.java:727) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722)
The part of my hdfs-sink in the flume.conf is looking like this:
Define a sink that outputs to hdfs
agent.sinks.hdfs-sink.channel = memory-channel agent.sinks.hdfs-sink.type = hdfs agent.sinks.hdfs-sink.hdfs.path = hdfs://localhost:8020/flume agent.sinks.hdfs-sink.hdfs.fileType = DataStream agent.sinks.hdfs-sink.hdfs.writeFormat = Text agent.sinks.hdfs-sink.hdfs.rollCount = 10 agent.sinks.hdfs-sink.hdfs.batchSize = 10 agent.sinks.hdfs-sink.hdfs.rollSize = 0
I hope anyone can help me.
If somebody has the same problem, here's the solution:
Replace all older jars in the flume/lib-directory by copying the newer ones from hadoop.
这篇关于通过Flume& Hadoop不起作用的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持!