问题描述
我已经设置了一个Kestrel服务器，并能够通过python pykestrel 库来设置和使用队列.
I've setup a Kestrel server and able to setup and use queues via the python pykestrel library.
我们有一个场景，其中python客户端写入Kestrel队列，而Storm spout需要从队列中读取.我尝试使用Storm-Kestrel库，但是遇到了问题.谷歌搜索似乎表明它不支持内存缓存端口(22133).
We have a scenario where a python client writes to Kestrel queue(s) and a Storm spout needs to read from the queue(s).I've tried using the storm-kestrel library but running into issues. Googling seems to suggest it doesn't support the memcache port (22133).
我已经在此处中添加了Maven绑定.没有使用Kestrel.Client使用KestrelThriftSpout喷口.编译很好，但是在运行时出现java.lang.NoClassDefFoundError错误.
I've added the maven bindings as provided here. Didn't use the KestrelThriftSpout spout, using Kestrel.Client. Compilation is fine but I get java.lang.NoClassDefFoundError error at runtime.
堆栈跟踪
1098 [main] ERROR org.apache.zookeeper.server.NIOServerCnxn - Thread Thread[main,5,main] died
java.lang.NoClassDefFoundError: Lnet/lag/kestrel/thrift/Kestrel$Client;
at java.lang.Class.getDeclaredFields0(Native Method)
at java.lang.Class.privateGetDeclaredFields(Class.java:2397)
at java.lang.Class.getDeclaredField(Class.java:1946)
at java.io.ObjectStreamClass.getDeclaredSUID(ObjectStreamClass.java:1659)
at java.io.ObjectStreamClass.access$700(ObjectStreamClass.java:72)
at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:480)
at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:468)
at java.security.AccessController.doPrivileged(Native Method)
at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:468)
at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:365)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1133)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347)
at backtype.storm.utils.Utils.serialize(Utils.java:52)
at backtype.storm.topology.TopologyBuilder.createTopology(TopologyBuilder.java:94)
at com.rr.storm.ArticleTolopolgy.main(ArticleTolopolgy.java:35)
Caused by: java.lang.ClassNotFoundException: net.lag.kestrel.thrift.Kestrel$Client
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 15 more
有人可以确认在这种情况下是否可以使用风暴茶est?
Can someone please confirm if storm-kestrel can or cannot be used in this scenario?
如果不是，是否建议在风暴喷口中使用另一个Kestrel客户端库来从Kestrel队列中读取数据?
If not, is there another kestrel client library that is recommended to be used from within a storm spout for reading data off of kestrel queue(s)?
推荐答案
抱歉，最新更新.
-
java.lang.NoClassDefFoundError是因为某些JAR不在类路径中.其中涉及多个依赖项，因此我们通过将所有依赖项打包到一个SINGLE JAR中解决了该问题.尤其是有一些依赖项冲突需要解决.涉及AWS SQS和Storm的讨厌代码，具体取决于不同的httpclient库.需要在发布时将AWS SQS Java SDK版本降级到1.4.7，而不是最新的1.7.2.
The java.lang.NoClassDefFoundError was because some JARs were not in classpath. There were several dependencies involved so we resolved the issue by packing all dependencies into a SINGLE JAR. There were several dependency conflicts that needed to be resolved esp. a nasty one involving AWS SQS and storm depending on different httpclient libraries. Required downgrading the AWS SQS Java SDK version to 1.4.7 instead of the latest 1.7.2 at the time of posting.
storm-kestrel 库看起来与kestrel兼容服务器，但如果理解错误，其他人可以纠正.
It doesn't look like the storm-kestrel library works with the kestrel server anymore but someone else can correct if the understanding is wrong.
将 pykestrel 用于Python客户端，效果很好.
Used pykestrel for Python client and it works great.
将 simple-kestrel-client 用于Java红k客户端(Storm spout )，效果很好
Used simple-kestrel-client for Java kestrel client (Storm spout) and it works great
使用的Java SDK v1.4.7 由于冲突而适用于AWS SQS，并且效果很好.
Used Java SDK v1.4.7 for AWS SQS due to the conflict and it works great.
这篇关于Java客户端库，用于从Storm Spout中读取Kestrel服务器队列的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！