问题描述
我已经在 Storm 工作了一段时间，但想开始开发.按照建议，我使用的是 IntelliJ(到目前为止，我使用的是 Eclipse，并且只针对 Java API 编写了拓扑).
I work with Storm for a while already, but want to get started with development. As suggested, I am using IntelliJ (up to now, I was using Eclipse and did only write topologies against Java API).
我也在看https://github.com/apache/storm/tree/master/examples/storm-starter#intellij-idea
此文档不完整.我无法首先在 Intellij 中运行任何东西.我可以弄清楚，我需要删除storm-core依赖的范围(在storm-starter pom.xml中).(在这里找到:storm-starter with intellij idea,maven 项目找不到类)
This documentation is not complete. I was not able to run anything in Intellij first. I could figure out, that I need to remove the scope of storm-core dependency (in storm-starter pom.xml). (found here: storm-starter with intellij idea,maven project could not find class)
在那之后，我能够构建项目.我也可以在 IntelliJ 中运行 ExclamationTopology 没有问题.但是，WordCountTopology 失败了.
After that I was able to build the project. I can also run ExclamationTopology with no problems within IntelliJ. However, WordCountTopology fails.
首先我收到以下错误:
java.lang.RuntimeException: backtype.storm.multilang.NoOutputException: 子进程的管道似乎坏了！没有输出读取.序列化器异常:回溯(最近一次调用最后一次):文件splitsentence.py"，第 16 行，在进口风暴导入错误:没有名为风暴的模块
However, I don't speak Python and was wondering what the problem is and why I could resolve it like this. Just want to get deeper into it. Maybe someone can explain.
不幸的是，我现在遇到了一个不同的错误:
Unfortunately, I am getting a different error now:
java.lang.RuntimeException: backtype.storm.multilang.NoOutputException: 子进程的管道似乎坏了！没有输出读取.序列化器异常:回溯(最近一次调用最后一次):文件splitsentence.py"，第 18 行，在类 SplitSentenceBolt(storm.BasicBolt):AttributeError: 'module' 对象没有属性 'BasicBolt'
我在网上没有找到任何解决方案.在 [email protected] 询问也无济于事.我的建议如下:
I did not find any solution on the Internet. Asking at [email protected] did not help either. I go the following suggestion:
我认为总是假设拓扑总是通过storm-命令行调用.因此工作目录将是 ${STORM-INSTALLATION}/bin/storm 由于storm.py 在这个目录中，splitSentence.py 将能够找到storm 模块.您能否将工作目录设置为存在storm.py 的路径，然后尝试.如果有效，我们可以稍后将其添加到文档中
然而，改变工作目录并没有解决问题.
However, chancing the working directory did not solve the problem.
而且由于我不熟悉 Python 并且我是 IntelliJ 的新手，所以我现在被困住了.因为 ExclamationTopology 运行，我想我的基本设置是正确的.
And as I am not familiar with Python and as I am new to IntelliJ, I am stuck now. Because ExclamationTopology runs, I guess my basic setup is correct.
我做错了什么?是否可以在 IntelliJ 中的 LocalCluster 中运行 WordcountTopology?
What do I do wrong? It is possible at all to run WordcountTopology in LocalCluster in IntelliJ?
推荐答案
不幸的是，AFAIK 不能在没有打包文件的情况下使用 LocalCluster 运行多语言功能.
Unfortunately, AFAIK you can't run multilang feature with LocalCluster without having packaged file.
ShellProcess 依赖于由 Supervisor 使用的 TopologyContext 的 codeDir.Worker 序列化为stormcode.ser，但多语言文件应该提取到序列化文件的外部，以便python/ruby/node/etc可以加载它.
ShellProcess relies on codeDir of TopologyContext, which is used by supervisor.Workers are serialized to stormcode.ser, but multilang files should extracted to outside of serialized file so that python/ruby/node/etc can load it.
用分发模式完成这个很容易，因为总是有用户提交的 jar，主管可以知道这是用户提交的内容.
Accomplishing this with distribute mode is easy because there's always user submitted jar, and supervisor can know it is what user submitted.
但是用本地模式完成这个并不容易，因为主管无法知道用户提交的jar，用户可以在没有打包的情况下将拓扑运行到本地模式.
But accomplishing this with local mode is not easy cause supervisor cannot know user submitted jar, and users can run topology to local mode without packaging.
因此，本地模式下的 Supervisor 从类路径中的每个 jar(以jar"结尾)中查找资源目录(resources")，并将第一次出现的内容复制到 codeDir.
So, Supervisor in local mode finds resource directory ("resources") from each jars (which ends with "jar") in classpath, and copy first occurrence to codeDir.
storm jar 将用户拓扑 jar 放在 classpath 的第一个，因此它可以毫无问题地运行.
storm jar places user topology jar to the first of classpath, so it can be run without issue.
所以通常情况下，ShellProcess 找不到splitsentence.py"是很自然的.也许您的工作目录或 PYTHONPATH 起到了作用.
So normally, it's natural for ShellProcess to not find "splitsentence.py". Maybe your working directory or PYTHONPATH did the trick.
这篇关于如何在 Intellij 中从 Storm-starter 运行 WordCountTopology的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！