我们还需要制作一个胖罐来提交Spark 2.0.0中的工作吗？

问题描述

在Spark 2.0.0的发布中， ，它说：

In the Spark 2.0.0's release note, it says that:

• 这是否意味着我们不再需要制作一个胖罐来提交工作？

  • Does this mean that we do not need to make a fat jar anymore for submitting jobs ?

    如果是，怎么样？因此的文档不是最新的。

    If yes, how ? Thus the documentation here isn't up-to-date.

    推荐答案

    可悲的是，没有。您仍然需要为Sparks部署创建一个超级JAR。

    Sadly, no. You still have to create an uber JARs for Sparks deployment.

    发行说明中的​​标题非常具有误导性。实际意义是Spark本身作为依赖项不再编译为uber JAR，而是像具有依赖项的普通应用程序JAR。您可以更详细地看到这一点@ ，称为允许在没有程序集的情况下构建Spark，并阅读名为，它描述了部署Spark的优点和缺点，而不是几个巨大的JAR（核心，流，SQL等） ..）但作为一个包含代码的几个相对常规大小的JAR和一个包含所有相关依赖项的 lib / 目录。

    The title from the release notes is very misleading. The actual meaning is that Spark itself as a dependency is no longer compiled into an uber JAR, but acts like a normal application JAR with dependencies. You can see this in more detail @ SPARK-11157 which is called "Allow Spark to be built without assemblies", and read the paper called "Replacing the Spark Assembly with good oldjars" which describes the pros and cons of deploying Spark not as several huge JARs (Core, Streaming, SQL, etc..) but as a several relatively regular sized JARs containing the code and a lib/ directory with all the related dependencies.

    如果您真的需要详细信息，请涉及几个关键部分。

    If you really want the details, this pull request touches several key parts.

    这篇关于我们还需要制作一个胖罐来提交Spark 2.0.0中的工作吗？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！