问题描述
我目前在将 uber-jar 部署到 Spark Streaming 应用程序时遇到问题，其中存在具有不同版本的一致 JAR，这导致 Spark 抛出运行时异常.有问题的库是 TypeSafe Config.
I'm currently facing a problem with deploying an uber-jar to a Spark Streaming application, where there are congruent JARs with different versions which are causing spark to throw run-time exceptions. The library in question is TypeSafe Config.
在尝试了很多事情之后，我的解决方案是推迟对提供的依赖项进行着色，这样它就不会在运行时与 Spark 提供的 JAR 冲突.
After attempting many things, my solution was to defer to shading the provided dependency so it won't clash with the JAR provided by Spark at run-time.
因此，我转到了 sbt-assembly 的文档在阴影下，我看到了以下示例:
Hence, I went to the documentation for sbt-assembly and under shading, I saw the following example:
assemblyShadeRules in assembly := Seq(
ShadeRule.rename("org.apache.commons.io.**" -> "shadeio.@1")
.inLibrary("commons-io" % "commons-io" % "2.4", ...).inProject
)
试图遮蔽 com.typesafe.config，我尝试将以下解决方案应用于我的 build.sbt:
Attempting to shade over com.typesafe.config, I tried applying the following solution to my build.sbt:
assemblyShadeRules in assembly := Seq(
ShadeRule.rename("com.typesafe.config.**" -> "shadeio.@1").inProject
)
我认为它应该重命名我的项目中对 TypeSafe Config 的任何引用.但是，这行不通.它匹配我项目中的多个类并导致它们从 uber jar 中删除.我在尝试运行 sbt assembly 时看到了这一点:
I assumed it was supposed to rename any reference to TypeSafe Config in my project. But, this doesn't work. It matches multiple classes in my project and causing them to be removed from the uber jar. I see this when trying to run sbt assembly:
Fully-qualified classname does not match jar entry:
jar entry: ***/Identifier.class
class name: **/Identifier.class
Omitting ***/OtherIdentifier.class.
Fully-qualified classname does not match jar entry:
jar entry: ***\SparkBaseJobRunner$$anonfun$1.class
class name: ***/SparkBaseJobRunner$$anonfun$1.class
我也尝试使用:
assemblyShadeRules in assembly := Seq(
ShadeRule.rename("com.typesafe.config.**" -> "shadeio.@1")
.inLibrary("com.typesafe" % "config" % "1.3.0")
这确实完成了 uber JAR 的组装过程，但没有达到预期的运行时效果.
This did finish the assemblying process of the uber JAR, but didn't have the desired run time effect.
我不确定我是否完全理解着色对我使用 sbt 的构建过程的影响.
I'm not sure I fully comprehend the effect shading has on my build process with sbt.
如何在我的项目中屏蔽对 com.typesafe.config 的引用，以便在运行时调用库时 Spark 将加载我的着色库并避免由版本控制引起的冲突?
How can I shade over references to com.typesafe.config in my project so when I invoke the library at run-time Spark will load my shaded library and avoid the clash caused by versioning?
我正在运行 sbt-assembly v0.14.1
I'm running sbt-assembly v0.14.1
推荐答案
结果 this是 sbt-assembly 中的一个错误，其中在 Windows 上完全破坏了阴影. 这导致源文件从 uber JAR 中删除，并且由于所述类不可用而导致测试失败.
Turns out this was a bug in sbt-assembly where shading was completely broken on Windows. This caused source files to be removed from the uber JAR, and for tests to fail as the said classes were unavailable.
我创建了一个拉请求来解决这个问题.从 SBT 0.14.3 版开始，着色功能正常工作.您需要做的就是更新到 plugins.sbt 中的相关版本:
I created a pull request to fix this. Starting version 0.14.3 of SBT, the shading feature works properly. All you need to do is update to the relevant version in plugins.sbt:
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.3")
为了给项目中的特定 JAR 着色，请执行以下操作:
In order to shade a specific JAR in your project, you do the following:
assemblyShadeRules in assembly := Seq(
ShadeRule.rename("com.typesafe.config.**" -> "my_conf.@1")
.inLibrary("com.typesafe" % "config" % "1.3.0")
.inProject
)
这将重命名要打包在 my_conf 中的 com.typesafe.config 程序集.然后你可以在你的程序集上使用 jar -tf 来查看它(为了简洁省略了不相关的部分):
This will rename the com.typesafe.config assembly to be packaged inside my_conf. You can then view this using jar -tf on your assembly (omitted irrelevant parts for brevity):
***> jar -tf myassembly.jar
my_conf/
my_conf/impl/
my_conf/parser/
编辑
我写了一篇博文为对更深入解释感兴趣的任何人描述问题和导致该问题的过程.
I wrote a blog post describing the issue and the process that led to it for anyone interested in a more in-depth explanation.
这篇关于在第三方类上着色的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！