本文介绍了使用 sbt 程序集构建胖 jar 时 build.sbt 中的重复数据删除错误的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！
问题描述
我正在尝试创建一个结合了 spark 和 restlet 框架的胖 jar，但我不断收到此重复数据删除错误.
I'm trying to create a fat jar of a combination of spark and restlet framework, but I keep getting this deduplication error.
这是运行sbt assembly的错误:
java.lang.RuntimeException: deduplicate: different file contents found in the following:
/Users/ccd/.ivy2/cache/org.eclipse.jetty.orbit/javax.transaction/orbits/javax.transaction-1.1.1.v201105210645.jar:META-INF/ECLIPSEF.RSA
/Users/ccd/.ivy2/cache/org.eclipse.jetty.orbit/javax.servlet/orbits/javax.servlet-3.0.0.v201112011016.jar:META-INF/ECLIPSEF.RSA
/Users/ccd/.ivy2/cache/org.eclipse.jetty.orbit/javax.mail.glassfish/orbits/javax.mail.glassfish-1.4.1.v201005082020.jar:META-INF/ECLIPSEF.RSA
/Users/ccd/.ivy2/cache/org.eclipse.jetty.orbit/javax.activation/orbits/javax.activation-1.1.0.v201105071233.jar:META-INF/ECLIPSEF.RSA
at sbtassembly.Plugin$Assembly$.sbtassembly$Plugin$Assembly$$applyStrategy$1(Plugin.scala:253)
at sbtassembly.Plugin$Assembly$$anonfun$15.apply(Plugin.scala:270)
at sbtassembly.Plugin$Assembly$$anonfun$15.apply(Plugin.scala:267)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251)
at scala.collection.AbstractTraversable.flatMap(Traversable.scala:105)
at sbtassembly.Plugin$Assembly$.applyStrategies(Plugin.scala:272)
at sbtassembly.Plugin$Assembly$.x$4$lzycompute$1(Plugin.scala:172)
at sbtassembly.Plugin$Assembly$.x$4$1(Plugin.scala:170)
at sbtassembly.Plugin$Assembly$.stratMapping$lzycompute$1(Plugin.scala:170)
at sbtassembly.Plugin$Assembly$.stratMapping$1(Plugin.scala:170)
at sbtassembly.Plugin$Assembly$.inputs$lzycompute$1(Plugin.scala:214)
at sbtassembly.Plugin$Assembly$.inputs$1(Plugin.scala:204)
at sbtassembly.Plugin$Assembly$.apply(Plugin.scala:230)
at sbtassembly.Plugin$Assembly$$anonfun$assemblyTask$1.apply(Plugin.scala:373)
at sbtassembly.Plugin$Assembly$$anonfun$assemblyTask$1.apply(Plugin.scala:370)
at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47)
at sbt.$tilde$greater$$anonfun$$u2219$1.apply(TypeFunctions.scala:42)
at sbt.std.Transform$$anon$4.work(System.scala:64)
at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:237)
at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:237)
at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:18)
at sbt.Execute.work(Execute.scala:244)
at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:237)
at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:237)
at sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:160)
at sbt.CompletionService$$anon$2.call(CompletionService.scala:30)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
[error] (*:assembly) deduplicate: different file contents found in the following:
[error] /Users/ccd/.ivy2/cache/org.eclipse.jetty.orbit/javax.transaction/orbits/javax.transaction-1.1.1.v201105210645.jar:META-INF/ECLIPSEF.RSA
[error] /Users/ccd/.ivy2/cache/org.eclipse.jetty.orbit/javax.servlet/orbits/javax.servlet-3.0.0.v201112011016.jar:META-INF/ECLIPSEF.RSA
[error] /Users/ccd/.ivy2/cache/org.eclipse.jetty.orbit/javax.mail.glassfish/orbits/javax.mail.glassfish-1.4.1.v201005082020.jar:META-INF/ECLIPSEF.RSA
[error] /Users/ccd/.ivy2/cache/org.eclipse.jetty.orbit/javax.activation/orbits/javax.activation-1.1.0.v201105071233.jar:META-INF/ECLIPSEF.RSA
[error] Total time: 8 s, completed 19/09/2014 11:50:50 AM
这是我的plugins.sbt文件
logLevel := Level.Warn
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.11.2")
这是我的build.sbt文件
import AssemblyKeys._ // put this at the top of the file
name := "Spark_sbt2"
version := "1.1"
scalaVersion := "2.10.4"
libraryDependencies += "org.apache.spark" %% "spark-core" % "1.1.0"
libraryDependencies += "org.apache.spark" %% "spark-mllib" % "1.1.0"
libraryDependencies += "org.apache.spark" %% "spark-streaming" % "1.1.0"
libraryDependencies += "org.apache.httpcomponents" % "httpclient" % "4.3.5"
libraryDependencies += "org.restlet.jse" % "org.restlet" % "2.2.2"
libraryDependencies += "org.restlet.jse" % "org.restlet.ext.jackson" % "2.2.2"
resolvers += "Akka Repository" at "http://repo.akka.io/releases/"
resolvers += "Restlet" at "http://maven.restlet.org"
assemblySettings
jarName in assembly := "spark-assembly.jar"
我不知道如何解决它.感谢有人能提供帮助！
I'm not sure how to fix it. Appreciate if anyone could help!
推荐答案
问题是 sbt-assembly 无法选择要拾取的 META-INF/ECLIPSEF.RSA 文件.试试这个来排除所有这样的文件:
The problem is sbt-assembly cannot choose which META-INF/ECLIPSEF.RSA file to pickup. Try this to exclude all such files:
excludedFiles in assembly <<=
(excludedFiles in assembly) {
(old) => (bases) =>
old(bases) ++ (bases flatMap (base =>
(base / "META-INF/ECLIPSEF.RSA").get))
}
如果您确实需要在 jar 中使用一些 RSA 密钥:
If you really need some RSA key inside jar:
assembledMappings in assembly <<= (assembledMappings in assembly) map { (old: File => Seq[(File, String)]) =>
(f: File) => old(f) :+ (file("path/to/choosen/rsa/key"), "META-INF/ECLIPSEF.RSA")
}
(或者不要在 excludeFiles 中提及它以获得适当的基础)
(or don't mention it in excludedFiles for appropriate base)
这篇关于使用 sbt 程序集构建胖 jar 时 build.sbt 中的重复数据删除错误的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！