scala - build.sbt:如何添加Spark依赖项

您好，我正在尝试在下面的build.sbt文件中下载spark-core，spark-streaming，twitter4j和spark-streaming-twitter:

name := "hello"

version := "1.0"

scalaVersion := "2.11.8"

libraryDependencies += "org.apache.spark" %% "spark-core" % "1.6.1"
libraryDependencies += "org.apache.spark" % "spark-streaming_2.10" % "1.4.1"

libraryDependencies ++= Seq(
  "org.twitter4j" % "twitter4j-core" % "3.0.3",
  "org.twitter4j" % "twitter4j-stream" % "3.0.3"
)

libraryDependencies += "org.apache.spark" % "spark-streaming-twitter_2.10" % "0.9.0-incubating"

我只是将此libraryDependencies在线上带了，所以我不确定要使用哪个版本等。

有人可以向我解释如何修复此.sbt文件。我花了几个小时试图找出答案，但没有一个建议起作用。我通过自制软件安装了scala，并且我使用的是2.11.8版本

我所有的错误都与:

Modules were resolved with conflicting cross-version suffixes.

最佳答案

问题是您正在混合Scala 2.11和2.10 Artifact 。你有:

scalaVersion := "2.11.8"

接着:

libraryDependencies += "org.apache.spark" % "spark-streaming_2.10" % "1.4.1"

需要2.10 Artifact 的位置。您还将混合使用Spark版本，而不是使用一致版本:

// spark 1.6.1
libraryDependencies += "org.apache.spark" %% "spark-core" % "1.6.1"

// spark 1.4.1
libraryDependencies += "org.apache.spark" % "spark-streaming_2.10" % "1.4.1"

// spark 0.9.0-incubating
libraryDependencies += "org.apache.spark" % "spark-streaming-twitter_2.10" % "0.9.0-incubating"

这是一个可解决两个问题的build.sbt:

name := "hello"

version := "1.0"

scalaVersion := "2.11.8"

val sparkVersion = "1.6.1"

libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-core" % sparkVersion,
  "org.apache.spark" %% "spark-streaming" % sparkVersion,
  "org.apache.spark" %% "spark-streaming-twitter" % sparkVersion
)

您也不需要手动添加twitter4j依赖项，因为它们是由spark-streaming-twitter临时添加的。

关于scala - build.sbt:如何添加Spark依赖项，我们在Stack Overflow上找到一个类似的问题：https://stackoverflow.com/questions/37958158/