如何在 Spark 2.0 中启用笛卡尔连接?

问题描述

我必须在 Spark 2.0 中交叉连接 2 个数据帧我遇到以下错误:

I have to cross join 2 dataframe in Spark 2.0 I am encountering below error:

用户类抛出异常:

org.apache.spark.sql.AnalysisException: Cartesian joins could be prohibitively expensive and are disabled by default. To explicitly enable them, please set spark.sql.crossJoin.enabled = true;

请帮我在哪里设置这个配置，我在eclipse中编码.

Please help me where to set this configuration, I am coding in eclipse.

推荐答案

由于错误消息明确指出您需要将 spark.sql.crossJoin.enabled = true 设置为您的 Spark 配置

As the error message clearly states you need to set spark.sql.crossJoin.enabled = true to your spark configuration

你可以像下面一样设置:

You can set the same something like below:

val sparkConf = new SparkConf().setAppName("Test")
sparkConf.set("spark.sql.crossJoin.enabled", "true")

然后通过传递这个SparkConf

val sparkSession = SparkSession.builder().config(sparkConf).getOrCreate()

那就加入吧...

这篇关于如何在 Spark 2.0 中启用笛卡尔连接?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！