如何在Spark 2.0中启用笛卡尔联接?

问题描述

我必须在Spark 2.0中交叉连接2个数据帧，遇到以下错误:

I have to cross join 2 dataframe in Spark 2.0 I am encountering below error:

用户类抛出异常:

org.apache.spark.sql.AnalysisException: Cartesian joins could be prohibitively expensive and are disabled by default. To explicitly enable them, please set spark.sql.crossJoin.enabled = true;

请帮助我在哪里设置此配置，我正在eclipse中进行编码.

Please help me where to set this configuration, I am coding in eclipse.

推荐答案

错误消息明确指出，您需要将spark.sql.crossJoin.enabled = true设置为spark配置

As the error message clearly states you need to set spark.sql.crossJoin.enabled = true to your spark configuration

您可以设置如下所示的内容:

You can set the same something like below:

val sparkConf = new SparkConf().setAppName("Test")
sparkConf.set("spark.sql.crossJoin.enabled", "true")

然后通过传递此SparkConf

val sparkSession = SparkSession.builder().config(sparkConf).getOrCreate()

然后加入您的行列...

Then do your join...

这篇关于如何在Spark 2.0中启用笛卡尔联接?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！