SPARK:失败:“联合"预期但“("找到

问题描述

我有一个名为 df 的数据框，其中包含名为 employee_id 的列.我在做:

I have a dataframe called df with column named employee_id. I am doing:

 df.registerTempTable("d_f")
val query = """SELECT *, ROW_NUMBER() OVER (ORDER BY employee_id) row_number FROM d_f"""
val result = Spark.getSqlContext().sql(query)

但是遇到以下问题.有什么帮助吗?

But getting following issue. Any help?

[1.29] failure: ``union'' expected but `(' found
SELECT *, ROW_NUMBER() OVER (ORDER BY employee_id) row_number FROM d_f
                            ^
java.lang.RuntimeException: [1.29] failure: ``union'' expected but `(' found
SELECT *, ROW_NUMBER() OVER (ORDER BY employee_id) row_number FROM d_f

推荐答案

Spark 2.0+

Spark 2.0 引入了窗口函数的原生实现 (SPARK-8641) 所以 HiveContext 应该不再需要了.尽管如此，与窗口函数无关的类似错误仍然可以归因于 SQL 解析器之间的差异.

Spark 2.0 introduces native implementation of window functions (SPARK-8641) so HiveContext should be no longer required. Nevertheless similar errors, not related to window functions, can be still attributed to the differences between SQL parsers.

火花

Spark 1.4.0 中引入了窗口函数，并且需要 HiveContext 工作.SQLContext不能在这里工作.

Window functions have been introduced in Spark 1.4.0 and require HiveContext to work. SQLContext won't work here.

确保您使用 Spark >= 1.4.0 并创建 HiveContext:

Be sure you you use Spark >= 1.4.0 and create the HiveContext:

import org.apache.spark.sql.hive.HiveContext
val sqlContext = new HiveContext(sc)

这篇关于SPARK:失败:“联合"预期但“("找到的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！