通过检查值是否在列表中以及其他条件来过滤Spark DataFrame

问题描述

作为一个简化的示例，我尝试使用以下代码过滤Spark DataFrame:

As a simplified example, I tried to filter a Spark DataFrame with following code:

val xdf = sqlContext.createDataFrame(Seq(
  ("A", 1), ("B", 2), ("C", 3)
)).toDF("name", "cnt")
xdf.filter($"cnt" >1 || $"name" isin ("A","B")).show()

然后出错:

org.apache.spark.sql.AnalysisException: cannot resolve '((cnt > 1) || name)' due to data type mismatch: differing types in '((cnt > 1) || name)' (boolean and string).;

什么是正确的方法?在我看来，它在name列之后停止读取.这是解析器中的错误吗?我正在使用Spark 1.5.1

What's the right way to do it? It seems to me that it stops reading after name column. Is it a bug in the parser? I'm using Spark 1.5.1

推荐答案

您必须在各个表达式之间加上括号:

You have to parenthesize individual expressions:

xdf.filter(($"cnt" > 1) || ($"name" isin ("A","B"))).show()

这篇关于通过检查值是否在列表中以及其他条件来过滤Spark DataFrame的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！