如何在CASE语句中使用数组类型列值

问题描述

我有一个包含两列的数据框，listA存储为Seq[String]，valB存储为String.我想创建第三列valC，该列将为Int类型，其值为
iff valB is present in listA then 1 otherwise 0

I have a dataframe with two columns, listA stored as Seq[String] and valB stored as String. I want to create a third column valC, which will be of Int type and its value is
iff valB is present in listA then 1 otherwise 0

我尝试执行以下操作:

val dfWithAdditionalColumn = df.withColumn("valC", when($"listA".contains($"valB"), 1).otherwise(0))

但是Spark无法执行此操作，并给出了以下错误:

But Spark failed to execute this and gave the following error:

cannot resolve 'contains('listA', 'valB')' due to data type mismatch: argument 1 requires string type, however, 'listA' is of array type.;

如何在CASE语句中使用数组类型的列值?

How do I use a array type column value in CASE statement?

谢谢，Devj

推荐答案

您可以编写一个简单的udf来检查数组中是否存在该元素:

You can write a simple udf that will check if the element is present in the array :

val arrayContains = udf( (col1: Int, col2: Seq[Int]) => if(col2.contains(col1) ) 1 else 0 )

然后调用它并以正确的顺序传递必要的列:

And then just call it and pass the necessary columns in the correct order :

df.withColumn("hasAInB", arrayContains($"a", $"b" ) ).show

+---+---------+-------+
|  a|        b|hasAInB|
+---+---------+-------+
|  1|   [1, 2]|      1|
|  2|[2, 3, 4]|      1|
|  3|   [1, 4]|      0|
+---+---------+-------+

这篇关于如何在CASE语句中使用数组类型列值的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！