如何在两个数组列之间找到公共元素?

问题描述

我有两个逗号分隔的字符串列(sourceAuthors 和 targetAuthors).

I have two comma-separated string columns (sourceAuthors and targetAuthors).

val df = Seq(
  ("Author1,Author2,Author3","Author2,Author3,Author1")
).toDF("source","target")

我想添加另一列 nCommonAuthors，其中包含常见作者的数量.

I'd like to add another column nCommonAuthors with the number of common Authors.

我试过这样做:

def myUDF = udf { (s1: String, s2: String) =>
  s1.split(",")
  s2.split(",")
  s1.intersect(s2).length
}
val newDF = myDF.withColumn("nCommonAuthors", myUDF($"source", $"target"))

我收到以下错误:

线程main"中的异常java.lang.UnsupportedOperationException:不支持类型Unit的架构

知道为什么我会收到这个错误吗?如何找到两列之间的共同元素?

Any idea why I get this error? How to find the common elements among two columns?

推荐答案

根据 SCouto 的回答，我为您提供了对我有用的完整解决方案:

Based on SCouto answer, I give you the complete solution that worked for me:

  def myUDF: UserDefinedFunction = udf(
(s1: String, s2: String) => {
  val splitted1 = s1.split(",")
  val splitted2 = s2.split(",")
  splitted1.intersect(splitted2).length
})

  val spark = SparkSession.builder().master("local").getOrCreate()

  import spark.implicits._

  val df = Seq(("Author1,Author2,Author3","Author2,Author3,Author1")).toDF("source","target")

  df.show(false)

+-----------------------+-----------------------+
|source                 |target                 |
+-----------------------+-----------------------+
|Author1,Author2,Author3|Author2,Author3,Author1|
+-----------------------+-----------------------+

  val newDF: DataFrame = df.withColumn("nCommonAuthors", myUDF('source,'target))

  newDF.show(false)

+-----------------------+-----------------------+--------------+
|source                 |target                 |nCommonAuthors|
+-----------------------+-----------------------+--------------+
|Author1,Author2,Author3|Author2,Author3,Author1|3             |
+-----------------------+-----------------------+--------------+

这篇关于如何在两个数组列之间找到公共元素?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！