Pyspark:在 UDF 中传递多列

问题描述

我正在编写一个用户定义的函数，它将获取数据框中除第一列之外的所有列并进行求和(或任何其他操作).现在数据框有时可以有 3 列或 4 列或更多.它会有所不同.

我知道我可以将 4 个列名称硬编码为 UDF 中的传递，但在这种情况下它会有所不同，所以我想知道如何完成它?

这里有两个例子，第一个我们有两列要添加，第二个我们有三列要添加.

解决方案

如果要传递给 UDF 的所有列都具有相同的数据类型，则可以使用数组作为输入参数，例如:

>>>从 pyspark.sql.types 导入 IntegerType>>>从 pyspark.sql.functions 导入 udf，数组>>>sum_cols = udf(lambda arr: sum(arr), IntegerType())>>>spark.createDataFrame([(101, 1, 16)], ['ID', 'A', 'B']) \... .withColumn('Result', sum_cols(array('A', 'B'))).show()+---+---+---+------+|身份证|A|B|结果|+---+---+---+------+|101|1|16|17|+---+---+---+------+>>>spark.createDataFrame([(101, 1, 16, 8)], ['ID', 'A', 'B', 'C'])\... .withColumn('Result', sum_cols(array('A', 'B', 'C'))).show()+---+---+---+---+------+|身份证|A|乙|C|结果|+---+---+---+---+------+|101|1|16|8|25|+---+---+---+---+------+

I am writing a User Defined Function which will take all the columns except the first one in a dataframe and do sum (or any other operation). Now the dataframe can sometimes have 3 columns or 4 columns or more. It will vary.

I know I can hard code 4 column names as pass in the UDF but in this case it will vary so I would like to know how to get it done?

Here are two examples in the first one we have two columns to add and in the second one we have three columns to add.

解决方案

If all columns you want to pass to UDF have the same data type you can use array as input parameter, for example:

>>> from pyspark.sql.types import IntegerType
>>> from pyspark.sql.functions import udf, array
>>> sum_cols = udf(lambda arr: sum(arr), IntegerType())
>>> spark.createDataFrame([(101, 1, 16)], ['ID', 'A', 'B']) \
...     .withColumn('Result', sum_cols(array('A', 'B'))).show()
+---+---+---+------+
| ID|  A|  B|Result|
+---+---+---+------+
|101|  1| 16|    17|
+---+---+---+------+

>>> spark.createDataFrame([(101, 1, 16, 8)], ['ID', 'A', 'B', 'C'])\
...     .withColumn('Result', sum_cols(array('A', 'B', 'C'))).show()
+---+---+---+---+------+
| ID|  A|  B|  C|Result|
+---+---+---+---+------+
|101|  1| 16|  8|    25|
+---+---+---+---+------+

这篇关于Pyspark:在 UDF 中传递多列的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！