如何将多个列连接成单个列(不事先知道其编号)?

问题描述

假设我具有以下数据框:

Let say I have the following dataframe:

agentName|original_dt|parsed_dt|   user|text|
+----------+-----------+---------+-------+----+
|qwertyuiop|          0|        0|16102.0|   0|

我希望创建一个新的数据框，其中又包含一列，该列具有该行所有元素的串联:

I wish to create a new dataframe with one more column that has the concatenation of all the elements of the row:

agentName|original_dt|parsed_dt|   user|text| newCol
+----------+-----------+---------+-------+----+
|qwertyuiop|          0|        0|16102.0|   0| [qwertyuiop, 0,0, 16102, 0]

注意:这只是一个例子.列数及其名称未知.它是动态的.

Note: This is a just an example. The number of columns and names of them is not known. It is dynamic.

推荐答案

我认为这非常适合您的情况这是一个例子

I think this works perfect for your case here is with an example

val spark =
    SparkSession.builder().master("local").appName("test").getOrCreate()
  import spark.implicits._
  val data = spark.sparkContext.parallelize(
    Seq(
      ("qwertyuiop", 0, 0, 16102.0, 0)
    )
  ).toDF("agentName","original_dt","parsed_dt","user","text")


  val result = data.withColumn("newCol", split(concat_ws(";",  data.schema.fieldNames.map(c=> col(c)):_*), ";"))        
  result.show()

+----------+-----------+---------+-------+----+------------------------------+
|agentName |original_dt|parsed_dt|user   |text|newCol                        |
+----------+-----------+---------+-------+----+------------------------------+
|qwertyuiop|0          |0        |16102.0|0   |[qwertyuiop, 0, 0, 16102.0, 0]|
+----------+-----------+---------+-------+----+------------------------------+

希望这对您有帮助！

这篇关于如何将多个列连接成单个列(不事先知道其编号)?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！