本文介绍了如何使VectorAssembler不压缩数据?的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！
问题描述
我想使用 VectorAssembler 将多列转换为一列，但是默认情况下，数据已压缩，没有其他选项.
I want to transform multiple columns to one column using VectorAssembler,but the data is compressed by default without other options.
val arr2= Array((1,2,0,0,0),(1,2,3,0,0),(1,2,4,5,0),(1,2,2,5,6))
val df=sc.parallelize(arr2).toDF("a","b","c","e","f")
val colNames=Array("a","b","c","e","f")
val assembler = new VectorAssembler()
.setInputCols(colNames)
.setOutputCol("newCol")
val transDF= assembler.transform(df).select(col("newCol"))
transDF.show(false)
输入为:
+---+---+---+---+---+
| a| b| c| e| f|
+---+---+---+---+---+
| 1| 2| 0| 0| 0|
| 1| 2| 3| 0| 0|
| 1| 2| 4| 5| 0|
| 1| 2| 2| 5| 6|
+---+---+---+---+---+
结果是:
+---------------------+
|newCol |
+---------------------+
|(5,[0,1],[1.0,2.0]) |
|[1.0,2.0,3.0,0.0,0.0]|
|[1.0,2.0,4.0,5.0,0.0]|
|[1.0,2.0,2.0,5.0,6.0]|
+---------------------+
我的预期结果是:
+---------------------+
|newCol |
+---------------------+
|[1.0,2.0,0.0,0.0,0.0]|
|[1.0,2.0,3.0,0.0,0.0]|
|[1.0,2.0,4.0,5.0,0.0]|
|[1.0,2.0,2.0,5.0,6.0]|
+---------------------+
我应该怎么做才能获得预期的结果?
What should I do to get my expect result?
推荐答案
如果您真的想将所有矢量强制为其密集表示形式，则可以使用用户定义的函数来实现:
If you really want to coerce all vectors to their dense representation, you can do it using a User Defined Function :
val toDense = udf((v: org.apache.spark.ml.linalg.Vector) => v.toDense)
transDF.select(toDense($"newCol")).show
+--------------------+
| UDF(newCol)|
+--------------------+
|[1.0,2.0,0.0,0.0,...|
|[1.0,2.0,3.0,0.0,...|
|[1.0,2.0,4.0,5.0,...|
|[1.0,2.0,2.0,5.0,...|
+--------------------+
这篇关于如何使VectorAssembler不压缩数据?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！