如何从scala的标量中减去向量?

问题描述

我有一个实木复合地板文件，其中包含两列(id，特征).我想从标量中减去特征，然后将输出除以另一个标量.镶木文件

I have parquet file which contain two columns (id,features).I want to subtract features from scalar and divide output by another scalar.parquet file

df.withColumn("features", ((df("features")-constant1)/constant2))

但是给我错误

推荐答案

我的scala火花代码如下.对向量sparkm数据类型执行任何操作的唯一方法是强制转换为字符串.还使用UDF进行减法和除法.

My scala spark code to this as below . Only way to do any operation on vector sparkm datatype is casting to string. Also used UDF to perform subtraction and division.

import spark.implicits._
import org.apache.spark.ml.linalg.Vectors
import org.apache.spark.sql.functions.udf
import org.apache.spark.sql.functions._
var df = Seq((1, Vectors.dense(35)),
  (2, Vectors.dense(45)),
  (3, Vectors.dense(4.5073)),
  (4, Vectors.dense(56)))
  .toDF("id", "features")
df.show()
val constant1 = 10
val constant2 = 2
val performComputation = (s: Double, val1: Int, val2: Int) => {
  Vectors.dense((s - val1) / val2)
}
val performComputationUDF = udf(performComputation)
df.printSchema()
df = df.withColumn("features",
  regexp_replace(df.col("features").cast("String"),
    "[\\[\\]]", "").cast("Double")
)

df = df.withColumn("features",
  performComputationUDF(df.col("features"),
    lit(constant1), lit(constant2))
)
df.show(20, false)

// Write State should with mode overwrite
df.write
  .mode("overwrite")
  .parquet("file:///usr/local/spark/dataset/output1/")

结果

+---+----------+
|id |features  |
+---+----------+
|1  |[12.5]    |
|2  |[17.5]    |
|3  |[-2.74635]|
|4  |[23.0]    |
+---+----------+

这篇关于如何从scala的标量中减去向量?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！