如何使用 Spark 2 屏蔽列?

If I have payment table as shown below and want to mask name and salary columns then I get mask columns in Sequence as +----+------+--------+----------+| id | name | salary | tax_code |+----+------+--------+----------+| 1 | abcd | 12345 | KT10 |+----+------+--------+----------+| 2 | qazx | 98765 | AD12d |+----+------+--------+----------+val mask = Seq("name", "salary")我试过这样的东西 mask.foreach(c => base.withColumn(c, regexp_replace(col(c), "^.*?$", "*** Masked ***" )) ) 但它没有返回任何东西.I tried something like this mask.foreach(c => base.withColumn(c, regexp_replace(col(c), "^.*?$", "*** Masked ***" ) ) ) but it did not returned anything.感谢@philantrovert，我找到了解决方案.这是我使用的解决方案:Thanks to @philantrovert, I found out the solution. Here is the solution I used:def maskData(base: DataFrame, maskColumns: Seq[String]) = { val maskExpr = base.columns.map { col => if(maskColumns.contains(col)) s"'*** Masked ***' as ${col}" else col } base.selectExpr(maskExpr: _*)}推荐答案你的陈述 mask.foreach(c => base.withColumn(c, regexp_replace(col(c), "^.*?$", "*** Masked ***" ) ) )将返回一个听起来不太好的List[org.apache.spark.sql.DataFrame].will return a List[org.apache.spark.sql.DataFrame] which doesn't sound too good.您可以使用 selectExpr 并使用以下命令生成您的 regexp_replace 表达式:You can use selectExpr and generate your regexp_replace expression using :base.show+---+----+-----+-------+| id|name| age|address|+---+----+-----+-------+| 1|abcd|12345| KT10 || 2|qazx|98765| AD12d|+---+----+-----+-------+val mask = Seq("name", "age")val expr = df.columns.map { col => if (mask.contains(col) ) s"""regexp_replace(${col}, "^.*", "** Masked **" ) as ${col}""" else col }这将为序列 maskArray[String] = Array(id, regexp_replace(name, "^.*", "** Masked **" ) as name, regexp_replace(age, "^.*", "** Masked **" ) as age, address)现在您可以在生成的序列上使用 selectExprNow you can use selectExpr on the generated Sequencebase.selectExpr(expr: _*).show+---+------------+------------+-------+| id| name| age|address|+---+------------+------------+-------+| 1|** Masked **|** Masked **| KT10 || 2|** Masked **|** Masked **| AD12d|+---+------------+------------+-------+ 这篇关于如何使用 Spark 2 屏蔽列?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！