问题描述
我正在使用 pyspark 并且我有一个数据帧对象 df
,这就是 df.printSchema()
的输出看起来像
I am using pyspark and I have a dataframe object df
and this is what the output of df.printSchema()
looks like
root
|-- M_MRN: string (nullable = true)
|-- measurements: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- Observation_ID: string (nullable = true)
| | |-- Observation_Name: string (nullable = true)
| | |-- Observation_Result: string (nullable = true)
我想过滤掉测量"中 Observation_ID 不是5"或10"的所有数组.所以目前当我运行 df.select('measurements').take(2)
我得到
I would like to filter out all the arrays in 'measurements' where the Observation_ID is not '5' or '10'. So currently when I run df.select('measurements').take(2)
I get
[Row(measurements=[Row(Observation_ID='5', Observation_Name='ABC', Observation_Result='108/72'),
Row(Observation_ID='11', Observation_Name='ABC', Observation_Result='70'),
Row(Observation_ID='10', Observation_Name='ABC', Observation_Result='73.029'),
Row(Observation_ID='14', Observation_Name='XYZ', Observation_Result='23.1')]),
Row(measurements=[Row(Observation_ID='2', Observation_Name='ZZZ', Observation_Result='3/4'),
Row(Observation_ID='5', Observation_Name='ABC', Observation_Result='7')])]
我希望在完成上述过滤并运行 df.select('measurements').take(2)
之后我得到
I would like that after I do the above filtering and run df.select('measurements').take(2)
I get
[Row(measurements=[Row(Observation_ID='5', Observation_Name='ABC', Observation_Result='108/72'),
Row(Observation_ID='10', Observation_Name='ABC', Observation_Result='73.029')]),
Row(measurements=[Row(Observation_ID='5', Observation_Name='ABC', Observation_Result='7')])]
有没有办法在 pyspark 中做到这一点?感谢您的帮助!
Is there a way to do this in pyspark? Thank you in anticipation for your help!
推荐答案
从 Spark 2.4 开始,您可以使用高阶函数 FILTER
从数组中过滤掉元素.所以如果你想删除 Observation_ID
不是 5 或 10 的元素,你可以这样做:
Since Spark 2.4, you can use Higher Order Function FILTER
to filter out elements from an array. So if you want to remove elements where Observation_ID
is not 5 or 10, you can do it as follows:
from pyspark.sql.functions import expr
df.withColumn('measurements', expr("FILTER(measurements, x -> x.Observation_ID = 5 OR x.Observation_ID = 10)"))
这篇关于使用 PySpark 删除 spark 数据帧中嵌套结构中的行(文本中的详细信息)的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持!