本文介绍了如何在spark数据框中展平结构?的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!
问题描述
我有一个具有以下结构的数据框:
I have a dataframe with the following structure:
|-- data: struct (nullable = true)
| |-- id: long (nullable = true)
| |-- keyNote: struct (nullable = true)
| | |-- key: string (nullable = true)
| | |-- note: string (nullable = true)
| |-- details: map (nullable = true)
| | |-- key: string
| | |-- value: string (valueContainsNull = true)
如何展平结构并创建一个新的数据帧:
How it is possible to flatten the structure and create a new dataframe:
|-- id: long (nullable = true)
|-- keyNote: struct (nullable = true)
| |-- key: string (nullable = true)
| |-- note: string (nullable = true)
|-- details: map (nullable = true)
| |-- key: string
| |-- value: string (valueContainsNull = true)
是否有类似爆炸的东西,但对于结构?
Is there something like explode, but for structs?
推荐答案
这应该适用于Spark 1.6或更高版本:
This should work in Spark 1.6 or later:
df.select(df.col("data.*"))
或
df.select(df.col("data.id"), df.col("data.keyNote"), df.col("data.details"))
这篇关于如何在spark数据框中展平结构?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持!