问题描述
StackOverflow 有一些关于如何更新数据框中嵌套列的答案.但是,其中一些看起来有点复杂.
StackOverflow has a few answers on how to update nested columns in a dataframe. However, some of them looks a bit complex.
在搜索时,我从 DataBricks 中找到了处理相同场景的文档:https://docs.databricks.com/user-guide/faq/update-nested-column.html
While searching, I found this documentation from DataBricks that handles the same scenarios: https://docs.databricks.com/user-guide/faq/update-nested-column.html
val updated = df.selectExpr("""
named_struct(
'metadata', metadata,
'items', named_struct(
'books', named_struct('fees', items.books.fees * 1.01),
'paper', items.paper
)
) as named_struct
""").select($"named_struct.metadata", $"named_struct.items")
那看起来也很干净.不幸的是,我不知道 Scala.我如何将其翻译成 Python?
That looks pretty clean as well. Unfortunately though, I don't know Scala. How would I translate this to Python?
推荐答案
这可能会帮助您入门;使用 1 行 ex 将您的 Databricks 链接转换为 python.供你探索
This might help you get started; converted your Databricks link to python using a 1 row ex. for you to explore
from pyspark.sql.functions import *
from pyspark.sql.types import *
schema = StructType()\
.add("metadata", StructType()\
.add("eventid", IntegerType(), True)\
.add("hostname", StringType(), True)\
.add("timestamp", StringType(), True))\
.add("items", StructType()\
.add("books", StructType()\
.add("fees", DoubleType(), True))\
.add("paper", StructType()\
.add("pages", IntegerType(), True)))
nested_row = [
(
{
"metadata": {
"eventid": 9,
"hostname": "999.999.999",
"timestamp": "9999-99-99 99:99:99"
},
"items": {
"books": {
"fees": 99.99
},
"paper": {
"pages": 9999
}
}
}
)
]
df = spark.createDataFrame(nested_row, schema)
df.printSchema()
df.selectExpr("""
named_struct(
'metadata', metadata,
'items', named_struct(
'books', named_struct('fees', items.books.fees * 1.01),
'paper', items.paper
)
) as named_struct
""").select(col("named_struct.metadata"), col("named_struct.items"))\
.show(truncate=False)
root
|-- metadata: struct (nullable = true)
| |-- eventid: integer (nullable = true)
| |-- hostname: string (nullable = true)
| |-- timestamp: string (nullable = true)
|-- items: struct (nullable = true)
| |-- books: struct (nullable = true)
| | |-- fees: double (nullable = true)
| |-- paper: struct (nullable = true)
| | |-- pages: integer (nullable = true)
+-------------------------------------+-----------------+
|metadata |items |
+-------------------------------------+-----------------+
|[9, 999.999.999, 9999-99-99 99:99:99]|[[99.99], [9999]]|
+-------------------------------------+-----------------+
+-------------------------------------+------------------------------+
|metadata |items |
+-------------------------------------+------------------------------+
|[9, 999.999.999, 9999-99-99 99:99:99]|[[100.98989999999999], [9999]]|
+-------------------------------------+------------------------------+
这篇关于PySpark:如何更新嵌套列?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持!