如何从PySpark中的数据框获取架构定义?

问题描述

在PySpark中，您可以定义一个架构并使用此预定义的架构读取数据源，例如. g.:

In PySpark it you can define a schema and read data sources with this pre-defined schema, e. g.:

Schema = StructType([ StructField("temperature", DoubleType(), True),
                      StructField("temperature_unit", StringType(), True),
                      StructField("humidity", DoubleType(), True),
                      StructField("humidity_unit", StringType(), True),
                      StructField("pressure", DoubleType(), True),
                      StructField("pressure_unit", StringType(), True)
                    ])

对于某些数据源，可以从数据源推断模式，并获得具有此模式定义的数据框.

For some datasources it is possible to infer the schema from the data-source and get a dataframe with this schema definition.

是否可以从以前推断数据的数据帧中获取模式定义(以上述形式)?

Is it possible to get the schema definition (in the form described above) from a dataframe, where the data has been inferred before?

df.printSchema()将架构打印为树，但是我需要重用架构，如上定义，所以我可以从另一个数据源读取具有先前推断出的该架构的数据源./p>

df.printSchema() prints the schema as a tree, but I need to reuse the schema, having it defined as above,so I can read a data-source with this schema that has been inferred before from another data-source.

推荐答案

是可以的.使用 property

Yes it is possible. Use DataFrame.schema property

以pyspark.sql.types.StructType返回此DataFrame的架构.

Returns the schema of this DataFrame as a pyspark.sql.types.StructType.

>>> df.schema
StructType(List(StructField(age,IntegerType,true),StructField(name,StringType,true)))

1.3版中的新功能.

New in version 1.3.

模式也可以导出为JSON并重新导入(如果需要).

这篇关于如何从PySpark中的数据框获取架构定义?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！