Hive Table: (Name_Age: Map[String, Int] and ID: Int)
+---------------------------------------------------------++------+
| Name_Age || ID |
+---------------------------------------------------------++------+
|"SUBHAJIT SEN":28,"BINOY MONDAL":26,"SHANTANU DUTTA":35 || 15 |
|"GOBINATHAN SP":35,"HARSH GUPTA":27,"RAHUL ANAND":26 || 16 |
+---------------------------------------------------------++------+
我已将 Name_Age 列分解为多行:
def toUpper(name: Seq[String]) = (name.map(a => a.toUpperCase)).toSeq
sqlContext.udf.register("toUpper",toUpper _)
var df = sqlContext.sql("SELECT toUpper(name) FROM namelist").toDF("Name_Age")
df.explode(df("Name_Age")){case org.apache.spark.sql.Row(arr: Seq[String]) => arr.toSeq.map(v => Tuple1(v))}.drop(df("Name_Age")).withColumnRenamed("_1","Name_Age")
+-------------------+
| Name_Age |
+-------------------+
| [SUBHAJIT SEN,28]|
| [BINOY MONDAL,26]|
|[SHANTANU DUTTA,35]|
| [GOBINATHAN SP,35]|
| [HARSH GUPTA,27]|
| [RAHUL ANAND,26]|
+-------------------+
但我想爆炸并创建 2 行:姓名和年龄
+-------------------+-------+
| Name | Age |
+-------------------+-------+
| SUBHAJIT SEN | 28 |
| BINOY MONDAL | 26 |
|SHANTANU DUTTA | 35 |
| GOBINATHAN SP | 35 |
| HARSH GUPTA | 27 |
| RAHUL ANAND | 26 |
+-------------------+-------+
有人可以帮忙修改爆炸代码吗?
最佳答案
您只需要删除 toUpper
调用 explode
函数:
import org.apache.spark.sql.functions.explode
val df = Seq((Map("foo" -> 1, "bar" -> 2), 1)).toDF("name_age", "id")
val exploded = df.select($"id", explode($"name_age")).toDF("id", "name", "age")
exploded.printSchema
// root
// |-- id: integer (nullable = false)
// |-- name: string (nullable = false)
// |-- age: integer (nullable = false)
之后您可以使用内置函数转换为大写:
import org.apache.spark.sql.functions.upper
exploded.withColumn("name", upper($"name"))
关于scala - 我有一个 Map 作为列数据类型的表,如何分解它以生成 2 列,一列用于 map ,另一列用于键?,我们在Stack Overflow上找到一个类似的问题:https://stackoverflow.com/questions/36649326/