我正在尝试在pyspark中拆分数据框
这是我的数据
df = sc.parallelize([[1, 'Foo|10'], [2, 'Bar|11'], [3,'Car|12']]).toDF(['Key', 'Value'])
df = df.withColumn('Splitted', split(df['Value'], '|')[0])
我有
+-----+---------+-----+
|Key|Value|Splitted |
+-----+---------+-----+
| 1| Food|10| F|
| 2| Bar|11 | B|
| 3| Caring 12| C|
+-----+---------+-----+
但是我想要
+-----+---------+-----+
|Key | Value|Splitted|
+-----+---------+-----+
| 1| 10| Food |
| 2| 11| Bar |
| 3| 12|Caring |
+-----+---------+-----+
有人可以指出我做错了什么吗？
What if i have a unique situation like this?
df = sc.parallelize([[1, 'Foo|10|we'], [2, 'Bar|11|we'], [3,'Car|12|we']]).toDF(['Key', 'Value'])
+---+---------+
|Key| Value|
+---+---------+
| 1|Foo|10|we|
| 2|Bar|11|we|
| 3|Car|12|we|
+---+---------+
最佳答案
您忘记了escape字符，应将转义字符包括为
df = df.withColumn('Splitted', split(df['Value'], '\|')[0])
如果要输出为
+---+-----+--------+
|Key|Value|Splitted|
+---+-----+--------+
|1 |10 |Foo |
|2 |11 |Bar |
|3 |12 |Car |
+---+-----+--------+
你应该做
from pyspark.sql import functions as F
df = df.withColumn('Splitted', F.split(df['Value'], '\|')).withColumn('Value', F.col('Splitted')[1]).withColumn('Splitted', F.col('Splitted')[0])
关于python - 在pyspark中拆分列，我们在Stack Overflow上找到一个类似的问题：https://stackoverflow.com/questions/48790246/