如何计算数据框中每一列每个不同值的出现?

问题描述

edf.select("x").distinct.show()显示edf DataFrame的x列中存在的不同值.

edf.select("x").distinct.show() shows the distinct values that are present in x column of edf DataFrame.

是否存在一种有效的方法来显示这些不同值在数据帧中出现的次数? (计算每个不同的值)

Is there an efficient method to also show the number of times these distinct values occur in the data frame? (count for each distinct value)

推荐答案

countDistinct可能是首选:

import org.apache.spark.sql.functions.countDistinct

df.agg(countDistinct("some_column"))

如果速度比精度更重要，则可以考虑approx_count_distinct(Spark 1.x中的approxCountDistinct):

If speed is more important than the accuracy you may consider approx_count_distinct (approxCountDistinct in Spark 1.x):

import org.apache.spark.sql.functions.approx_count_distinct

df.agg(approx_count_distinct("some_column"))

要获取值和计数:

df.groupBy("some_column").count()

在SQL(spark-sql)中:

SELECT COUNT(DISTINCT some_column) FROM df

和

SELECT approx_count_distinct(some_column) FROM df

这篇关于如何计算数据框中每一列每个不同值的出现?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！