问题描述
在执行 groupBy
时,是否可以将聚合函数应用于数据框的所有(或列表)列?换句话说,有一种方法可以避免对每一列执行此操作:
Is there a way to apply an aggregate function to all (or a list of) columns of a dataframe, when doing a groupBy
? In other words, is there a way to avoid doing this for every column:
df.groupBy("col1")
.agg(sum("col2").alias("col2"), sum("col3").alias("col3"), ...)
推荐答案
将聚合函数应用于多个列有多种方法。
There are multiple ways of applying aggregate functions to multiple columns.
GroupedData
类为最常用的功能提供了许多方法,包括 count
,最大
,最小
,平均值
和 sum
,可以直接如下使用:
GroupedData
class provides a number of methods for the most common functions, including count
, max
, min
, mean
and sum
, which can be used directly as follows:
-
Python:
Python:
df = sqlContext.createDataFrame(
[(1.0, 0.3, 1.0), (1.0, 0.5, 0.0), (-1.0, 0.6, 0.5), (-1.0, 5.6, 0.2)],
("col1", "col2", "col3"))
df.groupBy("col1").sum()
## +----+---------+-----------------+---------+
## |col1|sum(col1)| sum(col2)|sum(col3)|
## +----+---------+-----------------+---------+
## | 1.0| 2.0| 0.8| 1.0|
## |-1.0| -2.0|6.199999999999999| 0.7|
## +----+---------+-----------------+---------+
Scala
Scala
val df = sc.parallelize(Seq(
(1.0, 0.3, 1.0), (1.0, 0.5, 0.0),
(-1.0, 0.6, 0.5), (-1.0, 5.6, 0.2))
).toDF("col1", "col2", "col3")
df.groupBy($"col1").min().show
// +----+---------+---------+---------+
// |col1|min(col1)|min(col2)|min(col3)|
// +----+---------+---------+---------+
// | 1.0| 1.0| 0.3| 0.0|
// |-1.0| -1.0| 0.6| 0.2|
// +----+---------+---------+---------+
您也可以传递应汇总的列的列表
Optionally you can pass a list of columns which should be aggregated
df.groupBy("col1").sum("col2", "col3")
您还可以通过字典/带有a列的键和值作为函数的映射:
You can also pass dictionary / map with columns a the keys and functions as the values:
-
Python
Python
exprs = {x: "sum" for x in df.columns}
df.groupBy("col1").agg(exprs).show()
## +----+---------+
## |col1|avg(col3)|
## +----+---------+
## | 1.0| 0.5|
## |-1.0| 0.35|
## +----+---------+
Scala
Scala
val exprs = df.columns.map((_ -> "mean")).toMap
df.groupBy($"col1").agg(exprs).show()
// +----+---------+------------------+---------+
// |col1|avg(col1)| avg(col2)|avg(col3)|
// +----+---------+------------------+---------+
// | 1.0| 1.0| 0.4| 0.5|
// |-1.0| -1.0|3.0999999999999996| 0.35|
// +----+---------+------------------+---------+
最后,您可以使用varargs:
Finally you can use varargs:
-
Python
Python
from pyspark.sql.functions import min
exprs = [min(x) for x in df.columns]
df.groupBy("col1").agg(*exprs).show()
Scala
Scala
import org.apache.spark.sql.functions.sum
val exprs = df.columns.map(sum(_))
df.groupBy($"col1").agg(exprs.head, exprs.tail: _*)
还有其他一些方法可以达到类似的效果,但是在大多数情况下这些效果应该绰绰有余。
There are some other way to achieve a similar effect but these should more than enough most of the time.
另请参见:
- Multiple Aggregate operations on the same column of a spark dataframe
这篇关于Spark SQL:将聚合函数应用于列列表的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持!