使用 Spark DataFrame 列制作直方图

问题描述

我正在尝试使用数据框中的列制作直方图，看起来像

I am trying to make a histogram with a column from a dataframe which looks like

DataFrame[C0: int, C1: int, ...]

如果我用 C1 列制作直方图，我该怎么做?

If I were to make a histogram with the column C1, what should I do?

我尝试过的一些事情是

df.groupBy("C1").count().histogram()
df.C1.countByValue()

由于数据类型不匹配而不起作用.

Which do not work because of mismatch in data types.

推荐答案

pyspark_dist_explore 包非常好.如果您不想添加额外的依赖项，您可以使用这段代码绘制一个简单的直方图.

The pyspark_dist_explore package that @Chris van den Berg mentioned is quite nice. If you prefer not to add an additional dependency you can use this bit of code to plot a simple histogram.

import matplotlib.pyplot as plt
# Show histogram of the 'C1' column
bins, counts = df.select('C1').rdd.flatMap(lambda x: x).histogram(20)

# This is a bit awkward but I believe this is the correct way to do it 
plt.hist(bins[:-1], bins=bins, weights=counts)

这篇关于使用 Spark DataFrame 列制作直方图的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！