火花数据帧转换为大 pandas / R数据框要求

问题描述

我上运行Hadoop的纱的火花。请问这个转换工作？是否在转换之前收集（）发生？

I'm running Spark on Hadoop's YARN. How does this conversion work? Does a collect() take place before the conversion?

此外，我需要每一个从节点上安装Python和R的转换工作？我苦苦寻找本文档。

Also I need to install Python and R on every slave node for the conversion to work? I'm struggling to find documentation on this.

推荐答案

数据创建本地数据帧之前被收集。例如方法如下所示：

Data has to be collected before local data frame is created. For example toPandas method looks as follows:

def toPandas(self):
    import pandas as pd
    return pd.DataFrame.from_records(self.collect(), columns=self.columns)

您需要的Python，最佳所有的依赖关系，安装在每个节点上。

You need Python, optimally with all the dependencies, installed on each node.

SparkR对应（ as.data.frame ）仅仅是用于的别名收集。

SparkR counterpart (as.data.frame) is simply an alias for collect.

要总结这两种情况下的数据是收集来驱动节点，并转换为本地数据结构（ pandas.DataFrame 和 ::基地data.frame ）。

To summarize in both cases data is collected to the driver node and converted to the local data structure (pandas.DataFrame and base::data.frame in Python and R respectively).

这篇关于火花数据帧转换为大 pandas / R数据框要求的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！