什么是导致 Shuffle 的 Spark 转换?

问题描述

我很难在 Spark 文档中找到导致 shuffle 的操作和不会的操作.在这个列表中，哪些会导致洗牌，哪些不会?

I have trouble to find in the Spark documentation operations that causes a shuffle and operation that does not. In this list, which ones does cause a shuffle and which ones does not?

映射和过滤器没有.但是，我不确定其他人.

Map and filter does not. However, I am not sure with the others.

map(func)
filter(func)
flatMap(func)
mapPartitions(func)
mapPartitionsWithIndex(func)
sample(withReplacement, fraction, seed)
union(otherDataset)
intersection(otherDataset)
distinct([numTasks]))
groupByKey([numTasks])
reduceByKey(func, [numTasks])
aggregateByKey(zeroValue)(seqOp, combOp, [numTasks])
sortByKey([ascending], [numTasks])
join(otherDataset, [numTasks])
cogroup(otherDataset, [numTasks])
cartesian(otherDataset)
pipe(command, [envVars])
coalesce(numPartitions)

推荐答案

实际上很容易找到这一点，无需文档.对于这些函数中的任何一个，只需创建一个 RDD 并调用调试字符串，这是一个示例，您可以自己完成其余的工作.

It is actually extremely easy to find this out, without the documentation. For any of these functions just create an RDD and call to debug string, here is one example you can do the rest on ur own.

scala> val a  = sc.parallelize(Array(1,2,3)).distinct
scala> a.toDebugString
MappedRDD[5] at distinct at <console>:12 (1 partitions)
  MapPartitionsRDD[4] at distinct at <console>:12 (1 partitions)
    **ShuffledRDD[3] at distinct at <console>:12 (1 partitions)**
      MapPartitionsRDD[2] at distinct at <console>:12 (1 partitions)
        MappedRDD[1] at distinct at <console>:12 (1 partitions)
          ParallelCollectionRDD[0] at parallelize at <console>:12 (1 partitions)

如你所见，distinct 创建了一个 shuffle.找出这种方式而不是文档也特别重要，因为在某些情况下，某些功能需要或不需要 shuffle.例如，join 通常需要 shuffle，但如果你加入两个 RDD，来自同一个 RDD spark 的分支有时会忽略 shuffle.

So as you can see distinct creates a shuffle. It is also particularly important to find out this way rather than docs because there are situations where a shuffle will be required or not required for a certain function. For example join usually requires a shuffle but if you join two RDD's that branch from the same RDD spark can sometimes elide the shuffle.

这篇关于什么是导致 Shuffle 的 Spark 转换?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！