Part of finding out why a Spark job is slow. Next: data-skew.

Terms

  • Partition: one slice of your dataset. Spark runs one task per partition, so partitions are the unit of parallelism.
  • Shuffle: redistributing data across machines so that rows with the same key end up together. It is needed for groupBy, join, distinct, repartition and similar operations.

Why it is expensive

A shuffle writes data to local disk on every machine, sends it over the network, and reads it back. That is disk plus network plus serialization, which is far slower than work done in memory on one machine. Jobs with many shuffles of large data are slow almost by definition.

What you can control

  1. Do fewer shuffles. Filter and select columns before a wide operation, so less data moves. Spark’s optimizer helps, but not always.
  2. Right-size the partitions. Aim for roughly 128-256 MB each. Too few leaves cores idle; too many wastes time on scheduling. After a shuffle the count is spark.sql.shuffle.partitions, which defaults to 200 and is often wrong for very small or very large jobs.
  3. Pick the right tool to change partition count:
repartition(n)coalesce(n)
ShufflesYesNo
Can increase countYesNo
Even sizesYesNot guaranteed
Typical useFix skew, raise parallelismShrink output files before a write
// Spread evenly by key before a heavy operation
Dataset<Row> spread = df.repartition(400, col("user_id"));
// Fewer output files, no shuffle
spread.coalesce(50).write().parquet("s3://bucket/out/");

Adaptive Query Execution (spark.sql.adaptive.enabled, on by default in Spark 3.2+) can merge small shuffle partitions at runtime, so you tune less by hand.

How to see it

In the Spark UI, open the stage and check Shuffle Read/Write size. A stage that shuffles hundreds of GB is your first suspect.