Part of finding out why a Spark job is slow. Builds on shuffle-and-partitioning. Next: join-strategies.
The problem
Skew means the data is spread unevenly. If one customer_id owns 40% of the rows, the shuffle sends all of them to one partition, and that single task does 40% of the work while the rest finish early. The whole stage waits for it.
How to spot it
In the Spark UI stage page, compare task durations. If the median task takes 10 seconds and the max takes 10 minutes, or one task reads vastly more shuffle data than the others, it is skew.
Fixes, simplest first
- Turn on Adaptive Query Execution’s skew handling. In Spark 3.x,
spark.sql.adaptive.enabled=trueandspark.sql.adaptive.skewJoin.enabled=truelet Spark split oversized partitions during a sort-merge join automatically. Try this first. - Remove bad keys. Often the hot key is
nullor a placeholder like"unknown". Filter it out, or handle it separately. - Broadcast the smaller side. If one table is small, no shuffle of the big table is needed. See join-strategies.
- Salting, the manual fix. Add a random suffix to the hot key so its rows spread over several partitions, and replicate the other side to match:
int salts = 8;
Dataset<Row> bigSalted = big.withColumn("salt", expr("floor(rand() * " + salts + ")"));
Dataset<Row> smallExploded = small
.withColumn("salt", explode(sequence(lit(0), lit(salts - 1))));
Dataset<Row> joined = bigSalted.join(smallExploded, new String[]{"key", "salt"}, "inner");Salting multiplies the smaller side by the salt count, and it makes the code harder to read. Use it only when the first three do not work.
Hard-to-reverse note
Salting changes the shape of your pipeline logic. Keep it behind a clearly named function with a comment explaining why, or the next person will remove it.