Part of finding out why a Spark job is slow. Builds on shuffle-and-partitioning and data-skew.
The strategies
| Strategy | How it works | Shuffles big table? | Used when |
|---|---|---|---|
| Broadcast hash join | Copies the small table to every machine; joins locally | No | One side is small |
| Sort-merge join | Shuffles both sides by key, sorts, then merges | Yes, both | Default for two large tables |
| Shuffle hash join | Shuffles both, builds a hash table per partition | Yes, both | One side noticeably smaller, but too big to broadcast |
| Broadcast nested loop / cartesian | Compares every row with every row | Varies | Non-equality conditions; very slow |
How Spark picks
By default, Spark broadcasts a side if its size is below spark.sql.autoBroadcastJoinThreshold (10 MB by default). Otherwise it uses sort-merge for equality joins. The decision depends on Spark’s size estimates, which can be wrong when statistics are missing or after heavy filtering.
With Adaptive Query Execution on, Spark can switch a sort-merge join to a broadcast join at runtime once it sees the real size of a side after the shuffle.
When to override it
If you know a table is small, tell Spark:
import static org.apache.spark.sql.functions.broadcast;
Dataset<Row> joined = orders.join(broadcast(countries), "country_code");Broadcasting is often the single biggest speedup for joining a large fact table with a small dimension table.
Watch out
- A broadcast table must fit in memory on the driver and on every executor. Broadcasting something that is too big causes out-of-memory failures. Raising the threshold to hundreds of MB is risky at scale.
- Check the plan, not your assumption. Run
df.explain()and look forBroadcastHashJoinvsSortMergeJoin, or open the SQL tab in the Spark UI. - Non-equality joins (
a.time BETWEEN b.start AND b.end) cannot use a hash join and can explode in cost. Add an equality key, such as a date bucket, wherever possible.