Part of finding out why a Spark job is slow. Builds on shuffle-and-partitioning and data-skew.

The strategies

StrategyHow it worksShuffles big table?Used when
Broadcast hash joinCopies the small table to every machine; joins locallyNoOne side is small
Sort-merge joinShuffles both sides by key, sorts, then mergesYes, bothDefault for two large tables
Shuffle hash joinShuffles both, builds a hash table per partitionYes, bothOne side noticeably smaller, but too big to broadcast
Broadcast nested loop / cartesianCompares every row with every rowVariesNon-equality conditions; very slow

How Spark picks

By default, Spark broadcasts a side if its size is below spark.sql.autoBroadcastJoinThreshold (10 MB by default). Otherwise it uses sort-merge for equality joins. The decision depends on Spark’s size estimates, which can be wrong when statistics are missing or after heavy filtering.

With Adaptive Query Execution on, Spark can switch a sort-merge join to a broadcast join at runtime once it sees the real size of a side after the shuffle.

When to override it

If you know a table is small, tell Spark:

import static org.apache.spark.sql.functions.broadcast;
 
Dataset<Row> joined = orders.join(broadcast(countries), "country_code");

Broadcasting is often the single biggest speedup for joining a large fact table with a small dimension table.

Watch out

  • A broadcast table must fit in memory on the driver and on every executor. Broadcasting something that is too big causes out-of-memory failures. Raising the threshold to hundreds of MB is risky at scale.
  • Check the plan, not your assumption. Run df.explain() and look for BroadcastHashJoin vs SortMergeJoin, or open the SQL tab in the Spark UI.
  • Non-equality joins (a.time BETWEEN b.start AND b.end) cannot use a hash join and can explode in cost. Add an equality key, such as a date bucket, wherever possible.