The question this section answers: Why is my Spark job slow, and how do I find out?
Nearly every slow Spark job comes down to moving too much data, moving it unevenly, or joining it the wrong way. Start by opening the Spark UI and looking at the slowest stage (a group of tasks that run without moving data between machines). Then read these in order:
- What is a shuffle, and why is it expensive? Where the time usually goes.
- What is data skew, and how do I fix it? Why one task takes ten times longer than the rest.
- How does Spark choose how to join two tables? The biggest lever you control.
Scale assumption: the advice here is aimed at jobs that process tens of GB and up. On small data, none of this matters much.