The question this section answers: How do I build a correct, fault-tolerant streaming pipeline in Spark?
A streaming pipeline is correct when it gives the same answer a batch job would, even with late data and crashes. Three ideas get you there, and each page answers one of them in order:
- What is a watermark, and what happens to late data? Decides how long Spark waits for stragglers.
- How does state cleanup work in a stream-stream join? Uses watermarks to keep memory bounded when joining two streams.
- How does Spark recover from a crash without duplicates? Makes the whole thing restartable.
Plain-English terms used throughout: event time is when something happened (a timestamp in the data), as opposed to when Spark received it. State is what Spark remembers between micro-batches (a micro-batch is one small chunk of the stream processed at a time).