The question this section answers: How do I build a correct, fault-tolerant streaming pipeline in Spark?

A streaming pipeline is correct when it gives the same answer a batch job would, even with late data and crashes. Three ideas get you there, and each page answers one of them in order:

Plain-English terms used throughout: event time is when something happened (a timestamp in the data), as opposed to when Spark received it. State is what Spark remembers between micro-batches (a micro-batch is one small chunk of the stream processed at a time).