Part of building a correct streaming pipeline. Builds on stream-stream-join-state.
The problem
Your streaming job dies halfway through processing a batch. When it restarts, how does it avoid skipping data or writing the same results twice?
The answer: checkpoints plus a replayable source
You give the query a checkpoint location, a folder on reliable storage (HDFS or S3, not local disk):
StreamingQuery q = result.writeStream()
.format("parquet")
.option("path", "s3://bucket/out/")
.option("checkpointLocation", "s3://bucket/checkpoints/orders/")
.start();Spark saves three things there:
- Offsets: before processing a batch, it writes down which range of the source it is about to read. After a crash it re-reads exactly that range.
- Commits: after a batch finishes, it records that the batch is done, so it is not redone.
- State: the contents of aggregations and joins (see stream-stream-join-state), so they continue instead of starting empty.
What “exactly-once” really needs
Spark can promise end-to-end exactly-once only when all three hold:
- The source is replayable, meaning it can re-serve the same offsets (Kafka and files can; a plain socket cannot).
- Spark’s processing is deterministic enough that a re-run produces the same output.
- The sink is idempotent (writing the same thing twice has no extra effect) or transactional. The built-in file sink tracks which files belong to which batch, so retries do not duplicate.
For your own sinks, use foreachBatch. It hands you a batchId, which you can use to make writes safe to repeat:
result.writeStream().foreachBatch((batchDf, batchId) -> {
// upsert keyed by (batchId, row key) so a retried batch overwrites itself
}).start();Watch out (hard to reverse)
- Never delete or reuse a checkpoint casually. Deleting it makes the job start from scratch: it reprocesses the source or loses its place, and state is gone.
- Some code changes are incompatible with an old checkpoint, such as changing the stateful operators or their keys. Plan this before production, and test upgrades on a copy.
- Checkpoint size grows with state size, so the retention rules in watermarks-and-late-data matter here too.