Part of building a correct streaming pipeline. Builds on stream-stream-join-state.

The problem

Your streaming job dies halfway through processing a batch. When it restarts, how does it avoid skipping data or writing the same results twice?

The answer: checkpoints plus a replayable source

You give the query a checkpoint location, a folder on reliable storage (HDFS or S3, not local disk):

StreamingQuery q = result.writeStream()
    .format("parquet")
    .option("path", "s3://bucket/out/")
    .option("checkpointLocation", "s3://bucket/checkpoints/orders/")
    .start();

Spark saves three things there:

  1. Offsets: before processing a batch, it writes down which range of the source it is about to read. After a crash it re-reads exactly that range.
  2. Commits: after a batch finishes, it records that the batch is done, so it is not redone.
  3. State: the contents of aggregations and joins (see stream-stream-join-state), so they continue instead of starting empty.

What “exactly-once” really needs

Spark can promise end-to-end exactly-once only when all three hold:

  • The source is replayable, meaning it can re-serve the same offsets (Kafka and files can; a plain socket cannot).
  • Spark’s processing is deterministic enough that a re-run produces the same output.
  • The sink is idempotent (writing the same thing twice has no extra effect) or transactional. The built-in file sink tracks which files belong to which batch, so retries do not duplicate.

For your own sinks, use foreachBatch. It hands you a batchId, which you can use to make writes safe to repeat:

result.writeStream().foreachBatch((batchDf, batchId) -> {
    // upsert keyed by (batchId, row key) so a retried batch overwrites itself
}).start();

Watch out (hard to reverse)

  • Never delete or reuse a checkpoint casually. Deleting it makes the job start from scratch: it reprocesses the source or loses its place, and state is gone.
  • Some code changes are incompatible with an old checkpoint, such as changing the stateful operators or their keys. Plan this before production, and test upgrades on a copy.
  • Checkpoint size grows with state size, so the retention rules in watermarks-and-late-data matter here too.