Part of building a correct streaming pipeline. Builds on watermarks-and-late-data. Next: checkpoints-and-exactly-once.

The problem

To join two streams, Spark must remember rows from each side while it waits for matches. How does it know when a remembered row can be forgotten? Is it the watermark, the time condition in the join, or both?

Terms

  • State: the rows Spark keeps between micro-batches so it can match them with rows that arrive later.
  • Join condition: the ON clause. A time range inside it (for example “click within 1 hour after the impression”) says how long a row can still find a match.

The answer

Both, and they do different jobs:

  1. The watermark says how much time has passed. It also drops incoming rows that are too late.
  2. The time condition says how long a buffered row stays useful. Combined with the watermark, Spark derives a cleanup rule for each side: rows older than X can never match again, so delete them.

With neither, state grows forever, which on a big stream ends in out-of-memory errors or a huge checkpoint.

Example

Dataset<Row> impressions = spark.readStream()/* ... */
    .withWatermark("impressionTime", "2 hours");
Dataset<Row> clicks = spark.readStream()/* ... */
    .withWatermark("clickTime", "3 hours");
 
Dataset<Row> joined = impressions.join(
    clicks,
    expr("clickAdId = impressionAdId AND " +
         "clickTime >= impressionTime AND " +
         "clickTime <= impressionTime + interval 1 hour"));

An impression at 10:00 can only match a click up to 11:00. Once the watermark passes 11:00, no future click can match it, so Spark deletes it from state.

A mental model that helped

  • One rule decides which incoming rows are worth buffering: rows later than the watermark are dropped.
  • Another rule decides which already buffered rows are no longer needed: that comes from the time condition plus the watermark.

Global watermark

Each stream has its own watermark, but by default Spark uses one global watermark, the minimum across all inputs, so it never runs ahead of the slowest stream. The setting is spark.sql.streaming.multipleWatermarkPolicy (min by default, or max).

Scale notes

  • State size is roughly event rate multiplied by the retention window. A wider time range or a bigger delay means more memory.
  • In outer joins, an unmatched row can be emitted only after the watermark passes, so results arrive late.
  • An idle input stream holds the global watermark back and delays cleanup everywhere.