Part of building a correct streaming pipeline. Next: stream-stream-join-state.

The problem

Events arrive out of order. A click that happened at 10:01 might reach Spark at 10:09. If you are counting clicks per 10:00-10:05 window, when is it safe to say that window is final? Without a rule, Spark would have to keep every window open forever.

The answer

A watermark is Spark’s running estimate of how far event time has progressed:

watermark = (latest event time seen so far) - (delay you allow)

You set the delay yourself:

Dataset<Row> counts = events
    .withWatermark("eventTime", "10 minutes")
    .groupBy(window(col("eventTime"), "5 minutes"), col("userId"))
    .count();

With a 10 minute delay, once the latest event seen is 10:20, the watermark is 10:10. Windows that ended before 10:10 are considered complete.

What the watermark does

  1. Closes windows. In append output mode, a window’s result is emitted only after the watermark passes its end. Results are therefore delayed by at least the watermark delay.
  2. Frees state. Once a window is closed, Spark drops its aggregate from memory.
  3. Drops very late data. An event older than the current watermark may be discarded. Spark guarantees data within the delay is processed, not that data beyond it is dropped immediately.

Choosing the delay

It is a trade-off, not a free setting:

Longer delayShorter delay
Fewer dropped eventsMore dropped events
Slower resultsFaster results
More state in memoryLess state in memory

Measure how late your data really is (for example the 99th percentile gap between event time and arrival time) and start there. State size is roughly event rate multiplied by the delay window, so check it against your volume before production.

Watch out

  • The watermark only moves when new data arrives. A quiet stream holds it still and delays results.
  • With several input streams, Spark uses one global watermark, the minimum across them. See stream-stream-join-state.