Part of how table formats work. Next: small-files-and-compaction.
The problem
Object storage like S3 or HDFS only lets you add or delete whole files. There is no “update this row” and no way to change many files at once. So how can a table commit be all-or-nothing, with readers never seeing half a write?
The answer: never edit files, only change what the table points to
A table is defined by metadata: a list of exactly which data files belong to it right now. Data files are immutable (never modified after being written). A write goes like this:
- Write the new Parquet data files. At this point nobody can see them, because no table metadata mentions them.
- Create a new snapshot: a version of the file list that includes the new files (and excludes any removed ones).
- Commit by atomically publishing that snapshot as the table’s current version. This single step is what makes it all-or-nothing.
How step 3 is atomic differs by format:
- Delta Lake adds the next numbered file in a
_delta_logfolder (like000042.json). Creating it succeeds for only one writer. - Iceberg swaps a pointer to the current metadata file, using an atomic update in a catalog.
A reader first loads the current snapshot, then reads only the files it lists. A half-finished write is invisible, and the reader keeps a consistent view even while writes continue.
Concurrent writers
Formats use optimistic concurrency: each writer assumes nobody else is changing the same data, does its work, and checks at commit time. If another commit got in first, the writer checks for a real conflict (for example both touched the same files). With no conflict it retries on top of the new version. With a conflict, it fails and you retry the job.
What you get
- Updates and deletes: rewrite the affected files, and commit a snapshot that swaps old for new.
- Time travel: old snapshots still list old files, so you can query the table as of an earlier version.
- Safe reruns: a failed job leaves orphan files but no corrupted table.
Watch out
- Old files are not deleted at commit. You must run cleanup (
VACUUMin Delta,expire_snapshotsin Iceberg) or storage grows forever. Cleaning up also removes time travel to those versions, which is hard to undo. - Many concurrent writers to the same partitions can cause repeated conflicts. Partition your writes so jobs touch different data.