Part of how table formats work. Builds on acid-on-object-storage.
The problem
Because every commit writes new immutable files (see acid-on-object-storage), tables that are written often accumulate huge numbers of tiny files. A streaming job that commits every minute creates 1,440 batches a day, and with many partitions each batch can write many files.
Why it hurts
- Opening a file has a fixed cost (a network call, reading the Parquet footer). A million 1 MB files cost vastly more to scan than a thousand 1 GB files holding the same data.
- Metadata grows: the file list in each snapshot gets longer, so planning a query gets slower.
- Spark creates a task per file or file chunk, so you get a flood of tiny tasks.
The fix: compaction
Compaction rewrites many small files into fewer large ones, targeting roughly 128 MB to 512 MB each. The table commit swaps old files for new, so readers are not disturbed.
-- Delta Lake
OPTIMIZE events WHERE event_date = '2026-11-14';
-- Iceberg
CALL catalog.system.rewrite_data_files(table => 'db.events');Good practice
- Compact on a schedule, off-peak, and only recent partitions that actually have small files.
- Reduce the problem at the source: write fewer, larger files by coalescing before a write (see shuffle-and-partitioning), or use a longer trigger interval for streaming writes.
- Follow compaction with cleanup of the old files (see the cleanup warning in acid-on-object-storage).
Watch out
- Compaction is itself a big job that reads and rewrites data. At scale it costs real compute, so measure first.
- Compacting a partition while a writer is also adding to it can cause a commit conflict. Compact partitions that are no longer being written.