Databases / storage deep dive

Iceberg v3 deletion vectors: one bitmap, one data file, one snapshot rule

An Iceberg table keeps data files immutable, but real workloads still need updates and deletes. Version 3 answers with a deletion vector: a compressed bitmap of deleted row positions, stored as a Puffin blob and attached to exactly one data file through manifest metadata.

Databases / storage deep dive · immutable files, position deletes, Roaring bitmaps, Puffin, manifest planning, sequence numbers, and production debugging

2026-09-01

The concrete problem

Columnar data files such as Parquet are excellent for analytics because they are immutable, independently readable, and easy to place in object storage. They are awkward for a small update. If row 17 in orders-0001.parquet must disappear, rewriting a large file just to remove one row is wasteful.

Iceberg v2 solved this with position-delete files. A delete file stores pairs such as (orders-0001.parquet, 17). That preserves the original data file, but a busy table can accumulate many delete files. Every scan must discover the relevant files, compare sequence numbers, match partitions, and merge the positions while reading the data file. The logical answer is simple; the metadata and execution path are not.

On August 31, 2026, Amazon Redshift announced read/write support for Iceberg v3 and specifically described deletion vectors as compact compressed bitmaps for high-frequency update and delete workloads. The important engineering idea is not the product flag. It is the boundary: the bitmap is scoped to one immutable file, while the manifest and snapshot still decide when that bitmap applies.

From v1 to v3

“Replaced” does not mean old v2 position deletes vanish instantly. A v2 table upgraded to v3 may contain existing position-delete files. When a writer creates a DV for a data file, it must merge the old position deletes for that file into the bitmap. Position deletes that cover several data files may remain until each target is converted.

The on-disk data flow

Iceberg separates the bytes from the table state. The bitmap bytes live in a Puffin file, which can contain multiple typed blobs. A delete manifest carries the locator and scope needed to find that blob. A reader first plans the data file and applicable delete metadata, then fetches the DV content and filters rows by position.

The commit is still optimistic and snapshot-based. A reader that already loaded snapshot S sees the old set of data and delete files. A reader that loads snapshot S+1 sees the new manifest state. The DV does not mutate a Parquet file in place and cannot make an uncommitted delete visible.

What the manifest records

The logical delete file is represented by the ordinary Iceberg DeleteFile abstraction, with fields that tell a reader how to locate and scope the vector:

The Iceberg v3 specification requires at most one DV for a given data file in a snapshot. If a new delete arrives, the writer merges it with the existing DV (and any remaining position deletes for that file), writes a replacement DV, and removes the superseded file-scoped delete metadata from the committed snapshot.

Why a bitmap, and why Roaring?

A naïve bitmap for a billion-row file needs a billion bits, about 119 MiB, even if only a few rows are deleted. A sorted list of positions is compact for sparse deletes but requires a search or merge for each scan. Roaring bitmaps switch representation by density: sparse chunks can store arrays of integers, while dense chunks can use bitmaps or runs. The result is fast membership testing without paying for every possible row position.

Iceberg’s deletion-vector format accepts positive 64-bit positions but is optimized around 32-bit Roaring bitmaps. It divides a 64-bit position into a high 32-bit key and a low 32-bit sub-position. Each present high key selects a 32-bit Roaring bitmap; membership then becomes “find the chunk, test the low position.” This is a storage and CPU trade, not magic compression: random deletes can still produce larger vectors than clustered deletes.

position = 0x00000002_00000011
high_key = position >> 32       # 2
low_pos  = position & 0xffffffff # 17

chunks[high_key].contains(low_pos)  # true means the row is deleted

The current Apache Iceberg implementation adds an integrity envelope around the serialized bitmap: a big-endian length, a little-endian magic number plus portable Roaring payload, and a CRC-32 over the magic number and bitmap. The envelope is deliberately checked against manifest metadata before the vector is trusted.

Open-source implementation landmarks

Iceberg core handles format objects, writing, metadata indexing, and scan-task planning. The execution engine that reads Parquet and applies the positions may be Spark, Flink, Trino, or another integration. That separation is important: Iceberg tells the engine which delete files apply; the engine’s reader removes the affected rows from its scan output.

A small but revealing detail in DeleteFileIndex is that it keeps DVs in a map keyed by referenced data-file path. When a DV exists for a file, matching position-delete files are not also returned for that file; the writer’s one-DV rule makes this safe. Equality deletes are different: they are value predicates and can still be returned when their sequence and partition rules match.

The scope rules that prevent false deletes

For a DV to apply to a data file, all of these conditions must hold:

1. The data file’s file_path equals the DV’s referenced_data_file.
2. The data file’s data sequence number is less than or equal to the DV’s sequence number. A delete cannot reach backward from an older snapshot and erase data added later.
3. The partition spec and partition values match.

These checks are why a position alone is not a row identity. Position 17 in data-a.parquet says nothing about position 17 in data-b.parquet, and an offset can change meaning after a rewrite. The file path and snapshot metadata supply the missing context.

Tradeoffs and failure modes

Deletion vectors optimize the common “many positions in one known file” case. Equality deletes remain useful when the writer cannot identify a stable file and position without reading existing data. DVs also do not make an immutable file mutable: they defer physical cleanup until a rewrite or compaction makes the data file smaller and the delete metadata unnecessary.

Security and correctness boundaries

A DV is data-removal metadata, so treating it as ordinary low-value cache state is dangerous. A reader that loses the vector can return rows that the table snapshot says are deleted. A reader that accepts a forged vector can hide arbitrary rows. The protections are layered:

• Catalog commit: the manifest and snapshot pointer are committed atomically, so uncommitted delete metadata is not visible to ordinary snapshot readers.
• Object-store authorization: Puffin files and metadata need the same access-control boundary as data files. A range-read URL must not let an unauthorized party enumerate other blobs.
• Encryption: Iceberg’s DV metadata can carry encryption key metadata, while the Puffin container is read through the table’s file-I/O and encryption layer.
• Integrity: length, magic, CRC, and cardinality detect accidental corruption. They are not a replacement for authenticated encryption or trusted metadata authorization.
• Retention: do not physically delete a Puffin file merely because the current snapshot no longer references one blob; an older retained snapshot may still need it.

How to debug a production table

1. Start at the snapshot. Record the snapshot ID, operation, manifest-list path, and data-file path for the unexpected row.
2. Inspect delete candidates. For that data file, list every matching delete manifest entry, its content type, referenced path, partition, sequence number, offset, length, and cardinality.
3. Recompute applicability. Check path equality first, then partition equality, then data_sequence_number ≤ delete_sequence_number. This catches most “delete did not apply” reports.
4. Check the blob, not just metadata. Range-read exactly content_offset..content_offset + content_size_in_bytes, validate the Puffin footer and DV CRC, and compare set-bit cardinality with record_count.
5. Compare engines. Run the same snapshot through two readers or through an engine’s file-scan debug mode. If Iceberg plans the DV but the result still contains the row, the bug is downstream in delete application.
6. Check conversion history. For upgraded v2 tables, look for old position deletes covering the same file and verify that a conversion did not remove them before their positions were merged.

Useful metrics are delete files per data file, DV bytes per deleted row, DV cardinality divided by data-file record count, range-read bytes, scan CPU spent applying deletes, and the age of unreachable Puffin files. A rising DV density is a compaction signal; a rising number of tiny DVs is usually a write-pattern or batching problem.

Compact reproducible lab

Goal and prerequisites

This lab models the exact semantic core without requiring a JVM, Spark, or an object store: positions are collected into a set, serialized as a simple bitmap, and applied only to the referenced file when the sequence and partition rules match. It is intentionally not a wire-compatible Puffin writer; the point is to make the scope invariant observable before adding format machinery.

mkdir -p /tmp/iceberg-dv-lab
cd /tmp/iceberg-dv-lab
python3 --version

Save this as dv_lab.py:

from dataclasses import dataclass

@dataclass(frozen=True)
class DataFile:
    path: str
    partition: str
    sequence: int
    rows: list[str]

@dataclass(frozen=True)
class DeletionVector:
    referenced_path: str
    partition: str
    sequence: int
    bits: frozenset[int]

def applies(dv: DeletionVector, data: DataFile) -> bool:
    return (dv.referenced_path == data.path
            and data.partition == dv.partition
            and data.sequence <= dv.sequence)

def read_visible(data: DataFile, dv: DeletionVector) -> list[str]:
    if not applies(dv, data):
        return data.rows
    return [row for pos, row in enumerate(data.rows) if pos not in dv.bits]

data = DataFile("data-a.parquet", "day=2026-09-01", 10,
                ["alice", "bob", "carol", "dave", "erin", "fran"])
dv = DeletionVector(data.path, data.partition, 12, frozenset({1, 4}))

print("visible:", read_visible(data, dv))
print("applies-to-wrong-file:", applies(dv, DataFile(
    "data-b.parquet", data.partition, 10, data.rows)))
print("applies-to-newer-data:", applies(dv, DataFile(
    data.path, data.partition, 13, data.rows)))

Run it:

python3 dv_lab.py

Expected result:

visible: ['alice', 'carol', 'dave', 'fran']
applies-to-wrong-file: False
applies-to-newer-data: False

The two deleted positions are 1 and 4, so bob and erin disappear. The same bits do not apply to a different data-file path, and a DV at sequence 12 cannot delete data written at sequence 13.

Variation 1 — the classic bug: change referenced_path to "data-b.parquet" and rerun. The first result becomes the original six rows. A bitmap without its file identity is not a correct delete representation.

Variation 2 — merge before replacement: change the DV bits to frozenset({1, 4, 5}). The result removes three positions. This simulates a writer merging a previous DV with a new delete before publishing the replacement metadata. In a real implementation, the set is a serialized Roaring bitmap inside a Puffin blob, and the manifest’s cardinality must change from 2 to 3.

To inspect the real format after the semantic lab, use the Apache Iceberg source links below. A full Puffin fixture requires Iceberg’s Java build and dependencies; this environment’s static site does not assume those are installed, so the fallback lab keeps the important correctness checks runnable with the Python standard library.

The durable mental model

Iceberg v3 deletion vectors are a carefully scoped indirection. The data file remains immutable. The bitmap says which ordinal positions are hidden. Puffin stores the bytes efficiently. Delete manifests locate and describe the blob. Snapshot sequence and partition metadata decide whether it applies. The reader performs the final membership test.

When a result is wrong, ask in order: which snapshot, which data-file path, which DV blob, which positions, and which sequence/partition scope? That sequence follows the actual architecture and avoids the tempting but incorrect assumption that a row position is globally meaningful.

Primary sources

• Amazon Redshift now supports Apache Iceberg v3 tables, posted Aug 31, 2026
• Apache Iceberg Table Specification: deletion vectors and scan scope rules
• Apache Iceberg Puffin Specification
• Apache Iceberg source: BitmapPositionDeleteIndex, serialization, CRC, and cardinality validation
• Apache Iceberg source: BaseDVFileWriter, merging and Puffin writing
• Apache Iceberg source: DeleteFileIndex, path/partition/sequence matching
• Apache Iceberg source: DeletionVectorStruct, DV manifest fields
• Apache Iceberg tests: TestBitmapPositionDeleteIndex