Security / software supply chain deep dive

OCI image trust: why a container tag is not an identity

A production system rarely runs the string app:latest. It resolves a name to a manifest, selects a platform-specific child, downloads a graph of content-addressed blobs, verifies some digests, and finally gives a runtime permission to execute the result. Each step is a separate trust decision. Confusing a mutable tag with an immutable identity is how a repository compromise becomes a fleet-wide execution event.

Security / software supply chain deep dive · OCI manifests, registries, containerd resolution, digest verification, signatures, provenance, and a reproducible tamper lab

2026-09-06

The concrete problem

Suppose a deployment says registry.example/payments/api:prod. A human reads that as “the payment API release.” A container client reads it as a mutable lookup key. The registry can return a different manifest tomorrow, the manifest can point to different blobs, and a multi-platform image can select a different child for an ARM node than for an x86 node.

That indirection is useful: teams can promote a release by moving one tag, and one image name can support several CPU architectures. It is dangerous when the name is copied into a long-lived deployment, audit record, or incident report as though it uniquely identified executable bytes.

This matters beyond Docker. Recent reporting described active exploitation of JFrog Artifactory vulnerabilities and attackers reaching administrative control. The durable engineering lesson is not a claim about one exploit chain: an artifact repository sits directly on the path from developer intent to code execution, so repository authorization, content integrity, and release provenance must be treated as different controls.

What an OCI image actually is

An OCI image is a content graph described by JSON descriptors. A descriptor carries a media type, byte size, and digest. The digest is normally a cryptographic hash of the exact referenced bytes, commonly written as sha256:<hex>.

image name: registry.example/payments/api:prod
       │
       │  GET /v2/payments/api/manifests/prod
       ▼
index / manifest list (if multi-platform)
       │
       ├── linux/amd64 ── digest sha256:aaa...
       └── linux/arm64 ── digest sha256:bbb...
                              │
                              │  selected child manifest
                              ▼
image manifest
       ├── config descriptor ── sha256:ccc...
       ├── layer descriptor 1 ── sha256:ddd...
       └── layer descriptor 2 ── sha256:eee...
                                      │
                                      ▼
                              compressed blob bytes

The top-level object is either an image manifest or an image index. An index contains descriptors for child manifests and platform metadata. A manifest contains one config descriptor and an ordered list of filesystem-layer descriptors. The config and layers are blobs; the manifest itself is also content that can be addressed by a digest.

The data and control flow

A pull has two planes. The data plane moves manifests and blobs. The control plane decides which reference may resolve to which content, whether a caller may push or delete it, whether a signature is required, and whether a workload may execute it.

The dangerous race is a time-of-check/time-of-use gap. An admission component can resolve api:prod and approve digest A, while the node later pulls the tag again and receives digest B. The fix is to pass the immutable digest through the entire path, or to make the runtime itself enforce the same policy immediately before execution. “We checked the tag earlier” is not an identity guarantee.

Registry protocol details that matter

The OCI Distribution Specification standardizes the HTTP shape used by registries. A client starts with GET /v2/, obtains a bearer challenge when needed, then requests a manifest by tag or digest. It retrieves blobs through GET /v2/<name>/blobs/<digest>. Uploads are resumable: a client starts an upload, sends chunks, and commits with a digest.

Several subtle points follow:

• Authorization is reference-sensitive. A principal allowed to pull an image need not be allowed to push, delete, retag, or list repositories. Registry permissions should distinguish those verbs and repositories.
• Digest verification is byte-sensitive. The client hashes the received representation. A proxy that changes bytes, content encoding, or media type can break verification or, if verification is skipped, alter what is executed.
• Tags are mutable by design. A registry may support immutable-tag policy, but OCI itself does not turn a tag into a permanent pointer.
• Indexes add selection logic. A digest for an index identifies the index, not one architecture’s child. Policy may need to verify the selected child and its platform, not merely the top-level index.
• Garbage collection is graph traversal. Deleting an untagged manifest is safe only when no retained reference reaches its config or layer blobs. A repository that makes deletion decisions from tags alone can remove content still needed by a digest-pinned deployment.

Open-source implementation landmarks

The names change across releases, but the implementation path is stable enough to trace in open source.

In containerd, the resolver turns a reference into a descriptor, a fetcher retrieves content, and handlers walk descriptor graphs into a content store. The content store is keyed by digests, which is a strong integrity boundary only if every write and read validates the expected digest. The snapshotter then unpacks layers into a filesystem view. That last step is not “just decompression”: whiteouts, hard links, path collisions, filesystem limits, and snapshotter bugs all sit between verified bytes and the process that runs.

Digest, signature, and provenance are three different claims

Consider an approved digest D. Digest verification establishes that the bytes fetched now hash to D. It does not establish that the bytes are good, built from the intended source, or authorized by the release team. A signature can bind an identity to D, but only a verifier’s policy says which identities and issuers are trusted. Provenance can say that a builder produced D from commit X with workflow Y, but provenance itself is just another statement that needs an authenticated issuer and policy.

A practical policy might require all of the following: the workload is pinned to a digest; the digest is signed by a release identity from the expected issuer; the attestation names the expected repository and protected branch; the builder is in an allowlist; and the image is scanned or tested according to a documented exception process. Notice that “signed” is not the same as “safe,” and “scanned” is not the same as “authorized.”

Failure modes and security implications

Repository administration is therefore a high-blast-radius capability. The registry may not execute code itself, but it controls content that many nodes will execute later. Protecting only the production cluster misses the artifact path; protecting only the registry misses the node’s resolution and policy path.

How to debug “the image changed”

1. Capture all identities. Record the original reference, resolved index digest, selected child digest, config digest, layer digests, registry host, and platform. “Image version” is not enough.
2. Resolve twice. Compare the tag response before admission and at the node. If they differ, you have a mutable-reference race or an inconsistent mirror.
3. Inspect the manifest graph. Fetch the manifest by digest and recursively check each descriptor’s media type, size, and hash. Do not compare only the top-level tag string.
4. Check signature policy against the digest. A signature for a nearby tag, a different index, or a different child is not a signature for the executed bytes.
5. Join repository and runtime logs. Correlate push, tag mutation, delete, manifest GET, blob GET, admission, and container-create events. A repository audit trail without node-side resolution evidence leaves a gap.
6. Check the cache. Compare the content-store file or object hash with its expected digest. If adding a verification step changes the result, investigate the proxy, mirror, and store rather than treating the new result as noise.

Compact reproducible lab: a digest catches tampering, not intent

Goal: model the smallest useful part of an OCI trust boundary: a descriptor names bytes by digest, and verification detects mutation. Then observe that a valid digest says nothing about whether a human authorized those bytes.

Prerequisites: Python 3.9 or newer. No packages, registry, container runtime, or network access are required.

cat > /tmp/oci-trust-lab.py <<'PY'
import hashlib
import json
from pathlib import Path

blob = Path("/tmp/oci-demo-layer.tar")
blob.write_bytes(b"trusted-looking bytes\n")

def descriptor(path):
    data = path.read_bytes()
    digest = "sha256:" + hashlib.sha256(data).hexdigest()
    return {"mediaType": "application/vnd.oci.image.layer.v1.tar", "size": len(data), "digest": digest}

def verify(path, expected):
    actual = descriptor(path)
    if actual["size"] != expected["size"] or actual["digest"] != expected["digest"]:
        raise ValueError(f"descriptor mismatch: expected {expected['digest']}, got {actual['digest']}")

expected = descriptor(blob)
Path("/tmp/oci-descriptor.json").write_text(json.dumps(expected, indent=2) + "\n")
verify(blob, expected)
print("original: verified", expected["digest"])

blob.write_bytes(b"tampered bytes\n")
try:
    verify(blob, expected)
except ValueError as error:
    print("after mutation:", error)

print("integrity is not authorization: a newly computed digest is still a valid identity")
PY

python3 /tmp/oci-trust-lab.py

Expected result: the first verification succeeds. After the file changes, verification reports a descriptor mismatch. The final line is the important one: an attacker can compute a perfectly valid new digest. Hashes detect substitution; signatures and policy decide whether the new identity is acceptable.

Revealing variations:

• Change the initial bytes before expected = descriptor(blob). The lab still succeeds, demonstrating that integrity does not establish goodness or release authorization.
• Replace verify(blob, expected) with descriptor(blob) and print the result. The mutation becomes invisible to the caller, recreating the dangerous “trust whatever the repository returned” behavior.

Environment note: this portable lab intentionally stops at descriptor integrity. A real registry lab should use a disposable local registry and test authentication, tag mutation, multi-platform indexes, signatures, and garbage collection separately. Never experiment with production registry credentials or release tags.

The production rule

Promote by moving a tag, deploy by immutable digest, verify the selected platform child, and make the runtime consume the exact digest that policy approved. Bind signatures and provenance to that digest, not to a mutable name. Keep repository write authority narrower than pull authority, and make the audit trail answer one question unambiguously: which bytes did this node execute?