Networking / post-quantum security deep dive

Post-quantum DNSSEC: how a 2,420-byte signature becomes a transport problem

Cloudflare has enabled ML-DSA-44 validation in 1.1.1.1. The cryptographic headline is important, but the engineering lesson is more concrete: one post-quantum signature is larger than the UDP budget that DNS operators deliberately use to avoid fragmentation. The migration therefore changes packet sizing, TCP fallback, key-rollover behavior, and the meaning of “accept any valid signature.”

Networking / post-quantum security deep dive · DNSSEC chain of trust, EDNS(0), truncation and TCP retry, ML-DSA-44 wire encoding, downgrade policy, resolver implementation landmarks, and a live dig lab

2026-09-17

The concrete problem

DNS normally answers a deceptively small question: “What address belongs to this name?” A recursive resolver asks an authoritative server, receives an RRset such as an A record, caches it, and returns it to the client. DNSSEC adds authenticity by attaching an RRSIG record and publishing the keys and delegation evidence needed to verify it.

That design was already size-sensitive. DNS began with a 512-byte UDP limit. EDNS(0) lets a requestor advertise a larger UDP payload, but Internet operators commonly cap it around 1,232 bytes so a response fits beneath IPv6's 1,280-byte minimum link MTU after IP and UDP headers. RFC 9715 later recommends 1,400 bytes as a maximum DNS-over-UDP payload when the path permits it.

ML-DSA-44, the smallest parameter set selected for DNSSEC, has a 1,312-byte public key and a 2,420-byte signature. The signature alone exceeds the conservative UDP payload budget, before the DNS header, question, owner name, RRset, and EDNS OPT record exist. A cryptography change has become a transport and deployment change.

This is a current deployment issue, not a prediction exercise. On September 10, 2026, Cloudflare described enabling ML-DSA-44 validation on 1.1.1.1 and publishing a test zone. The useful question is what the resolver, authoritative server, and delegation chain must do when old and post-quantum algorithms coexist for years.

What DNSSEC is authenticating

DNSSEC authenticates DNS data; it does not encrypt the query. A validator starts from a trust anchor for the root zone and follows a chain:

The important records have different jobs:

For a normal answer, the resolver canonicalizes the RRset, finds an applicable RRSIG, looks up the signer DNSKEY, validates that key through the DS chain, and verifies the signature's time window. An AD bit in the recursive response means the resolver considers the returned data authenticated. In the lab below, the answer from 1.1.1.1 carries ad.

Why ML-DSA changes the packet

ML-DSA is a lattice-based signature scheme standardized by NIST in FIPS 204. DNSSEC does not invent a new cryptographic encoding for it. The DNSSEC Internet-Draft maps the FIPS 204 byte strings directly into the existing DNSKEY and RRSIG fields and assigns DNSSEC algorithm number 18 to ML-DSA-44.

The numbers are not merely bandwidth costs. A DNS response is assembled under a size limit selected hop by hop. EDNS(0) is not an end-to-end capability negotiation: a stub-to-recursive hop and a recursive-to-authoritative hop can advertise different payload sizes. A response that fits from the recursive resolver to a stub may still have required TCP on the resolver-to-authoritative leg.

DNSKEY answers are especially awkward. To keep older validators working, a zone may publish both an existing ECDSA or RSA key and an ML-DSA-44 key, with signatures from both. During a key rollover there may be old and new keys simultaneously. The DNSKEY RRset therefore grows exactly where the validator needs the most metadata.

Data flow: UDP first, TCP when TC says so

The normal large-response path is a deliberate two-transport state machine:

1. The requestor sends a UDP query with EDNS(0), a buffer-size advertisement, and the DNSSEC OK (DO) bit.
2. The responder composes the answer. If it cannot fit safely, it sets the DNS header's TC bit rather than sending a fragmented response.
3. The resolver retries the same question over TCP. DNS over TCP prefixes each DNS message with a two-octet length, so the receiver knows where one message ends even when the stream splits it across reads.
4. The resolver validates the complete response and returns the result to its own client. The client-facing hop may independently use UDP, DoT, or DoH.

TCP fallback is not a failure. It is the protocol's escape hatch for responses that cannot fit the datagram contract. It does have costs: a connection handshake, more state in authoritative servers, more work in load balancers and firewalls, and a possible availability cliff when a zone suddenly causes a much larger fraction of queries to use TCP.

Coexistence creates a downgrade decision

DNSSEC's compatibility rule is intentionally permissive: a validator can accept any single valid path. That matters because a resolver that understands only ECDSA must continue to validate a zone while an ML-DSA-aware resolver is being deployed. Publishing both algorithms preserves reachability.

But if ECDSA becomes forgeable in the future, “one valid path is enough” lets an attacker present a forged ECDSA-only path while suppressing or replacing the ML-DSA path. Supporting ML-DSA is not the same as requiring it.

Cloudflare's described policy uses the authenticated parent DS RRset as the signal. If the parent endorses a supported post-quantum algorithm, 1.1.1.1 requires a valid ML-DSA-44 path instead of falling back to a conventional signature. Conventional signatures can remain for old resolvers, but they no longer satisfy this stricter resolver policy.

This policy must be applied across the entire chain. A post-quantum child below a conventional parent is still dependent on a conventional link. Rotating the child's key more often does not repair that gap: an attacker who can forge a vulnerable parent delegation can manufacture a path to the child.

Open-source implementation landmarks

A production resolver is easiest to debug by following the data through modules rather than staring at the final AD bit. Names differ by project, but these are the useful landmarks in widely used open-source implementations:

At the code level, the recurring structs are a parsed DNS message, an RRset, an RRSIG metadata record, a DNSKEY, a DS digest, and a validator context holding trust state and pending dependencies. The critical functions tend to fall into five groups: parse wire records; canonicalize an RRset; dispatch by algorithm number; verify a signature and chain the key; and choose UDP truncation or TCP retry. A resolver can have a correct ML-DSA primitive and still be insecure if the algorithm-number dispatch or “must use ML-DSA” policy is wrong.

For algorithm 18, the DNSSEC draft specifies pure ML-DSA-44 with an empty context string. The public key is the 1,312-octet FIPS 204 encoding; the signature is the 2,420-octet encoding. There is no room for an implementation to silently substitute HashML-DSA, trim bytes, or reinterpret the base64 presentation.

Failure modes and tradeoffs

The tradeoff is not “fast UDP versus slow TCP” in isolation. Keeping the answer in UDP saves a round trip and server state, but fragmentation creates reliability and security risk. Using TCP for large answers spends connection and CPU resources, but gives a framed, complete message. Dual signing preserves compatibility, but expands responses and needs a deliberate downgrade policy. A migration that optimizes only cryptographic security can accidentally create an availability failure.

Compact reproducible lab: see algorithm 18 force TCP

Goal: query Cloudflare's public ML-DSA-44 test name twice: once with normal dig behavior, which retries after truncation, and once with retries disabled using +ignore, which exposes the truncated UDP packet.

Prerequisites: a POSIX shell and the BIND utilities package providing dig. No account or API key is required. Run:

dig @1.1.1.1 valid.mldsa44.dnstest.dev A +dnssec +stats +time=3 +tries=1

dig @1.1.1.1 valid.mldsa44.dnstest.dev A +dnssec +ignore +stats +time=3 +tries=1

dig @1.1.1.1 valid.mldsa44.dnstest.dev A +dnssec +tcp +stats +time=3 +tries=1

Observe: the first command should print a truncation warning, then a successful answer. In a run on September 17, 2026, it returned status: NOERROR, flags including ad, an RRSIG A 18 record, and MSG SIZE rcvd: 2563 after TCP retry. The second command should show flags: qr tc rd ra, ANSWER: 0, an EDNS UDP size of 1232, and a small response because the full answer was intentionally not retried. The third command should return the complete signed answer without the UDP truncation warning.

; normal dig behavior
;; Truncated, retrying in TCP mode.
;; ->HEADER<- opcode: QUERY, status: NOERROR
;; flags: qr rd ra ad; ...
;; RRSIG A 18 ...
;; MSG SIZE  rcvd: 2563

; +ignore: UDP packet exposed
;; flags: qr tc rd ra; ...
;; ANSWER: 0
;; EDNS PSEUDOSECTION: ... udp: 1232
;; MSG SIZE  rcvd: 54

Variation 1: remove +dnssec. The resolver no longer requests DNSSEC records from this hop, so the response should not carry the ML-DSA RRSIG. This does not disable validation inside every recursive resolver; it changes the client request's DO bit and what data is returned.

Variation 2: query the key material over TCP:

dig @1.1.1.1 dnstest.dev DNSKEY +dnssec +tcp +stats +time=3 +tries=1

Look for algorithm 18 in the DNSKEY/RRSIG records and compare the total message size. Exact answers and sizes can change as the test zone rotates records, so treat the flags and transport behavior as the stable observation, not a byte-for-byte fixture.

Lab boundary: dig demonstrates transport and resolver-result behavior; it does not independently prove the mathematics of ML-DSA or reproduce the entire root-to-zone validation chain. For that, use a resolver with DNSSEC diagnostics, capture the upstream leg, and verify the same zone with a validator that explicitly reports algorithm 18.

Debugging a production incident

1. Split the path into hops: client → recursive resolver and recursive resolver → authoritative server. Record transport, EDNS payload, DO/AD/TC flags, response size, and resolver identity for each.
2. Use dig +ignore to prove whether UDP is truncated, then dig +tcp to test TCP independently. If TCP works from a shell but not from the resolver, inspect resolver egress ACLs, NAT state, and authoritative TCP capacity.
3. Ask for DNSSEC reason codes rather than only SERVFAIL. Distinguish timeout, malformed packet, unsupported algorithm, expired signature, missing DS, bad DS digest, invalid RRSIG, and policy-required-ML-DSA failure.
4. Compare the exact DNSKEY RRset before and during rollover. Count keys, RRSIGs, and bytes; test both IPv4 and IPv6 paths because their MTU and fragmentation behavior can differ.
5. Check the algorithm chain. A child can advertise ML-DSA while a parent delegation remains conventional. Confirm which DS records are authenticated at every zone cut before claiming end-to-end post-quantum protection.
6. Monitor ratios, not just averages: TCP fallback by authoritative, SERVFAIL by reason, validation CPU by algorithm, DNSKEY response size, and cache-hit rate. A small percentage change at Internet scale can saturate a listener.

Production design rules

Authoritative operators should treat ML-DSA rollout as a protocol-capacity change. Test the largest possible DNSKEY and signed negative answers, publish both algorithms only with a rollover plan, keep TCP/53 healthy, and exercise real recursive resolvers from multiple networks. A lab with one clean IPv6 path is not an Internet compatibility test.

Recursive operators should keep algorithm support, validation policy, transport fallback, and observability separately configurable. The crypto provider answers “can I verify this byte string?” The validator answers “is this key authorized for this signer?” The policy layer answers “is a conventional path sufficient when the parent has authenticated algorithm 18?” Mixing those decisions makes downgrade bugs difficult to see.

Do not describe a zone as post-quantum secure because one leaf RRSIG uses ML-DSA-44. The property depends on the complete authenticated chain and on the resolver enforcing the stronger path. Until the root, registries, registrars, authoritative servers, and validating resolvers support the migration, the useful near-term goal is measured interoperability and a clean failure mode.

The durable mental model is simple: cryptographic agility is a distributed-systems problem. DNSSEC gives the wire protocol a place for a new algorithm, FIPS 204 gives it a primitive, and IANA gives it a number. Production safety still depends on packet budgets, transport fallback, key-rollover coordination, and a policy that refuses to let compatibility silently become downgrade.