Observability / reliability deep dive
OpenTelemetry tail sampling: why the collector must remember a trace before deciding
Head sampling decides at the first span and saves resources. Tail sampling waits long enough to see the trace’s outcome, then keeps errors and slow requests. That useful hindsight is a bounded in-memory state machine—and its correctness depends on every span of a trace reaching the same decision-maker.
2026-08-30
The concrete problem
Suppose an API receives one million requests per minute. Most succeed in 40 milliseconds; a small fraction fail after a database timeout or spend 12 seconds in a downstream service. If telemetry is recorded at full volume, storage and network cost become unreasonable. If sampling is decided when the root span starts, the sampler does not yet know whether the request will be interesting.
Tail sampling changes the question from “should I keep this span?” to “now that I have seen enough of this trace, should I keep the trace?” The collector temporarily holds spans, waits for a decision window, evaluates policies over the accumulated trace, and exports either the trace or nothing.
Head sampling versus tail sampling
| Property | Head sampling | Tail sampling |
|---|---|---|
| Decision point | Usually in the SDK before or as the root span is created. | In a collector after spans have accumulated for a trace. |
| Can select by final latency? | Not reliably; the final duration is unknown. | Yes, if the trace remains in the decision window long enough. |
| Resource cost | Low downstream cost because dropped spans are never sent. | Higher collector memory, CPU, and network cost before export. |
| Failure mode | A request is lost before its importance is known. | The sampler overloads, evicts, or receives only part of a trace. |
These mechanisms can be combined. A head sampler can reduce ordinary traffic before it leaves an application, while a tail sampler keeps a carefully selected percentage of the traces that survive. But if the SDK has already dropped a span, no downstream processor can reconstruct it.
The data path
The incoming unit is a batch, not necessarily a whole trace. A receiver may get a root span in one request, a child span in another request, and a late error span after that. The processor groups spans by the W3C trace ID. It must also tolerate out-of-order arrival: a child can arrive before its parent, and a remote service can finish after the local span that called it.
At the end of the decision window, the processor asks each configured policy whether the trace matches. A typical policy set is an OR: keep traces whose status is ERROR, keep traces slower than a threshold, and probabilistically keep a small share of the rest. The exact policy composition is configuration, but the lifecycle is always state accumulation → decision → release.
Important implementation landmarks
The open-source implementation lives in the OpenTelemetry Collector Contrib tail sampling processor. The names below refer to the implementation’s conceptual boundaries; minor field names can change as the collector evolves.
| Landmark | Responsibility | Operational consequence |
|---|---|---|
Config and policy factory | Parse decision_wait, trace capacity, expected arrival rate, policy list, and decision-cache settings. | A syntactically valid configuration can still be dangerously undersized for the traffic shape. |
Trace ID map / traceData | Own the accumulated spans and the metadata used while a trace waits. | Memory grows with active traces × spans per trace × attribute payload, not merely request rate. |
| Ingest path | Split a batch by trace ID, append data, and create or refresh the per-trace state. | Duplicate and late spans need deliberate handling; “one batch equals one trace” is wrong. |
| Decision ticker | Periodically find traces old enough to decide and hand them to policy evaluation. | The window is a latency budget. Longer waits improve completeness but retain more memory. |
| Policy evaluators | Run status, latency, attribute, probabilistic, rate-limiting, composite, and boolean policies. | Policy order and composition determine whether errors override a low baseline rate. |
| Decision cache | Remember sampled or rejected trace IDs for a bounded period so late spans follow the existing choice. | Consistency costs memory; an unbounded cache is another denial-of-service surface. |
The processor’s memory is deliberately temporary. A kept trace is passed onward and its accumulator can be released; a rejected trace is released without export. A collector restart loses all undecided traces. Tail sampling is therefore an observability filter, not a durable queue.
Trace affinity: the distributed systems trap
Two collector replicas behind an ordinary round-robin load balancer do not automatically form one tail sampler. Replica A may hold the root and database spans while replica B holds the error span. Each replica then sees an incomplete trace and may reject it.
The usual architecture is two tiers:
- Gateway collectors receive traffic and use a load-balancing exporter with
traceIDas the routing key. - Dedicated tail-sampling collectors receive all spans for a trace on the same backend instance, accumulate them, and export the final decision.
Hashing is not a guarantee by itself. Membership changes can remap trace IDs, a failed sampler can discard state, and retries can create duplicate spans. A consistent-hash ring reduces remapping during scaling, but it does not provide durable state or exactly-once delivery.
Configuration that expresses the intent
This minimal pipeline keeps every error trace and every trace whose total observed latency is at least 100 milliseconds. It drops the rest. The memory limiter belongs before tail sampling so the process has a chance to refuse new work before the accumulator consumes the last available memory.
receivers:
otlp:
protocols:
http:
endpoint: 0.0.0.0:4318
processors:
memory_limiter:
check_interval: 1s
limit_mib: 256
tail_sampling:
decision_wait: 5s
num_traces: 10000
expected_new_traces_per_sec: 100
policies:
- name: keep-errors
type: status_code
status_code:
status_codes: [ERROR]
- name: keep-slow
type: latency
latency:
threshold_ms: 100
batch:
timeout: 1s
exporters:
debug:
verbosity: detailed
service:
pipelines:
traces:
receivers: [otlp]
processors: [memory_limiter, tail_sampling, batch]
exporters: [debug]
num_traces is a cap on active trace state, not a target sample count. If it is too small, traces must be evicted according to the processor’s safeguards, and the result is incomplete by design. Estimate active state as roughly new traces per second × decision window, then add headroom for burstiness, fan-out, retries, and unusually large traces.
Compact reproducible lab
Goal and prerequisites
Run one collector locally, send one slow trace and one fast trace over OTLP/HTTP JSON, and observe that the debug exporter prints the slow trace after the five-second decision window. You need Docker 20.10 or newer and curl. The image is pinned so the experiment does not silently change under your feet.
mkdir -p /tmp/otel-tail-lab
cd /tmp/otel-tail-lab
cat > collector.yaml <<'YAML'
receivers:
otlp:
protocols:
http:
endpoint: 0.0.0.0:4318
processors:
tail_sampling:
decision_wait: 5s
num_traces: 100
expected_new_traces_per_sec: 10
policies:
- name: slow
type: latency
latency:
threshold_ms: 100
exporters:
debug:
verbosity: detailed
service:
pipelines:
traces:
receivers: [otlp]
processors: [tail_sampling]
exporters: [debug]
YAML
docker run --rm --name otel-tail-lab \
-p 4318:4318 \
-v "$PWD/collector.yaml:/etc/otelcol-contrib/config.yaml:ro" \
otel/opentelemetry-collector-contrib:0.159.0 \
--config=/etc/otelcol-contrib/config.yaml
Leave that terminal running. In a second terminal, send two traces. The timestamps are generated by the shell so the duration is encoded in the span itself.
now=$(date +%s%N)
slow_end=$((now + 250000000))
fast_end=$((now + 10000000))
curl -sS http://127.0.0.1:4318/v1/traces \
-H 'content-type: application/json' \
--data-binary "{\"resourceSpans\":[{\"resource\":{\"attributes\":[{\"key\":\"service.name\",\"value\":{\"stringValue\":\"tail-lab\"}}]},\"scopeSpans\":[{\"spans\":[{\"traceId\":\"11111111111111111111111111111111\",\"spanId\":\"1111111111111111\",\"name\":\"slow\",\"startTimeUnixNano\":\"$now\",\"endTimeUnixNano\":\"$slow_end\"},{\"traceId\":\"22222222222222222222222222222222\",\"spanId\":\"2222222222222222\",\"name\":\"fast\",\"startTimeUnixNano\":\"$now\",\"endTimeUnixNano\":\"$fast_end\"}]}]}]}"
sleep 7
Expected result: the collector acknowledges the HTTP request, waits about five seconds, and the debug exporter prints the trace ID made of 11s. The 22-trace is not exported. The important observation is the delay: the collector cannot classify latency until it has waited for the decision window.
Variation 1: change threshold_ms to 1; both traces should be exported. Variation 2: change decision_wait to 500ms, then send the slow span in two separate requests with the second request delayed by one second. The trace may be decided before the late span arrives, demonstrating that a tail sampler has a bounded view, not omniscience.
If the pinned image cannot be pulled, Docker will report a registry or tag error before the collector starts. Check available tags in the official image repository and substitute a published tag; if Docker is unavailable, use the same configuration with a locally built opentelemetry-collector-contrib binary. The OTLP request and expected policy behavior are unchanged.
Tradeoffs and failure modes
| Failure or choice | What happens | Mitigation |
|---|---|---|
| Decision window too short | Late error or downstream span arrives after the decision; a trace looks healthy when it was not. | Measure end-to-end span arrival delay and set the window from a percentile plus headroom. |
| Window too long | More active traces and more memory per trace; exporter latency also increases. | Use a bounded window, cap trace count, and monitor active traces and refused data. |
| Large fan-out trace | One request creates thousands of spans and dominates accumulator memory. | Limit span attributes and events, cap trace size upstream, and isolate high-cardinality workloads. |
| Collector restart | Undecided in-memory traces disappear. | Treat loss as acceptable for sampled telemetry, or place durable buffering elsewhere; do not pretend tail sampling is a WAL. |
| Load-balancer remap | Scale-out or a failed node splits new traces across sampler instances. | Use trace-ID routing, consistent hashing, graceful draining, and a loss budget for ownership changes. |
| Policy starvation | A baseline probabilistic policy may consume a quota before error/latency intent is expressed correctly. | Use explicit composite or OR policy semantics and test the policy with known traces. |
| Attribute explosion | Unbounded URLs, user IDs, or exception payloads inflate every buffered span and leak sensitive data. | Redact and cap attributes before tail sampling; do not use raw user input as an operationally unlimited dimension. |
Security and privacy implications
Tail sampling temporarily concentrates raw telemetry in the collector. That state can include authorization headers accidentally recorded as attributes, SQL fragments, email addresses, stack traces, or request bodies. Memory protection is therefore a privacy control as well as a reliability control. Apply attribute filtering and redaction before the tail sampler when possible, and restrict collector diagnostics because debug exporters print payloads.
A malicious or merely pathological client can create many unique trace IDs, enormous fan-out traces, or spans that arrive just before the deadline. This is an application-layer memory-exhaustion attack. Bound the number of traces, bound span and attribute sizes, use the collector’s memory_limiter, monitor refused telemetry, and rate-limit untrusted sources. Sampling is not authorization: never rely on a policy to decide whether sensitive data is safe to collect.
Trace IDs are identifiers, not secrets. Nevertheless, trace context can cross trust boundaries. Follow the W3C Trace Context rules, validate incoming headers, and avoid copying arbitrary baggage into high-cardinality or sensitive attributes.
How to debug a production sampler
- Prove arrival completeness. Compare spans received by the gateway with spans entering each tail-sampling instance. Check trace-ID routing, retry behavior, and collector restarts.
- Measure decision lag. Track span arrival time relative to the trace’s last span. If many important spans arrive after
decision_wait, increasing the window may be cheaper than blaming the policy. - Inspect state and refusal signals. Watch active trace count, processor queue or map pressure, process RSS, memory-limiter refusals, and exporter backpressure.
- Test policies with fixtures. Send known traces: fast success, slow success, fast error, late error, and a trace split across collectors. Verify each expected decision instead of inferring correctness from aggregate sample rate.
- Separate “not received” from “not sampled.” A backend absence can mean SDK head sampling, collector drop, policy rejection, exporter failure, or query/index lag. Correlate metrics and logs at each boundary.
The most revealing production dashboard has four lines together: spans received, traces currently held, traces sampled, and spans refused or dropped. Sample rate alone hides the failure mode; a stable 2% can mean healthy policy operation or a collector that is silently losing 98% of all data before policy evaluation.
The durable mental model
Tail sampling is a small distributed system inside the telemetry pipeline. It has identity (trace_id), ownership (which sampler instance holds that identity), a bounded state store (the accumulator), a clock (the decision window), a rule engine (policies), and failure semantics (eviction, restart, late data, and backpressure).
When tail sampling disappoints, ask four questions in order: did the span reach the sampler, did all spans for the trace reach the same sampler, was the trace still alive when the important span arrived, and did the policy say to keep it? That sequence turns “observability is missing” into a testable data-flow problem.