Observability / reliability deep dive

OpenTelemetry exemplars: how one trace ID makes a metric spike explainable

A metric can tell you that the 99th percentile got worse; a trace can tell you why one request got stuck. An exemplar is the small bridge between them: a sampled measurement kept beside an aggregated metric point, carrying a trace and span identity without turning every request into a time series.

Observability / reliability deep dive · metric aggregation, trace context, sampling reservoirs, OpenMetrics, Prometheus, cardinality, and a reproducible wire-format lab

2026-09-07

The concrete problem

Imagine an API dashboard showing http_request_duration_seconds. The histogram says that requests above 2 seconds have increased. That is excellent evidence that users are hurting, but it is not yet a diagnosis. Which route? Which customer? Which downstream call? Which deployment? The metric deliberately discarded most request-level identity while aggregating millions of observations.

The obvious fix—put trace_id in a metric label—destroys the metric. Every trace creates a new label value, so the backend sees a new time series for every request. Storage, indexing, query fan-out, and memory grow with traffic rather than with the small set of dimensions operators actually intend to aggregate.

An exemplar lets an operator click from a visible outlier in a histogram or counter to one trace. It is not a raw event stream, a replacement for tracing, or a guarantee that every bad request is retained. It is a bounded set of statistically or semantically selected pointers.

What an exemplar is

At the data-model level, a metric data point contains an aggregate: count, sum, bucket counts, or a current value. An exemplar is a sampled measurement associated with that same point. OpenTelemetry’s exemplar structure carries the measurement value and time, attributes filtered out of the aggregate, and optional trace_id and span_id.

request observations:  0.12  0.18  0.21  ...  3.84  ...  0.16
                           │                       │
                           └─ histogram aggregate  ─┘
                              count=10,000
                              buckets=[...]
                              exemplar={
                                value: 3.84,
                                trace_id: 4bf92f...,
                                span_id: 00f067...
                              }

The aggregate remains queryable as a low-cardinality time series such as http_request_duration_seconds_bucket{route="/checkout",le="5"}. The exemplar is attached to a particular bucket or sum sample as metadata. A backend can use its trace ID to construct a trace URL without indexing that ID as part of the metric identity.

Data flow: from an instrument call to a clickable trace

The crucial timing detail is that the SDK cannot preserve an unbounded list until export. It offers each measurement to a reservoir during Record, then collects the reservoir at export time. The active span context must therefore be available at the measurement call, and the span must usually be sampled if a trace-based exemplar filter is in use.

Sampling is the implementation

“Keep an exemplar” is not one algorithm. OpenTelemetry separates the decision into a filter and a reservoir.

A fixed-size reservoir of k slots keeps each of the first k measurements, then uses decreasing probability for later measurements. This is useful for a gauge, sum, or any aggregate where a small representative set is enough. A histogram reservoir has one slot per bucket, so a slow request in the 5–10 second bucket is not crowded out by fast requests in the 0–0.1 second bucket.

That bucket alignment is a deliberate semantic trade. It makes outliers visible, but it does not preserve the globally slowest request, the first request after a deployment, or every error unless the application emits an appropriate metric and filter. Sampling is a product decision disguised as a data structure.

Open-source implementation landmarks

The following path is worth reading in source because it shows where the aggregate and the exemplar share a measurement without becoming one data structure.

In the Go SDK’s histogram path, the aggregate uses atomic counters while the exemplar reservoir has its own synchronization. That separation matters: high-rate recording must not require a lock for every bucket increment merely because a few annotations are retained. The reservoir is a bounded side path attached to the aggregation, not a second unbounded log.

The wire format and its sharp edges

Classic Prometheus text format has no standard place for an exemplar after a sample. OpenMetrics extends the line with an exemplar label set and optional timestamp:

# TYPE http_request_duration_seconds histogram
http_request_duration_seconds_bucket{le="5"} 9987 # {trace_id="4bf92f3577b34da6a3ce929d0e0e4736",span_id="00f067aa0ba902b7"} 3.84 1725667200123
# EOF

The sample’s labels still define the time series. The labels after # describe the selected observation. A Prometheus-compatible server must negotiate or otherwise enable OpenMetrics handling, and its exemplar storage must be enabled and sized. A collector can receive OTLP exemplars and silently drop them at an exporter boundary if OpenMetrics is not selected.

There is another non-obvious constraint: an exemplar value should be meaningful for the sample it annotates. For a histogram bucket, the observed value must fit the bucket’s meaning. A backend can store the pointer, but it cannot repair a producer that attached a trace from one request to a metric observation from another.

Tradeoffs, failure modes, and security

Exemplars are observability data, so they inherit observability’s privacy problem. A trace ID is usually not a secret, but it is a cross-system correlation handle. Exemplar attributes can be worse: putting a user ID, authorization token, URL query string, or raw exception text into them creates a searchable side channel. Treat them with the same access control and redaction policy as traces.

How to debug a missing trace link

1. Prove the trace context exists at the record call. Log only the sampled flag and a safely truncated ID in a controlled test environment. Looking at a parent span elsewhere is not enough.
2. Inspect the SDK export. Use a console or in-memory metric reader and check the exported point’s exemplar list before involving a collector.
3. Inspect OTLP. Capture the metric export or use a collector debug exporter. Confirm the exemplar is attached to the expected histogram or monotonic sum point.
4. Inspect content negotiation. enable_open_metrics alone is not proof that the scrape client received OpenMetrics. Check the response Content-Type and body.
5. Check storage and join configuration. Confirm exemplar storage is enabled and that the trace UI knows how to turn the trace ID into a trace-backend URL.
6. Compare retention windows. A valid metric exemplar can outlive the trace it references; that is a lifecycle mismatch, not necessarily a broken metric pipeline.

Compact reproducible lab: keep the ID out of the series

Goal: parse a minimal OpenMetrics histogram line and show the distinction between series labels and exemplar labels without installing a monitoring stack.

Prerequisites: Python 3.9 or newer. The lab uses only the standard library.

cat > /tmp/otel-exemplar-lab.py <<'PY'
import re

line = ('http_request_duration_seconds_bucket{route="/checkout",le="5"} 9987 '
        '# {trace_id="4bf92f3577b34da6a3ce929d0e0e4736",'
        'span_id="00f067aa0ba902b7"} 3.84 1725667200123')

sample, exemplar = line.split(' # ', 1)
name_and_labels, value = sample.rsplit(' ', 1)
name, label_text = name_and_labels.split('{', 1)
series_labels = dict(re.findall(r'(\w+)="([^"]*)"', label_text))
exemplar_labels = dict(re.findall(r'(\w+)="([^"]*)"', exemplar))
exemplar_value = exemplar.split('} ', 1)[1].split()[0]

print('series identity:', name, tuple(sorted(series_labels.items())))
print('aggregate value:', value)
print('exemplar value:', exemplar_value)
print('exemplar link:', exemplar_labels['trace_id'])
print('trace_id is a series label:', 'trace_id' in series_labels)

assert 'trace_id' not in series_labels
assert exemplar_labels['trace_id'] == '4bf92f3577b34da6a3ce929d0e0e4736'
PY
python3 /tmp/otel-exemplar-lab.py

Expected result:

series identity: http_request_duration_seconds_bucket (('le', '5'), ('route', '/checkout'))
aggregate value: 9987
exemplar value: 3.84
exemplar link: 4bf92f3577b34da6a3ce929d0e0e4736
trace_id is a series label: False

Variation 1: add ,trace_id="4bf..." inside the first brace and rerun. The parser will report that the series identity now includes the trace ID. In a real backend, every distinct trace ID would create another time series.

Variation 2: delete the # {...} suffix. The aggregate still parses and remains useful, but the join to an individual trace is gone. This is what an exporter or protocol boundary that drops exemplars looks like from the operator’s point of view.

Optional full-stack extension: run a Prometheus server with exemplar storage enabled, expose this line using an OpenMetrics-aware HTTP handler, then query the series and inspect the exemplar UI. The exact container tag and flags change across Prometheus releases; the wire-format distinction above is the invariant to verify first.

Production pattern

For a checkout service, keep metric attributes to dimensions such as route, method, status class, and service instance. Start a span for the request, propagate its context into the latency histogram’s Record call, use a trace-based filter, and export OpenMetrics from the collector. Configure the metrics UI with a trace-backend URL template such as https://traces.example/trace/${trace_id}, then test the complete path with a deliberately slow request.

The result is a useful division of labor: metrics answer “is the system outside its normal envelope?” and exemplars offer a few coordinates for “show me one real request inside that envelope.” The design works because the high-cardinality join key is sampled and stored beside the aggregate rather than promoted into the identity of the aggregate.