Wiki · Concept · Last reviewed June 25, 2026

OpenTelemetry Tail Sampling

OpenTelemetry tail sampling is a trace-retention strategy that waits until all or most spans are available before deciding whether to keep a trace. For AI agents, it is a way to preserve rare failures, policy violations, slow tool chains, and high-risk actions without storing every ordinary trace forever.

Category: Concept Published: June 25, 2026 Modified: June 25, 2026 Last reviewed: June 25, 2026 Tags: OpenTelemetry, tail sampling, observability, AI agents, audit trails

Definition

Tail sampling is an OpenTelemetry trace-sampling method where the sampling decision is made after considering all or most spans in a trace. The OpenTelemetry sampling documentation contrasts it with head sampling, which decides early and cannot inspect the whole trace. Tail sampling can retain traces based on conditions such as errors, latency, specific attributes, service volume, or other domain criteria.

In the OpenTelemetry Collector, the Tail Sampling Processor is the main component for this job. The upstream processor README says it samples traces according to defined policies, groups spans by trace_id, and requires all spans for a given trace to reach the same collector instance for effective decisions. Its documented status is beta for traces, and it is distributed in the contrib and Kubernetes collector builds.

How It Works

A tail sampler holds trace data long enough to decide what the trace means. The processor has required policies and tunable settings such as decision_wait, num_traces, expected new traces per second, decision caches, and maximum trace size. Policy types include latency, status code, string attributes, numeric attributes, boolean attributes, trace state, trace flags, probabilistic sampling, rate limiting, span count, OTTL conditions, and composite policies.

This makes tail sampling more expressive than a fixed percentage at the edge. It can keep all traces containing an error, traces whose overall duration exceeds a threshold, traces from a newly deployed service, or traces with an attribute that marks a special review class. The price is state. The sampler must buffer enough spans, route trace fragments consistently, and monitor whether traces are dropped too early.

AI Agent Telemetry

AI agent systems create long, branching traces. A single user request may pass through a model gateway, retrieval system, policy evaluator, tool server, payment service, browser automation layer, human approval step, and final response. Storing every span may be expensive and privacy-invasive. Dropping traces too early may erase the very incident that later needs review.

Tail sampling fits this tension. An agent platform can keep traces where a tool call failed, a policy guardrail blocked an action, a human approval was requested, a model route changed, latency crossed a threshold, or an incident label was attached. It can sample routine healthy traces at lower rates while retaining evidence-rich traces for AI Agent Observability and AI Audit Trails.

Governance Use

The governance question is not simply "did we sample?" It is which traces were made durable and why. Tail sampling policies can become an accountability boundary: a rule that keeps failed tool executions helps later incident review; a rule that drops ordinary traces reduces surveillance; a bad rule can hide systemic harm by retaining only spectacular errors and discarding quiet degradation.

Good policy design should separate operational debugging from formal evidence. Engineering may need representative traces for performance analysis. Security may need all traces matching abuse markers. Compliance may need preservation for high-risk workflows. Privacy teams may require that raw prompts, retrieved documents, credentials, and personal identifiers be minimized before or during export.

Limits

Tail sampling is not a neutral memory machine. OpenTelemetry's sampling documentation warns that tail sampling is harder to implement and operate than head sampling because it is stateful, resource-sensitive, and must evolve with the system. The processor README also describes scaling requirements: when collector fleets are scaled, traces must be routed so all spans for the same trace reach the same tail-sampling instance, often through a separate layer with load balancing.

Sampling also changes measurement. If retained traces overrepresent errors, slow paths, flagged users, or special agent workflows, they should not be treated as an unbiased picture of all behavior unless the sampling design supports that inference. The point of tail sampling is often usefulness, not statistical purity.

Minimum Record

An AI deployment using tail sampling should keep a versioned record of collector topology, processor version, policy names, policy types, thresholds, attribute keys, decision wait time, memory limits, cache settings, routing guarantees, dropped-trace metrics, export destinations, and retention class. It should also record what happens to unsampled data: immediate discard, aggregate metrics, temporary low-cost storage, or restricted incident buffer.

Source Discipline

Use OpenTelemetry project documentation for the sampling model, Collector component status, processor configuration, and sensitive-data responsibilities. Use vendor documents only for vendor backends or hosted sampling systems. Treat every sampling policy as production governance code, not merely a cost knob.

Spiralist Reading

Tail sampling is the ritual of choosing which machine journeys become public memory. It can protect people from total telemetry capture, and it can preserve the rare trace that explains a bad delegation. The discipline is to make the choice visible: what was kept, what was dropped, who decided, and what forms of harm the policy is unable to see.

Sources

Return to Wiki