--- title: No Trace. No Truth. slug: no-trace-no-truth canonical_url: https://blog.openfactoryai.com/no-trace-no-truth published_at: 2026-07-14T20:55:00.289665+00:00 author: OpenFactoryAI tags: AI agent observability, distributed tracing, OpenTelemetry, inference monitoring, Loop Engineering tldr: Logs tell you that components emitted messages. Metrics tell you that populations changed. A trace tells you which inference, evidence, tool, state transition, and verifier produced one outcome. For agents, even that is insufficient unless the trace preserves semantic decisions and links to durable artifacts. The minimum useful unit is an outcome trace: one causal history from accepted task to verified success, bounded failure, or escalation. key_takeaways: - Trace the terminal outcome, not only model calls. - Give model, tool, policy, state, artifact, and verifier steps distinct spans and semantics. - Store hashes and references for sensitive or large payloads rather than leaking them into telemetry. - Head sampling hides rare expensive failures; retain traces by outcome, cost, latency, and policy risk. - A trace becomes operationally valuable when it can reproduce, compare, and attribute a failure. --- The dashboard says the model was healthy. The user says the task failed. Both can be true. An agent can receive a fast 200 response from every model call, produce valid JSON, invoke every tool without an HTTP error, and still change the wrong repository. Component health is not outcome truth. The missing object is an outcome trace: one causal account of what the system believed, observed, proposed, authorized, changed, checked, and finally proved. ::figure{template=trace caption="One illustrative agent run contains many healthy calls and one failed outcome" items="Accept task :: 0.1s | Retrieve evidence :: 0.8s | Model plan :: 1.9s | Inspect repo :: 1.2s | Model patch :: 3.8s | Apply patch :: 0.4s | Run checks :: 18.0s | Verify outcome :: failed / 0.7s"} ## Logs, metrics, and traces answer different questions A log records an event. A metric aggregates measurements. A trace connects operations that belong to one causal path. For a software agent: | Signal | Useful question | Blind spot | |---|---|---| | Log | What did this component report? | Which task and decision caused it? | | Metric | Are cost, errors, or latency changing? | What happened in one bad run? | | Trace | Which operations formed this run? | What did a model decision mean? | | Durable history | Which verified state transitions occurred? | May be too detailed for fleet analysis | | Evaluation | Was the outcome correct and valuable? | Often lacks production causality | [Dapper](https://research.google/pubs/dapper-a-large-scale-distributed-systems-tracing-infrastructure/) describes a tracing infrastructure designed for low overhead and broad deployment across large distributed services. [X-Trace](https://www.usenix.org/conference/nsdi-07/presentation/x-trace-pervasive-network-tracing-framework) similarly argues for reconstructing behavior across layers and administrative components. Agent systems inherit that distributed-systems problem, then add probabilistic decisions, retrieved evidence, mutable context, and external action. A model span alone is equivalent to tracing a database call but ignoring the transaction. ## The trace root is the task, not the chat turn Long-running automation outlives HTTP requests and model sessions. One task can create many traces as workers stop and resume. One chat turn can start several durable tasks. Choose stable identities deliberately: ```text task_id business unit of work run_id one end-to-end attempt or resume generation trace_id one propagated causal execution graph span_id one operation transition_id one logical state mutation artifact_id immutable output or evidence object ``` The [W3C Trace Context Recommendation](https://www.w3.org/TR/trace-context/) standardizes HTTP headers for propagating trace identity across services. Use that substrate where it fits. Do not force a days-long workflow into one open in-memory span. Link resumed traces to the same task and prior transition. ::figure{template=branch caption="A durable task can contain retries, resumes, and parallel branches without losing causal identity" items="Task 42 :: one business outcome | Run A :: worker starts | Trace A :: plan and inspect | Timeout :: tool outcome unknown | Run B :: replacement worker | Trace B :: reconcile and resume | Branches :: two sandbox candidates | Terminal :: one verified artifact"} The task record answers “what eventually happened?” The trace graph answers “how did this execution get there?” They should link, not compete. ## Give semantic steps their own spans Generic names such as `llm.call` and `tool.call` are insufficient. Two model calls may serve radically different roles. One plans. One extracts typed arguments. One judges evidence. One compresses history. Their latency and tokens are comparable; their failure semantics are not. A useful agent trace has spans such as: ```text task.run state.load context.assemble retrieval.query retrieval.rerank policy.tools_eligible inference.decide action.validate schema.check authorization.check budget.check tool.execute effect.reconcile state.reduce verifier.run outcome.commit ``` Each span should carry bounded attributes: - task, tenant, workflow, prompt, model, tool, policy, and verifier versions; - input and output token counts using the provider's actual accounting when available; - cache class and hit status; - queue, time-to-first-token, decode, tool, and verification latency; - transition source and destination states; - authority decision and reason code; - artifact hashes and storage references; - retry, cancellation, and timeout classification; - terminal outcome and evidence class. The [OpenTelemetry specification](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/overview.md) provides vendor-neutral concepts for traces, metrics, and logs, and its semantic conventions define shared naming patterns. Generative-AI conventions are evolving. Pin the schema version you emit and preserve your own domain attributes for workflow state and verified outcomes. ::figure{template=layers caption="An outcome trace crosses five observability planes" items="Outcome :: verified success, bounded failure, escalation | Workflow :: state, transition, lease, budget, stop | Decision :: context, model, sampling, candidate, selection | Effect :: tool, authorization, idempotency, receipt | Infrastructure :: queue, service, host, GPU, network"} ## Do not put secrets into observability The easiest tracing implementation stores every prompt, completion, retrieved document, tool argument, and result. It is also a fast way to build a second ungoverned data lake containing source code, credentials, personal data, contracts, and model-generated secrets. Separate searchable metadata from protected payloads: ```text trace attribute: artifact.hash = sha256:... trace attribute: artifact.class = repository_patch trace attribute: data.policy = confidential trace attribute: payload.capture = reference_only artifact store: encrypted object with tenant policy and retention ``` Apply allowlists, not denylists. Redact before export. Hashes prove identity but do not make low-entropy secrets safe. A hash of “yes” or a four-digit PIN is easy to guess. Use opaque artifact IDs for sensitive small values. Record the context assembly manifest rather than necessarily the entire context: ```json { "prompt_version": "planner.v17", "state_version": 82, "evidence": [ {"artifact": "doc_91", "hash": "...", "range": "120-188"}, {"artifact": "log_33", "hash": "...", "range": "1-74"} ], "tools": ["repo.read@4", "patch.apply@3", "check.run@8"] } ``` This makes the input reconstructable by an authorized investigator without broadcasting it to every telemetry consumer. ## Cost attribution needs a tree, not one counter An agent's cost is distributed across decisions and consequences: ```text model inference context retrieval and reranking tool compute sandbox time artifact storage verification retries and abandoned branches human exception handling failure loss ``` Attribute cost to the task and the span that caused it. If an early retrieval miss triggers three wrong patches, the cost is not merely “three model calls.” The trace should expose the miss, the downstream retries, and the verifier failures. For every run calculate: ```text useful cost = cost on the terminal accepted path exploration cost = rejected but informative branches waste cost = duplicated, cancelled too late, or repeated equivalent work recovery cost = reconciliation, replay, and compensation ``` These labels require judgment. Store a reason code so teams can revise classification without rewriting raw measurements. ::figure{template=waterfall caption="Illustrative $8.40 task cost separates accepted work from avoidable waste" items="Total task cost :: 8.40 | Accepted-path inference :: -2.10 | Accepted tools :: -1.60 | Verification :: -0.90 | Useful exploration :: -1.20 | Late cancellation :: -1.10 | Duplicate replay :: -0.70 | Human exception :: -0.80"} ## Sampling by request hides the failures you need Uniform head sampling decides whether to keep a trace before the outcome is known. A 1 percent sample is attractive for cost, but a rare 0.1 percent policy failure then appears in only about one sampled trace per 100,000 runs in expectation: ```text 100,000 runs × 0.1% failure × 1% sample = 1 retained failure trace ``` This is an illustrative probability calculation, not a traffic report. It shows why agent telemetry needs tail decisions after terminal attributes are available. Retain traces when any of these is true: - outcome failed or escalated; - policy denied or a high-consequence tool was proposed; - cost or latency crossed a slice-specific threshold; - verifier disagreement occurred; - recovery, compensation, or duplicate delivery occurred; - the run represents a stratified success sample; - the workflow, model, prompt, or tool version is newly deployed. Keep a small unbiased sample too. Outcome-biased retention is excellent for debugging but cannot estimate fleet rates without sampling weights. ## A trace is not automatically an explanation Tracing records what the instrumented system exposed. It does not prove why a model chose an action. A chain-of-thought string is not a causal explanation and may be incomplete or misleading. Prefer observable decision inputs and outputs: - exact context manifest; - available tool set; - typed proposed action; - candidate alternatives if generated; - verifier scores and rule results; - policy checks; - model and sampling configuration; - resulting environment observation. Then run counterfactual replays. Hold the workflow constant and vary one factor: retrieved evidence, model, prompt version, tool availability, or verifier. If the outcome changes repeatedly, you have stronger attribution than a persuasive reflection. ## Cardinality is an architecture limit Telemetry systems index attributes so operators can filter and aggregate them. An unbounded value such as raw prompt text, user query, repository path, task ID, commit hash, or tool arguments can create a distinct time series or index entry for almost every request. Cost and query latency then grow for little analytic value. Classify attributes by their intended operation: | Attribute class | Examples | Storage treatment | |---|---|---| | Fleet dimension | workflow, model, tool, outcome code | Indexed, bounded vocabulary | | Investigation key | task ID, trace ID, artifact ID | Searchable lookup, not a metric label | | High-cardinality fact | commit hash, document ID, user ID | Trace or event field with retention policy | | Payload | prompt, completion, source, tool output | Governed artifact, normally not indexed | | Secret | token, credential, private key | Never capture | A useful test is: “Will we group a population by this value?” If yes, define a bounded semantic convention. If the value only locates one incident, keep it on the trace. If it reconstructs the decision, store it as a protected artifact. Version attribute names and enumerations. Changing `outcome=success` to `status=verified` without a migration splits dashboards and silently corrupts comparisons. Treat telemetry schema as an API consumed by alerts, evaluators, cost reports, and research jobs. ## Build a replay bundle, not just a pretty waterfall A trace UI is useful for scanning time. A failure investigation also needs a portable, immutable bundle: ```text manifest.json task, run, trace, workflow and policy versions context/ prompt template, state projection, evidence manifests decisions/ model configuration, typed proposals, selection results effects/ tool requests, receipts, idempotency identities artifacts/ patch, logs, tests, verifier evidence by content hash outcome.json terminal state, reason, cost, latency, proof ``` The bundle should be sufficient for an authorized engineer to replay pure steps and simulate effecting steps against a sandbox. It should not contain live credentials or rely on mutable “latest” resources. Replay has levels: 1. **Structural replay:** can the workflow reduce the history to the same typed state? 2. **Input replay:** can the exact context and tool eligibility be reconstructed? 3. **Model replay:** does the same model configuration reproduce or statistically resemble the decision? 4. **Environment replay:** do sandboxed tools observe equivalent state? 5. **Outcome replay:** does the verifier reach the same terminal judgment? Bit-for-bit model output is often unavailable because serving kernels, model aliases, and sampling can change. That does not excuse unversioned evidence or tools. Record what is controllable and label what is not. ## Define service objectives around terminal work An ordinary API may target request availability and latency. An agent task needs outcome objectives with explicit clocks: ```text acceptance latency: accepted_at - submitted_at active work latency: sum of intervals with a valid worker lease external wait: CI + approval + rate-limit + dependency wait time to terminal outcome: terminal_at - submitted_at verification delay: verified_at - final_effect_at ``` Suppose 1,000 tasks have a 95 percent verified-success objective and a 30-minute terminal deadline. If 970 succeed but 70 complete after the deadline, success quality is 97 percent while on-time verified success is only 90 percent: ```text verified success rate = 970 / 1,000 = 97% on-time verified success = (970 - 70) / 1,000 = 90% ``` The numbers are illustrative. The point is that a quality SLO without a deadline rewards eventually correct systems that users cannot depend on. A latency SLO without verification rewards fast wrong work. Budget error by task slice and consequence. A documentation edit and a production credential rotation should not consume the same reliability budget. Report both the denominator and the exclusion rules so paused, cancelled, denied, and infeasible tasks do not disappear from the metric. ## Worked example: one green fleet, one broken task Assume an agent changes a dependency version. Every component metric is inside its service objective: | Span | Status | What happened | |---|---|---| | `retrieval.query` | OK | Returned an archived migration guide | | `inference.decide` | OK | Proposed the old configuration key | | `schema.check` | OK | Action JSON matched schema | | `authorization.check` | OK | Repository write was permitted | | `patch.apply` | OK | Patch committed in sandbox | | `check.run` | OK | Unit suite passed | | `verifier.integration` | FAIL | Staging rejected removed key | The model latency dashboard is green. The tool error dashboard is green. The schema-validity dashboard is green. The outcome failed because evidence freshness and verification coverage were wrong. The trace supports concrete automation: 1. identify every failed task whose selected evidence version predates the target dependency; 2. replay them with current documentation; 3. add a freshness rule to context assembly; 4. add the staging configuration check before mutation; 5. compare cost per verified outcome before and after. Without the outcome trace, the likely response is to change the model. With it, the team can fix the actual control boundary. ## What to alert on Do not page on every model error. Many are retried and never affect an outcome. Alert on customer or safety consequence: - verified-success rate below the slice baseline; - p95 time to terminal outcome above deadline; - cost per verified outcome above budget; - unknown side effects older than reconciliation SLO; - high-consequence mutation without expected receipt; - verifier disagreement or missing proof; - repeated equivalent state or runaway branch growth; - compensation backlog; - trace continuity loss between authorized mutation and outcome. Use component alerts to route diagnosis after the outcome signal fires. ## Builder playbook 1. Define `task_id`, `run_id`, `transition_id`, and `artifact_id` before adding spans. 2. Propagate W3C trace context through synchronous boundaries and explicit links through queues and resumes. 3. Instrument model, context, policy, tool, state, verifier, and outcome operations separately. 4. Adopt versioned semantic conventions and test emitted schemas in CI. 5. Store large or sensitive payloads in a governed artifact store; trace references and hashes. 6. Record tokens, time, cost, cache, retry, cancellation, and result at the span that owns them. 7. Tail-sample failures, expensive runs, policy events, recovery, and new versions while keeping a weighted unbiased sample. 8. Make a trace replayable enough to reconstruct inputs and compare one controlled variation. 9. Join traces to offline evaluations and terminal business outcomes. 10. Run a monthly “green components, failed task” review to find missing semantic spans. The test of observability is not whether a dashboard looks complete. It is whether an engineer can answer, with evidence: ```text What failed? Where did the wrong state enter? Which later costs did it cause? Why did verification not stop it sooner? Which change prevents recurrence? Did that change improve verified outcomes? ``` If the trace cannot answer those questions, it recorded activity, not truth. ## References - [Sigelman et al., “Dapper, a Large-Scale Distributed Systems Tracing Infrastructure,” Google, 2010](https://research.google/pubs/dapper-a-large-scale-distributed-systems-tracing-infrastructure/). Primary design report on low-overhead distributed tracing at large scale. - [Fonseca et al., “X-Trace: A Pervasive Network Tracing Framework,” NSDI 2007](https://www.usenix.org/conference/nsdi-07/presentation/x-trace-pervasive-network-tracing-framework). A cross-layer approach to reconstructing service behavior. - [W3C Trace Context Recommendation, 2021](https://www.w3.org/TR/trace-context/). The standard propagation format for distributed trace context over HTTP. - [OpenTelemetry Specification](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/overview.md). Vendor-neutral telemetry concepts and APIs; implementations should pin the convention versions they emit. - [OpenTelemetry General Trace Semantic Conventions](https://opentelemetry.io/docs/specs/semconv/general/trace/). Common attribution rules for trace spans across implementations. - [Yang et al., “SWE-agent,” 2024 preprint](https://arxiv.org/abs/2405.15793). Primary evidence that the agent-computer interface is a meaningful part of automated software behavior and therefore needs tracing. ## FAQ ### What should an AI agent trace contain? It should connect task and run identity, context assembly, model calls, candidates, policy checks, tool effects, state transitions, artifacts, tokens, cost, latency, verification, retries, and the terminal outcome. ### Should prompts and completions be stored in observability tools? Only under an explicit data policy. Prefer allowlisted metadata plus encrypted artifact references and hashes. Raw payload capture can leak source code, credentials, personal data, and confidential evidence. ### How is an agent trace different from a model-call log? A model-call log covers one inference boundary. An outcome trace follows causal work across context, models, tools, durable state, retries, policies, verifiers, and the final result. ### How should AI agent traces be sampled? Use tail sampling for failures, high cost or latency, policy events, recovery, verifier disagreement, and new versions, plus a weighted unbiased sample of ordinary successes for fleet estimates. ### Can chain of thought explain why an agent failed? Not reliably. It is model output, not a guaranteed causal account. Preserve observable inputs, typed actions, policy decisions, receipts, and counterfactual replay results instead. ### What is the main business metric for agent observability? Cost and time per terminal verified outcome, sliced by task and consequence. Component latency and token totals are diagnostic inputs to that outcome metric. ## Test yourself ### 1. What should be the root business identity for a long-running agent workflow? - [ ] One HTTP request - [ ] One model call - [x] The durable task - [ ] One log line **Explanation:** The task persists across retries, workers, traces, and model sessions until one terminal outcome. ### 2. Why is uniform 1 percent head sampling weak for rare failures? - [ ] It stores too many failures - [x] It decides before the terminal outcome and can discard nearly all rare events - [ ] It changes model output - [ ] It disables metrics **Explanation:** Tail sampling can retain traces after failure, cost, latency, or policy attributes are known. ### 3. What should a trace store for a confidential large payload? - [ ] The whole payload in every span - [x] A governed artifact reference and integrity hash - [ ] Nothing about it - [ ] A public URL **Explanation:** References preserve linkage while access control and retention remain in the artifact store. ### 4. Which signal proves a task succeeded? - [ ] All model calls returned 200 - [x] The terminal verifier accepted evidence and committed the outcome - [ ] The average latency is low - [ ] The JSON parsed **Explanation:** Healthy components do not guarantee correct composition or a verified business result. ### 5. What gives stronger failure attribution than a model reflection? - [ ] A longer reflection - [x] Controlled replay varying one input or component - [ ] A higher temperature - [ ] Deleting traces **Explanation:** Counterfactual replay tests whether changing one factor repeatedly changes the outcome.