---
title: From Prompt to Proof
slug: from-prompt-to-proof
canonical_url: https://blog.openfactoryai.com/from-prompt-to-proof
published_at: 2026-07-14T20:41:00.289665+00:00
author: OpenFactoryAI
tags: inference, agent runtime, observability, latency, verification
tldr: A production inference request does not end when the model stops generating. It ends when the system has validated the output, controlled any side effects, committed the result, and recorded enough evidence to explain what happened. Measure time-to-first-token for responsiveness, but govern automation with time, cost, and reliability per verified outcome.
key_takeaways:
  - Time-to-first-token measures responsiveness, not automation lead time.
  - Prefill and decode are different workloads with different bottlenecks and scheduling behavior.
  - Tool calls turn text generation into state-changing distributed execution.
  - Verification must have a declared scope, authority, and failure path.
  - A useful trace joins intent, inference, actions, evidence, cost, and the committed outcome.
---

A user clicks **approve refund**. The interface begins streaming text 212 milliseconds later. The model finishes its first answer at 1.012 seconds. The refund is not safely approved until 2.72 seconds.

Which number is the latency?

All three, for different owners.

- **212 ms time-to-first-token** tells the product team when the interface starts feeling responsive.
- **1.012 s response time** tells the inference team when one generation completed.
- **2.72 s time-to-verified-outcome** tells the business when the intended work became acceptable and durable.

Optimizing the first while ignoring the third is how fast demos become slow automation. The missing 1.708 seconds contains the work that makes the output valuable: tool execution, a second decision, policy checks, verification, persistence, and evidence.

This post follows one request all the way through. The numbers are an illustrative trace, not a benchmark. The architecture is the reusable part.

::figure{template=layers caption="One request has three finish lines" items="Verified outcome · 2720 ms :: side effect checked, committed, and attributable | Complete model response · 1012 ms :: tokens exist, action not yet accepted | First token · 212 ms :: interface can begin rendering | Request accepted · 0 ms :: intent and acceptance rule enter the system"}

## The request begins before the prompt

The prompt is not the raw material. **Intent is.**

For an informational chat, intent may be loose: answer the question helpfully. For automation, it needs an operational shape:

```text
Goal: approve or reject refund request R-1842
Authority: may approve up to $100 for an eligible order
Required evidence: order exists, payment settled, item eligible, no prior refund
Side effect: create refund exactly once
Acceptance: provider receipt stored and ledger reconciled
Escalation: human review on policy ambiguity or inconsistent state
```

The system will eventually convert part of this into model context. It should not convert all authority into prose and hope the model obeys it. Monetary limit, idempotency, and permission belong in executable controls around the model.

This gives the request two parallel objects:

1. **A work object**, containing the task, context, and evolving state.
2. **An acceptance object**, containing the checks and authority required before the result counts.

If only the work object exists, the system knows how to act but not how to finish.

## Stage 1: the gateway admits or rejects the request

The gateway is the front door. It authenticates the caller, checks tenant and rate limits, assigns a request identifier, applies coarse policy, and establishes the budget envelope.

In the worked trace this stage takes 12 ms. That number is not inherently good. The useful questions are:

- Did unauthorized requests die here, before paid work?
- Did the gateway attach the correct tenant, user, region, and policy version?
- Did it establish an end-to-end deadline rather than a separate timeout for every downstream call?
- Did it create or propagate a trace identifier without recording secrets?

The [W3C Trace Context](https://www.w3.org/TR/trace-context/) standard defines how trace identifiers can cross process boundaries. Correlation is necessary, but it is not proof. A trace identifier tells us which spans belong together. It does not tell us whether the caller had authority or the result was correct.

The first span should therefore carry references to policy and identity, not merely timing:

```json
{
  "trace_id": "7f...",
  "request_id": "req_1842",
  "tenant": "org_27",
  "intent_type": "refund_decision",
  "policy_version": "refund-v12",
  "deadline_ms": 5000,
  "max_direct_cost_usd": 0.08
}
```

Sensitive data should be tokenized or referenced, not copied indiscriminately into telemetry.

## Stage 2: the router chooses a policy, not a favorite model

The router takes 10 ms in the example. It selects a route based on the job class, current availability, quality requirement, latency budget, data boundary, and cost ceiling.

For a multi-step workflow, routing is not necessarily one decision. The system might use:

- a small classifier for request type;
- a capable model for ambiguous policy reasoning;
- deterministic code for eligibility arithmetic;
- a cheaper model to write a customer explanation;
- a fallback route when a provider times out.

The route must be recorded as a decision with inputs and policy version. Recording only the selected model loses the reason and makes a later cost or quality change impossible to explain.

This is also the first place where an end-to-end deadline should constrain local ambition. If 4,978 ms remain, the router can select a different plan than if 600 ms remain. A model that is optimal in isolation may be impossible inside the remaining workflow budget.

## Stage 3: context assembly creates the model's actual world

The application and the model do not share a world. The context layer assembles one.

For the refund task, it may retrieve the policy section, order facts, payment status, previous customer events, tool descriptions, output schema, and recent workflow state. In the trace this takes 20 ms, but its larger cost appears later: every token added here must pass through prefill, may occupy KV-cache memory, and may be replayed on a retry.

Context needs provenance at the item level:

| Context item | Source | Version or timestamp | Why included | Expiry |
|---|---|---|---|---|
| Refund policy | Policy repository | v12 | Defines eligibility | Re-evaluate on policy update |
| Order facts | Order service | read at 10:03:14Z | Establishes amount and item | Re-read before commit if mutable |
| Payment status | Ledger service | sequence 98102 | Proves settled payment | Require monotonic sequence |
| Prior refunds | Refund service | read at 10:03:15Z | Prevents duplication | Re-check under commit lock |

Without this table, a later reviewer sees a prompt string. With it, the reviewer can ask whether a stale entity or missing source caused the decision.

## Stage 4: queueing decides when inference actually starts

At 42 ms the request reaches the serving system. It does not begin model execution until 132 ms. The 90 ms gap is queue time.

This is the latency teams routinely misattribute to the model.

Autoregressive serving is unusual because requests have a prompt-processing phase and then many token-generation iterations. [Orca](https://www.usenix.org/conference/osdi22/presentation/yu) introduced iteration-level scheduling so a serving batch could change between iterations rather than remaining fixed until every request finished. That mechanism improves utilization and responsiveness under the paper's evaluated workloads, but it also makes scheduling policy visible to product behavior.

A long generation can be preempted. A short interactive request can be favored. A large prompt can occupy memory before producing a token. Admission control can reject work rather than let a queue violate every deadline.

::figure{template=queue caption="Queue time is product latency even when no model is running" items="R1842 :: 90 ms | R1843 :: short | R1844 :: long | R1845 :: batch | R1846 :: wait"}

Record at least:

- arrival time and admission decision;
- priority class and deadline;
- prompt and expected output length buckets;
- queue wait and preemption time;
- batch membership by iteration;
- cancellation or deadline expiry.

Raw tokens per second cannot explain a request that spent most of its life waiting.

## Stage 5: prefill reads the prompt

From 132 to 212 ms, the model processes the assembled input. This is **prefill**. It computes internal representations for the prompt and creates key-value cache state used during generation.

Prefill is generally parallel across prompt positions and can be compute intensive. Decode is sequential across generated tokens and often constrained differently by memory movement and KV-cache access. Treating them as one undifferentiated "model latency" hides optimization choices.

Two systems papers explain why:

- [FlashAttention](https://papers.nips.cc/paper_files/paper/2022/hash/67d57c32e20fd0a7a302cb81d36e40d5-Abstract-Conference.html) shows that attention performance depends on reads and writes across GPU memory levels, not only arithmetic complexity. Its IO-aware tiling reduces that movement for the evaluated attention workloads.
- [PagedAttention](https://doi.org/10.1145/3600006.3613165) applies virtual-memory-style block management to the KV cache, addressing fragmentation and duplication that can waste serving capacity.

These innovations sit below the application, but application decisions drive their workload. Resending a 40,000-token static policy on every loop step creates prefill work. Starting many branches creates KV-cache pressure. Cancelling late wastes scheduled compute.

At 212 ms, prefill completes and the first generated token becomes available. This is the **time-to-first-token**, measured from request arrival.

It is a real user-experience metric. It is not the end of the job.

## Stage 6: decode produces tokens one iteration at a time

From 212 to 1,012 ms, the target model generates the structured decision and explanation. Suppose it emits 80 tokens over 800 ms after the first token. A simplified inter-token time is about 10 ms per token.

Do not report that as 100 tokens per second without stating the measurement boundary. Client-visible streaming can include network buffering, tokenization, safety processing, and application rendering. Server throughput across a batch is also different from one request's inter-token latency.

Three metrics belong side by side:

```text
TTFT = first_token_time - request_arrival
TPOT = (last_token_time - first_token_time) / generated_tokens_after_first
response_time = last_token_time - request_arrival
```

For a non-streaming machine action, TPOT may matter less than response completion. For a conversational surface, TTFT strongly shapes perceived responsiveness. For the factory, both remain intermediate measures.

The output is constrained to a schema:

```json
{
  "decision": "approve",
  "amount_usd": 74.20,
  "policy_clause": "refund-v12#3.2",
  "evidence_refs": ["order:1842@71", "ledger:98102"],
  "confidence": 0.91
}
```

Valid JSON only proves syntax. It does not prove that clause 3.2 applies, the amount is correct, or the evidence still holds.

## Stage 7: an action turns inference into a distributed transaction

At 1.012 seconds, the system has a proposed action. It now calls the refund provider and internal ledger. The tool operation takes 750 ms in the trace.

This is the moment an agent stops being a text generator and becomes a participant in a distributed system.

[ReAct](https://arxiv.org/abs/2210.03629) formalized an influential pattern of interleaving reasoning and actions in external environments. The research demonstrates why external observations can improve task trajectories. Production adds a harder requirement: external actions may be irreversible, billable, rate-limited, slow, or only ambiguously acknowledged.

Every state-changing tool call needs:

- an authorization check outside the model;
- a typed request with semantic validation;
- an idempotency key tied to the work object;
- a deadline and bounded retry policy;
- an outcome classified as success, rejection, ambiguous, or failure;
- the provider's receipt or sequence number;
- a compensation or escalation path where possible.

If the provider times out after accepting the refund, "retry" can pay twice. The correct next action may be to query by idempotency key, not repeat the command.

::figure{template=branch caption="A tool timeout has more than one reality" items="Refund call times out :: transport result is unknown | Not accepted :: safe to retry under policy | Accepted once :: retrieve the existing receipt | Accepted but unrecorded :: reconcile before any new action | State unknowable :: stop and escalate"}

The model should never decide this purely from an error string. The runtime owns side-effect semantics.

## Stage 8: verification decides whether the result counts

At 1.762 seconds, the provider has returned a receipt. The workflow performs a second model or deterministic policy decision, then executes verification from 2.20 to 2.65 seconds.

Verification is not one green check. It is a set of scoped claims:

| Check | Claim established | Important non-claim |
|---|---|---|
| Schema validation | Required fields and types exist | Values are truthful |
| Policy evaluation | Declared rules accept these facts | Facts are current or complete |
| Idempotency lookup | One operation exists for this key | Downstream accounting is correct |
| Ledger reconciliation | Amount and provider receipt agree | Customer communication is adequate |
| Model evaluator | Explanation meets a rubric on this sample | The evaluator is unbiased or infallible |
| Human review | Authorized person accepted the exception | Every similar case will be safe |

A verifier must expose what it checked, the version of the bar, the evidence used, and the authority behind acceptance. Otherwise "verified" is merely a status label.

The acceptance object created at the beginning now becomes executable. If every mandatory claim clears, the workflow may commit. If a claim fails, the workflow must classify the failure and decide whether to repair, retry, compensate, or escalate.

## Stage 9: commit makes the outcome durable

From 2.65 to 2.72 seconds, the system writes the final decision, provider receipt, ledger sequence, trace reference, policy version, verifier results, and accepting authority.

Commit is a separate stage because an action that occurred but was not durably recorded is not a safely completed automation. The next request may repeat it. An auditor may see a payment without a rationale. A user may receive contradictory status.

The durable record should link provenance, not copy every sensitive value. [W3C PROV-O](https://www.w3.org/TR/prov-o/) supplies a useful vocabulary: entities were used or generated by activities, and agents were associated with those activities. A practical system can implement a smaller schema while keeping the relationships:

```text
intent entity
  used by -> decision activity
policy entity
  used by -> decision activity
decision entity
  generated by -> decision activity
refund receipt entity
  generated by -> tool activity
acceptance entity
  generated by -> verification activity
human or service identity
  associated with -> acceptance activity
```

At 2.72 seconds, the request reaches its third finish line: a verified outcome exists.

## The complete illustrative trace

::figure{template=trace caption="Illustrative request trace, durations in milliseconds" items="Gateway admission :: 12 ms | Routing :: 10 ms | Context assembly :: 20 ms | Queue wait :: 90 ms | Prefill :: 80 ms | Decode :: 800 ms | Tool effect :: 750 ms | Follow-up decision :: 438 ms | Verification :: 450 ms | Commit :: 70 ms"}

| Stage | Start | End | Duration | Output |
|---|---:|---:|---:|---|
| Gateway | 0 ms | 12 ms | 12 ms | Admitted work and trace identity |
| Routing | 12 ms | 22 ms | 10 ms | Route policy decision |
| Context | 22 ms | 42 ms | 20 ms | Versioned context manifest |
| Queue | 42 ms | 132 ms | 90 ms | Scheduled request |
| Prefill | 132 ms | 212 ms | 80 ms | KV state and first token |
| Decode | 212 ms | 1,012 ms | 800 ms | Proposed structured action |
| Tool | 1,012 ms | 1,762 ms | 750 ms | Provider receipt |
| Follow-up | 1,762 ms | 2,200 ms | 438 ms | Reconciled decision |
| Verification | 2,200 ms | 2,650 ms | 450 ms | Acceptance evidence |
| Commit | 2,650 ms | 2,720 ms | 70 ms | Durable verified outcome |

The first model response completes at only 37.2 percent of total lead time. Cutting decode latency in half saves 400 ms. Eliminating an unnecessary tool round trip could save more. Preventing one human escalation could save minutes or hours.

This is why the right optimization depends on the trace.

## Cost follows the same boundary

Suppose this illustrative execution costs:

- first model call: $0.018;
- follow-up model call: $0.006;
- external tool and infrastructure: $0.004;
- automated verification compute: $0.003.

Direct execution cost is $0.031.

If 12 percent of attempts require a full retry and attempts are independent, the expected direct cost of eventual success is:

```text
$0.031 / (1 - 0.12) = $0.0352
```

Now add a six percent human-escalation rate. At four review minutes and $60 per hour, expected review cost is $0.24 per attempted outcome. The economic center moves from tokens to exceptions. Before failure loss, the illustrative total is already about $0.275 per verified outcome.

Post 03 develops this unit economics in full. The lesson here is narrower: a trace that ends at the model API cannot calculate it.

## Observability is a vertical, not another stage

[OpenTelemetry](https://opentelemetry.io/docs/specs/otel/trace/api/) models a distributed path as related spans carrying timing, attributes, events, links, and status. [Dapper](https://research.google/pubs/dapper-a-large-scale-distributed-systems-tracing-infrastructure/) demonstrated why reconstructing a request path across services is useful at large scale.

Agent workflows need those primitives plus domain semantics:

- intent and acceptance identifiers;
- model route and policy version;
- context item provenance, counts, and redacted hashes;
- queue, prefill, decode, and streaming timings;
- observable model output, not private hidden reasoning;
- tool request, authorization, idempotency key, and effect receipt;
- state transition and retry cause;
- verifier version, result, and evidence references;
- cost attributed by stage;
- final outcome and later correction.

Do not put secrets, raw credentials, unnecessary personal data, or undisclosed private reasoning into the trace. Observability should reduce risk, not create a second ungoverned data lake.

::figure{template=layers caption="The observability vertical crosses every production layer" items="Outcome :: accepted, rejected, escalated, corrected | Evidence :: verifier, policy, authority, receipt | Action :: tool request, effect, idempotency | Inference :: route, context, queue, prefill, decode | Intent :: caller, goal, risk class, deadline, budget"}

## Four dashboards, four owners

One universal latency chart will satisfy nobody.

### Product dashboard

- time-to-first-token where streaming matters;
- time-to-useful-preview;
- time-to-verified-outcome;
- cancellation and abandonment;
- human escalation and correction rate.

### Inference dashboard

- queue time by priority and request shape;
- prefill latency by input tokens;
- decode latency and inter-token time by output bucket;
- batch occupancy, preemption, and KV-cache pressure;
- model/provider error and fallback rate.

### Automation dashboard

- verified success rate;
- attempts, branches, and tool calls per outcome;
- side-effect ambiguity and compensation rate;
- cost per verified outcome;
- exception age and recovery time.

### Trust dashboard

- which policy and verifier versions accepted production work;
- acceptance by risk tier and authority;
- provenance completeness;
- later reversals or incidents by original evidence;
- checks skipped, degraded, or overridden.

The identifiers join these dashboards. Their objectives remain distinct.

## Builder playbook: instrument one path end to end

1. **Choose one state-changing workflow.** Read-only chat hides the hardest boundaries.
2. **Write the acceptance object first.** Name the checks and authority required for completion.
3. **Propagate one trace and deadline.** Do not reset the clock at every service.
4. **Split inference timing.** Record queue, prefill, first token, decode, and complete response separately.
5. **Wrap tools as typed effects.** Require authorization, idempotency, receipts, and an ambiguous-result state.
6. **Record context provenance.** Store sources and versions without leaking raw sensitive content.
7. **Make verification scoped.** Each check states what it proves and what it does not.
8. **Commit the receipt.** Persist outcome, evidence, policy, and authority together.
9. **Inject failures.** Timeout after side effect, stale context, verifier outage, duplicate delivery, and deadline expiry.
10. **Optimize the largest outcome span.** Do not assume it is decode.

## References

1. [Gyeong-In Yu et al., *Orca: A Distributed Serving System for Transformer-Based Generative Models* (OSDI 2022)](https://www.usenix.org/conference/osdi22/presentation/yu). Introduces iteration-level scheduling and selective batching for autoregressive serving.
2. [Woosuk Kwon et al., *Efficient Memory Management for Large Language Model Serving with PagedAttention* (SOSP 2023)](https://doi.org/10.1145/3600006.3613165). Explains KV-cache fragmentation and virtual-memory-style block management.
3. [Tri Dao et al., *FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness* (NeurIPS 2022)](https://papers.nips.cc/paper_files/paper/2022/hash/67d57c32e20fd0a7a302cb81d36e40d5-Abstract-Conference.html). Establishes why memory movement is central to attention performance.
4. [Shunyu Yao et al., *ReAct: Synergizing Reasoning and Acting in Language Models* (ICLR 2023)](https://arxiv.org/abs/2210.03629). A foundational reasoning-and-action loop for agents interacting with environments.
5. [OpenTelemetry Trace API specification](https://opentelemetry.io/docs/specs/otel/trace/api/). Defines trace, span, event, attribute, status, and link primitives.
6. [W3C Trace Context Recommendation](https://www.w3.org/TR/trace-context/). Defines interoperable trace-context propagation across process boundaries.
7. [W3C PROV-O Recommendation](https://www.w3.org/TR/prov-o/). Provides a standard vocabulary for entities, activities, agents, use, generation, and attribution.
8. [Benjamin H. Sigelman et al., *Dapper, a Large-Scale Distributed Systems Tracing Infrastructure* (2010)](https://research.google/pubs/dapper-a-large-scale-distributed-systems-tracing-infrastructure/). Describes practical distributed path reconstruction at scale.


## FAQ

### What is time-to-first-token?

Time-to-first-token, or TTFT, is the elapsed time from a request boundary to the first generated token becoming available at the chosen measurement point. It typically includes admission, routing, queueing, and prefill. It is valuable for perceived responsiveness but does not measure when an automated task is verified or committed.

### What is the difference between prefill and decode?

Prefill processes the input context and creates model state such as the KV cache. Decode then generates output autoregressively, usually one token per request iteration. Prefill can be compute intensive and scales with input length, while decode has sequential dependencies and often different memory and scheduling constraints.

### When is an inference request complete?

A text-generation request may be complete when the final token is delivered. An automation request is complete only when required tool effects are resolved, verification has cleared a declared bar, the result is durably committed, and the evidence needed to explain it is linked to the outcome.

### Why do tool calls require idempotency keys?

A client may time out after a provider accepted a side effect. Retrying without a stable idempotency key can repeat a payment, message, or mutation. The key lets the system retrieve or reconcile the original operation rather than blindly create another one.

### What should an AI workflow trace record?

Record intent and policy references, route decisions, context provenance, queue and inference timing, observable outputs, tool authorization and receipts, state transitions, retry causes, verifier versions and results, costs, final outcome, and accepting authority. Avoid secrets, unnecessary personal data, and private hidden reasoning.

### Why is cost per model call not cost per automation outcome?

An outcome can require multiple model calls, tools, retries, verification, human escalation, persistence, and failure recovery. Cost per verified outcome includes all of them and divides by work that actually clears the acceptance rule.


## Test yourself

### 1. Which metric best measures when a state-changing automation becomes acceptable?

- [ ] Time-to-first-token
- [ ] Raw tokens per second
- [x] Time-to-verified-outcome
- [ ] Prompt length

**Explanation:** The verified-outcome boundary includes actions, checks, and durable commit, not merely generation responsiveness.

### 2. What work normally occurs during prefill?

- [x] The input context is processed and KV state is created
- [ ] Every tool side effect is committed
- [ ] The final token is streamed
- [ ] A human signs the result

**Explanation:** Prefill processes prompt tokens before autoregressive decode begins.

### 3. A refund provider times out after receiving a request. What should the runtime assume?

- [ ] The refund definitely failed
- [ ] The refund definitely succeeded
- [x] The result is ambiguous and must be reconciled using the idempotency key
- [ ] The model should guess from the error text

**Explanation:** A transport timeout does not reveal whether the side effect occurred. Reconciliation prevents duplicates.

### 4. What does a trace identifier establish by itself?

- [ ] The output is correct
- [x] Related telemetry belongs to one propagated request path
- [ ] The caller was authorized
- [ ] The verifier was independent

**Explanation:** Trace context enables correlation. Correctness and authority require additional evidence.

### 5. Why should queue time be separated from model execution time?

- [ ] Queue time never affects users
- [x] It identifies capacity and scheduling delay that model benchmarks hide
- [ ] It makes tokenization faster
- [ ] It proves business intent

**Explanation:** A request can be slow while the model itself is fast because it waited for admission or service.