---
title: The Token Bill Lies
slug: cost-per-verified-outcome
canonical_url: https://blog.openfactoryai.com/cost-per-verified-outcome
published_at: 2026-07-14T20:42:00.289665+00:00
author: OpenFactoryAI
tags: inference economics, automation roi, cost per outcome, routing, agent reliability
tldr: Cost per token cannot tell you whether automation creates value. Use cost per verified outcome: every model call, tool, runtime, retry, human escalation, recovery, and expected failure loss divided by outcomes that clear a declared acceptance bar. Then compare that complete unit against the same finished good produced by the current process.
key_takeaways:
  - The denominator is more dangerous than the numerator; define the same completed outcome for humans and automation.
  - Retry cost grows from failure structure, not just failure rate.
  - Human review can dominate token spend even when escalation is rare.
  - False acceptance belongs in expected loss, not hidden inside a success rate.
  - Routing and caching create value only when verified outcome quality holds.
---

An inference invoice can fall by 90 percent while an automation becomes more expensive.

The paradox disappears when the unit changes.

A model provider sells input and output tokens. A business buys resolved cases, reconciled invoices, qualified leads, merged changes, or completed investigations. Between the token and the outcome sit context assembly, routing, tool calls, retries, verification, human escalation, recovery, and the cost of a wrong result that slipped through.

The correct governing metric is:

```text
cost per verified outcome
= total execution + review + recovery + expected failure loss
  ------------------------------------------------------------
  outcomes that cleared the declared acceptance rule
```

This is not a request to ignore token cost. It is a request to put token cost in the correct row.

::figure{template=layers caption="The invoice stops where the economics begins" items="Verified outcome :: the same finished good used for comparison | Accepted-error loss :: what cleared the bar but was still wrong | Human escalation :: review, correction, exception handling | Reliability tax :: retries, branches, recovery, compensation | Direct execution :: models, tools, runtime, storage, verification | Token price :: one input inside one or more model calls"}

## The denominator is the first engineering decision

Suppose a support team handles "cases." A human process counts a case complete when the customer's problem is resolved and recorded. The AI dashboard counts a case complete when a draft response is generated.

The AI will look spectacular. It is measuring a different product.

Use a completion contract that both paths can satisfy:

```text
Outcome: eligible refund resolved

Complete when:
1. eligibility facts were checked against current policy;
2. refund executed exactly once;
3. provider receipt and internal ledger agree;
4. customer received the correct status;
5. no unresolved exception remains.
```

Now the comparison can be economic. Without the contract, every cost model is a ratio with a manufactured denominator.

The word **verified** needs equal care. It means that declared checks passed. It does not mean metaphysically correct. If a policy checker omitted a jurisdiction rule and accepted an invalid refund, that outcome was verified under an insufficient bar. Later-discovered false acceptance remains an expected loss and a signal to revise the bar.

## Build the numerator from the trace

Post 02 followed a request through gateway, routing, context, queueing, prefill, decode, tools, verification, and commit. Each span can produce cost.

| Cost class | Examples | Common mistake |
|---|---|---|
| Model | input, cached input, output, embeddings, evaluator calls | Counting only the first call |
| Tool | search, browser, database, external API, sandbox, code execution | Treating free API calls as zero operational cost |
| Runtime | gateway, queue, orchestration, storage, networking, observability | Ignoring small costs multiplied by many steps |
| Reliability | retries, fallback calls, duplicate work, compensation | Reporting average successful-path cost |
| Verification | tests, policy engines, model judges, security scans | Calling verification overhead instead of outcome production |
| Human | review, escalation, correction, approval, on-call recovery | Assuming a five-percent escalation rate is economically small |
| Loss | wrong action, delay, churn, incident, regulatory exposure | Excluding accepted failures because they are rare |
| Fixed | integration, evaluation set, process redesign, maintenance | Comparing variable token cost with fully loaded labor cost |

The ledger should be attributable per outcome. A monthly provider bill divided by tickets is better than nothing, but it cannot show whether long contexts, a fallback, or a verifier consumed the marginal dollar.

## A general equation

For a workflow with attempts indexed by `i`, let:

```text
C_i = model_i + tool_i + runtime_i + verification_i
H   = expected human review and escalation cost
R   = expected recovery and compensation cost
L   = expected loss from accepted but incorrect outcomes
V   = number of verified correct outcomes

cost per verified outcome = (sum(C_i) + H + R + L) / V
```

For planning one homogeneous task, a simplified expected model is useful:

```text
E[C_outcome]
= E[C_attempts_until_verified]
  + P(human) * C_human
  + P(recovery) * C_recovery
  + P(false_accept) * loss_if_wrong
```

If attempts are independent, identically distributed, safe to retry, and succeed with probability `p`, then expected attempts until success are `1/p`. The direct portion becomes:

```text
E[C_attempts_until_verified] = C_attempt / p
```

That equation is convenient and frequently wrong.

## Retry math fails when failures repeat

An independent transient timeout may justify the geometric model. A missing database permission, stale policy, malformed schema, or impossible task will fail again for the same reason. Replaying it increases cost without increasing success probability.

Separate at least four failure classes:

1. **Transient:** rate limit, overloaded provider, temporary network fault.
2. **Route-specific:** one model, region, or tool cannot complete the task, while a fallback may.
3. **Input-specific:** the task lacks required evidence or violates a precondition.
4. **Systematic:** prompt, policy, schema, evaluator, or architecture is wrong for this class.

Only the first class naturally supports same-route retry. The second supports a changed route. The third should request information or escalate. The fourth should stop the line and repair the system.

::figure{template=branch caption="A failed attempt is a classification problem" items="Attempt fails :: do not replay by reflex | Transient :: bounded backoff on same route | Route-specific :: change model, region, or tool | Missing precondition :: acquire evidence or escalate | Systematic :: stop class, repair policy or runtime"}

Let `q` be the fraction of failures that are genuinely recoverable and `r` the success probability of the retry conditional on that class. A more honest budget is:

```text
expected retry spend
= P(failure) * q * C_retry
  + P(failure) * (1 - q) * C_classify_and_stop
```

The stop path creates value. It prevents a cheap failure from becoming an expensive loop.

## Worked example: manual, naive automation, engineered automation

Consider the same bounded operational outcome under three production designs. All numbers are illustrative and are included so they can be replaced.

### Manual baseline

- Average labor time: 12 minutes.
- Loaded labor rate: $48 per hour.
- Direct labor: `12/60 * $48 = $9.60`.
- Expected rework and error cost: $1.00.
- **Expected cost per completed outcome: $10.60.**

The baseline must include its own errors, waiting, rework, and supervision. Automation should not be compared with an imaginary perfect human or an artificially incomplete one.

### Naive automation

One attempt contains:

- model calls: $0.090;
- external tools: $0.020;
- runtime and telemetry: $0.010;
- automated verification: $0.015;
- **direct attempt cost: $0.135.**

Seventy-two percent of attempts clear verification. Under the simplifying independent-retry assumption:

```text
$0.135 / 0.72 = $0.1875 direct cost per verified success
```

Twenty percent of tasks escalate for six minutes at $48 per hour:

```text
0.20 * 6/60 * $48 = $0.96 expected human cost
```

Two percent are accepted but later produce an average $40 correction or loss:

```text
0.02 * $40 = $0.80 expected failure loss
```

Total:

```text
$0.1875 + $0.96 + $0.80 = $1.9475
```

The model line is under nineteen cents per verified success. The outcome costs about **$1.95**. Tokens are less than ten percent of the unit.

### Engineered automation

Now change the system, not merely the model:

- route mechanical subtasks to cheaper capability;
- constrain outputs before tool invocation;
- retrieve only current policy and facts;
- stop unrecoverable retries;
- use deterministic verification where possible;
- design the human queue around true exceptions;
- reconcile side effects with idempotency keys.

Assumptions:

- direct attempt cost: $0.070;
- verified success per attempt: 84 percent;
- human escalation: 8 percent for four minutes;
- accepted-error rate: 0.8 percent at $40 average loss.

Calculation:

```text
direct       = $0.070 / 0.84              = $0.0833
human        = 0.08 * 4/60 * $48          = $0.2560
failure loss = 0.008 * $40                = $0.3200
total                                         $0.6593
```

The engineered unit is about **$0.66**.

::figure{template=bar caption="Illustrative cost per completed outcome, USD" items="Manual baseline :: 10.60 | Naive automation :: 1.95 | Engineered automation :: 0.66"}

The 94 percent difference from manual to engineered automation is not a forecast. It is the result of these assumptions. Change the wage, escalation, success, or loss terms and the answer changes. That sensitivity is the point.

## Human review is often the largest variable cost

A five percent escalation rate sounds small. If each escalation consumes ten minutes of a specialist paid $90 per hour, it adds:

```text
0.05 * 10/60 * $90 = $0.75 per outcome
```

That can exceed the entire inference bill.

This changes how a team should optimize. Cutting token cost from ten cents to five cents saves five cents. Cutting unnecessary escalation from five to three percent saves thirty cents under the assumptions above.

Do not respond by removing review blindly. Review is a control. Improve the system around it:

- route only uncertain or high-consequence cases;
- give reviewers the exact evidence and failed check;
- distinguish approval from correction and data acquisition;
- measure review time by failure class;
- feed stable corrections back into deterministic policy or evaluation sets;
- never make the queue look better by accepting silently.

The goal is not zero humans. It is zero avoidable human minutes and explicit authority for the minutes that remain.

## Accepted errors need an economic tail

Average loss can hide the most important shape. A wrong product description may cost a correction. A duplicated refund costs money and trust. A bad access-control change can produce a rare incident orders of magnitude larger.

Model expected loss by severity bucket:

```text
E[loss] = sum(P(false_accept_j) * impact_j)
```

Example:

| Accepted failure | Probability | Impact | Expected cost |
|---|---:|---:|---:|
| Minor correction | 1.5% | $8 | $0.12 |
| Customer-impacting action | 0.2% | $150 | $0.30 |
| Severe policy incident | 0.01% | $10,000 | $1.00 |

The rare incident dominates at $1.00 expected cost. Averages are not permission to automate it. Risk appetite, capital constraints, irreversibility, and legal duties can require a hard gate even when expected value is positive.

::figure{template=waterfall caption="Illustrative expected cost added by accepted failures" items="Minor corrections :: +0.12 | Customer impact :: +0.30 | Severe tail :: +1.00"}

Verification investment should follow this tail. Do not spend the same check budget on a reversible draft and a production permission change.

## Model price deflation does not guarantee outcome deflation

The [2026 Stanford AI Index](https://hai.stanford.edu/ai-index/2026-ai-index-report/technical-performance) reports rapidly improving agent benchmark performance while noting that agents still fail roughly one in three attempts on OSWorld and that top-model competition is shifting toward cost, reliability, and domain performance.

Cheaper capability creates at least four opposing effects:

1. The same call costs less.
2. Builders add more calls, branches, and verification.
3. Work expands to harder tasks with lower base success.
4. Higher volume makes rare exceptions operationally visible.

Outcome cost falls only if engineering gains outrun workflow expansion and exception cost.

This is why the correct graph has two axes. Track direct inference cost and total outcome cost. If the first falls while the second rises, the system is spending its savings elsewhere. That may be rational if quality or scope improved, but it must be visible.

## Routing changes the numerator, subject to the denominator

[FrugalGPT](https://arxiv.org/abs/2305.05176) studies cascades that choose among language models to reduce cost while preserving or improving evaluated performance. [RouteLLM](https://proceedings.iclr.cc/paper_files/paper/2025/hash/5503a7c69d48a2f86fc00b3dc09de686-Abstract-Conference.html) learns a router from preference data and reports more than twofold cost reduction without quality loss on its public benchmark evaluations.

Those papers support a powerful production pattern. They do not license a universal savings claim.

A route policy changes:

- model price and latency;
- probability of verified success;
- output length and tool behavior;
- fallback frequency;
- evaluator calibration;
- privacy and data-boundary compliance;
- correlation between producer and verifier errors.

Evaluate the route on the complete unit:

```text
route value
= cost per verified outcome under fixed acceptance bar
  + latency and capacity effects
  + risk constraints
```

A cheaper route that increases escalation or false acceptance can have negative value. A more expensive model used only for ambiguous cases can lower the total unit.

## Productivity evidence is heterogeneous because tasks are heterogeneous

The empirical literature is a warning against importing one percentage into a spreadsheet.

A [2026 Management Science paper](https://doi.org/10.1287/mnsc.2025.00535) combines three randomized field experiments with 4,867 developers and reports a 26.08 percent increase in completed tasks, with a standard error of 10.3 percent. The authors explicitly note noise and variation across experiments, and larger effects for less experienced developers.

A different [randomized study of experienced open-source developers](https://arxiv.org/abs/2507.09089) examined 16 participants completing 246 tasks in mature repositories they knew well. In that setting, early-2025 AI tools increased completion time by 19 percent even though developers expected and perceived a speedup.

A [customer-support field study](https://doi.org/10.1093/qje/qjae044) of 5,172 agents found an average productivity increase around 14 percent, with larger gains among less experienced workers and changes across handling time, concurrency, and resolution.

These results can coexist. The intervention, user expertise, task, codebase, workflow, outcome measure, and model era differ.

The correct question is not, "Does AI increase productivity by X percent?" It is:

> For this population doing this work under this process, which complete outcome changed, by how much, and at what quality and cost?

That is a unit-economics question with an experimental design attached.

## Fixed cost determines whether a good unit is a good business case

Suppose engineered automation costs $12,000 to integrate and validate. Its variable outcome cost is $0.66 versus $10.60 manually, a contribution saving of $9.94.

Simple payback before ongoing fixed operations is:

```text
payback outcomes = $12,000 / $9.94 = 1,208 outcomes
```

At 8,000 monthly outcomes, gross monthly savings are $79,520 and simple payback is about 0.15 month. At 200 monthly outcomes, savings are $1,988 and payback is about 6.0 months.

Now add $2,000 monthly for monitoring, evaluation maintenance, on-call ownership, and vendor minimums. The low-volume case becomes negative until volume or value rises.

::figure{template=matrix caption="Automation candidates by monthly volume and value at risk" items="High-volume reversible :: 85,80 | High-volume consequential :: 85,25 | Low-volume reversible :: 20,70 | Low-volume consequential :: 20,15"}

Volume is not the only way to win. One rare workflow may justify automation if speed captures revenue or avoids a large delay. Put that value explicitly in the model rather than pretending all labor minutes are equal.

## The CFO worksheet

For each candidate workflow, collect one month of baseline data:

### Baseline

- completed outcomes;
- labor minutes by role;
- queue and lead time;
- rework and exception rate;
- error severity and loss;
- demand abandoned or delayed;
- fixed systems and supervision cost.

### Automated path

- attempts and branches per outcome;
- input, cached-input, and output tokens by step;
- model route and fallback;
- tool and sandbox cost;
- verification cost;
- human escalation rate and minutes;
- recovery and compensation;
- false acceptance found later;
- fixed build and monthly operating cost.

### Decision outputs

```text
variable saving per outcome
monthly contribution after ongoing fixed cost
payback period
cost per verified outcome at p50 and p95
expected loss and worst credible loss
capacity and lead-time change
```

Do not approve an automation business case with no acceptance rule, no baseline error rate, or no owner for later corrections.

## Three moves that usually improve the unit

### 1. Prevent work before making work cheaper

Reject unauthorized, duplicate, impossible, or already-resolved tasks at the gateway. A request prevented costs less than a request routed well.

### 2. Spend capability where uncertainty is concentrated

Use deterministic code for arithmetic and hard policy. Use cheaper models for mechanical transformations. Reserve expensive reasoning and human judgment for ambiguity or consequence. This is routing across methods, not only models.

### 3. Treat the verifier as production equipment

An accurate, cheap verifier raises the safe automation frontier. It stops retries, routes exceptions, lowers accepted-error loss, and creates training signals. Measure its false acceptance and false rejection separately.

## Builder playbook: run a 30-day unit experiment

1. Select one bounded outcome with at least 100 monthly examples or unusually high time value.
2. Write a shared completion and acceptance contract for manual and automated paths.
3. Measure the manual baseline, including errors and rework.
4. Shadow-run automation without side effects for a representative sample.
5. Record every model, tool, retry, verifier, and human minute by outcome.
6. Review false acceptance after a delay, not only at initial completion.
7. Segment results by task difficulty and exception class.
8. Calculate variable cost, fixed payback, expected loss, and tail constraints.
9. Change one lever at a time: prevent, route, cache, constrain, verify, or redesign.
10. Expand only if verified outcome economics hold outside the happy path.

The model invoice is useful evidence. It is not the business case.

## References

1. [Lingjiao Chen, Matei Zaharia, and James Zou, *FrugalGPT* (TMLR 2024)](https://arxiv.org/abs/2305.05176). Studies model selection and cascades for cost-quality optimization.
2. [Isaac Ong et al., *RouteLLM: Learning to Route LLMs from Preference Data* (ICLR 2025)](https://proceedings.iclr.cc/paper_files/paper/2025/hash/5503a7c69d48a2f86fc00b3dc09de686-Abstract-Conference.html). A learned routing framework with benchmark cost-quality evaluation.
3. [Stanford HAI, *2026 AI Index: Technical Performance*](https://hai.stanford.edu/ai-index/2026-ai-index-report/technical-performance). Current synthesis of model and agent benchmark trends, reliability, and competitive convergence.
4. [Joel Becker et al., *Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity*](https://arxiv.org/abs/2507.09089). A randomized trial showing a slowdown in a specific experienced-developer setting.
5. [Kevin Zheyuan Cui et al., *The Effects of Generative AI on High-Skilled Work* (Management Science, 2026)](https://doi.org/10.1287/mnsc.2025.00535). Three field experiments reporting heterogeneous developer productivity effects.
6. [Erik Brynjolfsson, Danielle Li, and Lindsey Raymond, *Generative AI at Work* (Quarterly Journal of Economics, 2025)](https://doi.org/10.1093/qje/qjae044). Field evidence from customer support, including heterogeneous effects and outcome components.


## FAQ

### What is cost per verified outcome?

It is the total cost of execution, retries, tools, infrastructure, verification, human escalation, recovery, and expected accepted-error loss divided by outcomes that clear a declared acceptance rule. It connects inference inputs to the finished work the business actually buys.

### Why is cost per token a poor AI automation KPI?

A workflow can use many model calls and non-model steps, and cheap calls can increase retries or review. Cost per token measures one input price. It does not reveal whether the task succeeded, was accepted, or created a costly error.

### How should retry cost be calculated?

Classify failure first. Independent transient failures can use a geometric expected-attempt model. Route-specific failures need a changed route, missing preconditions need evidence or escalation, and systematic failures should stop. Treating all failures as independently retryable understates cost and can create infinite loops.

### How do human reviews affect AI ROI?

Multiply escalation probability by average review minutes and the loaded reviewer rate, then include correction and queue effects. Even a small escalation percentage can cost more than all model calls, especially when reviewers are scarce specialists.

### Should expected loss be enough to approve automation?

No. Expected value is one input. Legal duties, irreversibility, capital limits, safety constraints, and worst credible loss can require hard gates even when the average economics look positive.

### How can model routing reduce cost safely?

Evaluate routes under a fixed acceptance bar and include routing overhead, fallbacks, escalation, and false acceptance. Use cheaper capability where it preserves verified outcomes and reserve stronger models or humans for uncertainty and consequence.


## Test yourself

### 1. What must be equal when comparing manual and automated unit economics?

- [ ] Token count
- [x] The definition of a completed outcome
- [ ] Number of tools
- [ ] Provider list price

**Explanation:** Different completion definitions manufacture a misleading denominator.

### 2. A systematic policy error causes every retry to fail. Which model is inappropriate?

- [ ] Classify and stop
- [ ] Repair the policy
- [x] The independent geometric retry assumption
- [ ] Escalate the class

**Explanation:** Geometric retry assumes each attempt has an independent chance of success.

### 3. Five percent of tasks need ten minutes from a $90-per-hour reviewer. What expected cost does this add per task?

- [ ] $0.075
- [ ] $0.45
- [x] $0.75
- [ ] $7.50

**Explanation:** 0.05 multiplied by 10/60 hour multiplied by $90 equals $0.75.

### 4. Where should an accepted but later-discovered wrong outcome appear?

- [ ] Nowhere because it passed verification
- [ ] Only in model token cost
- [x] Expected failure loss and verifier quality
- [ ] As a successful cache hit

**Explanation:** Verification has scope and can falsely accept. Later loss remains part of the unit.

### 5. When does a cheaper model route create value?

- [ ] Whenever its token price is lower
- [x] When total cost per verified outcome falls within latency and risk constraints
- [ ] Only when it has fewer parameters
- [ ] When it never calls a verifier

**Explanation:** The complete outcome unit, not model price alone, determines value.