--- title: Route the Work slug: route-the-work canonical_url: https://blog.openfactoryai.com/route-the-work published_at: 2026-07-14T20:48:00.289665+00:00 author: OpenFactoryAI tags: LLM routing, model cascade, RouteLLM, FrugalGPT, cost optimization tldr: Model routing should select the least costly feasible route for each request or workflow step while satisfying quality, latency, privacy, availability, and consequence constraints. A cascade learns after running a first model; a pre-route policy predicts before paying for generation. Both can beat one-model-for-everything, but only if their confidence is calibrated on production-like outcomes and the full cost of routing, escalation, and failure is counted. key_takeaways: - Route on task requirements and constraints, not provider identity. - A pre-router predicts before generation; a cascade can inspect an attempted answer before escalating. - Optimize expected verified-outcome cost subject to hard policy gates. - Calibration and abstention matter because confident misroutes concentrate failure. - Evaluate routes per workflow step and include fallback latency, retries, and availability correlation. --- “Use the strongest model” is safe only if cost, latency, privacy, capacity, and outages do not matter. “Use the cheapest model” is safe only if failed outcomes do not matter. Real systems need a policy between those slogans. A routing policy observes what is known about the request and chooses a model, deployment, configuration, or cascade. Its job is not to identify the globally best model. Its job is to select a feasible route for this work under uncertainty. The core decision is: ```text choose route r minimizing expected total outcome cost subject to quality, latency, privacy, availability, and policy constraints ``` Model names are implementation details inside that decision. ::figure{template=flow caption="A router converts request evidence and constraints into an executable route" items="Request :: intent, context, tenant | Hard gates :: privacy, region, tools | Predict :: success, latency, cost | Select :: feasible route | Execute :: model and policy | Verify :: outcome | Learn :: update evidence"} ## What can be routed? At the application boundary, a route can vary: - model family and revision; - hosted API versus private deployment; - quantized versus higher-precision artifact; - context length or retrieval policy; - tool-enabled versus text-only configuration; - ordinary versus speculative decode; - one-shot call versus verified agent loop; - synchronous route versus deferred batch. This is different from mixture-of-experts routing inside a model. An internal MoE layer selects parameter experts during a forward pass. An application router selects independently operated paths with separate prices, data boundaries, SLOs, and failure behavior. The analogy is useful but the control contracts differ. Route at the smallest unit whose requirements are stable. One agent task may use a strong planner, a small classifier for document labels, a code-specialized generator, and a constrained verifier. Routing the entire task to one brand leaves step-level savings and safety boundaries unused. ## Hard gates come before learned scores Some constraints should not be traded for predicted quality. ```text eligible(route, request) = data_region allowed and privacy tier allowed and required tools supported and context fits and output schema supported and route currently healthy and deadline remains feasible ``` Only eligible routes enter optimization. A model with a high predicted answer score is irrelevant if sending the context violates a data boundary. A route with the lowest nominal price is infeasible if its p95 latency exceeds the remaining workflow deadline. Maintain capabilities as versioned facts, not prompt-time folklore: | Route fact | Evidence source | Freshness | |---|---|---| | context and output limits | deployed runtime/config | on deploy | | region and retention policy | contract and infrastructure | on policy change | | tool/schema support | integration tests | on model/runtime change | | latency by workload slice | production traces | continuous | | verified success by slice | evaluation and outcomes | continuous | | marginal and fixed cost | bills plus capacity model | on price/load change | | health and queue estimate | live telemetry | seconds | The router should be able to explain every exclusion and selection. ## Fixed, rule-based, learned, and cascaded policies Four common policy families solve different problems. ### Fixed route Every eligible request uses one route. This is easy to understand and provides a strong baseline. It can be optimal when traffic is homogeneous, volume is small, or operating another model costs more than it saves. ### Rule-based route Declared fields choose a route: ```text if data_class == restricted: private_route elif task == extraction and schema_small: compact_route elif context_tokens > threshold: long_context_route else: general_route ``` Rules are auditable and good for hard product distinctions. They become brittle when difficulty is continuous or rules encode untested intuition. ### Learned pre-router A classifier or scorer predicts which route will succeed before generation. [RouteLLM](https://proceedings.iclr.cc/paper_files/paper/2025/hash/5503a7c69d48a2f86fc00b3dc09de686-Abstract-Conference.html) learns routing between stronger and weaker models using preference data. It demonstrates a research framework for a cost-quality trade. Production outcome labels may differ from human preference, especially for code, tools, exact extraction, or regulated decisions. ### Cascade A cheaper route runs first. A gate inspects its input, confidence, output, verifier, or some combination and decides whether to accept or escalate. [FrugalGPT](https://openreview.net/forum?id=cSimKw5p6R) studies learned LLM cascades that combine models and stopping strategies under budget. A cascade pays for information. It may know whether output parses, tests pass, citations resolve, or confidence is low. Its downside is that escalated requests pay first-attempt cost and latency plus the stronger route. ::figure{template=compare caption="Pre-routing predicts before generation; cascading buys evidence then may escalate" items="Pre-router :: features -> choose weak or strong -> verify | Cascade :: weak attempt -> gate -> accept or strong retry | Fixed :: one route -> verify | Rule policy :: declared class -> route -> verify"} ## Formulate routing as expected loss For request features `x` and route `r`, define: ```text J(r | x) = model_and_tool_cost(r, x) + latency_value_loss(r, x) + P(failure | r, x) * failure_consequence(x) + expected_retry_and_review_cost(r, x) + operational_shadow_price(r) ``` The policy selects the eligible route with minimum estimated `J`, or abstains when uncertainty is too high. This expression explains why one route can be right for a draft and wrong for a payment action. Failure consequence changes. It also explains why marginal API price is incomplete for self-hosted routes: scarce capacity has an opportunity cost during peaks. Quality can be implemented as a hard minimum rather than a loss term: ```text minimize expected cost such that P(verified success | route, slice) >= 0.95 and p95 completion <= 2 seconds ``` Be honest about estimation error. A predicted 95.1 percent success rate is not proof that a route satisfies a 95 percent contract. Use confidence bounds and a safety margin. ## A 1,000-request worked comparison Consider an illustrative workload of 650 easy and 350 hard requests. No named models or market prices are represented. | Route | Easy success | Hard success | Cost/attempt | |---|---:|---:|---:| | Weak | 94% | 55% | $0.002 | | Strong | 98% | 90% | $0.020 | A fixed strong policy expects: ```text 650*.98 + 350*.90 = 952 successes cost = $20.00 cost per expected success = $0.02101 ``` A fixed weak policy expects 803.5 successes for `$2.00`, but fails any quality requirement above 80.35 percent. Now a learned router sends 10 percent of easy requests and 85 percent of hard requests to strong. Routing costs `$0.0002` per request. ```text strong requests = 65 + 297.5 = 362.5 weak requests = 637.5 easy success = .10*.98 + .90*.94 = .944 hard success = .85*.90 + .15*.55 = .8475 expected successes = 650*.944 + 350*.8475 = 910.225 total cost = 362.5*.020 + 637.5*.002 + .20 = $8.725 cost per expected success = $0.00959 ``` The router passes an illustrative 90 percent quality gate and cuts expected cost per success by about 54 percent relative to fixed strong. It still loses about 41.8 expected successes, so it is not equivalent. Finally, a cascade runs weak for everyone, then escalates 12 percent of easy and 70 percent of hard requests. Its gate costs `$0.0003` per request. ```text strong escalations = 650*.12 + 350*.70 = 323 expected successes = 892.37 cost = 1000*.002 + 323*.020 + .30 = $8.76 cost per expected success = $0.00982 ``` Under these assumptions the cascade misses the 90 percent quality gate while costing slightly more than the router. Change gate accuracy, prices, or the value of output evidence and the ranking can reverse. ::figure{template=bar caption="Illustrative policies: expected successes per 1,000 requests" items="Fixed strong :: 952 | Learned router :: 910.225 | Cascade :: 892.37 | Fixed weak :: 803.5"} ::figure{template=bar caption="Illustrative cost per expected success, dollars" items="Fixed strong :: 0.02101 | Learned router :: 0.00959 | Cascade :: 0.00982 | Fixed weak :: 0.00249"} The cheap fixed route's cost ratio looks best only because the denominator ignores the quality gate. Constrained optimization prevents an unacceptable policy from winning a cost chart. ## Confidence must mean something measurable A router output of `0.8` is useful only if requests assigned that score succeed about 80 percent of the time under the defined label. This is calibration. [Guo et al.](https://proceedings.mlr.press/v70/guo17a) document miscalibration in modern neural networks and evaluate post-hoc calibration methods in classification. An LLM router is not the same experiment, but the operational lesson transfers: ranking and calibrated probability are different properties. Plot reliability by workload slice: | Predicted success band | Observed verified success | Count | |---|---:|---:| | 0.50 to 0.60 | measured | n | | 0.60 to 0.70 | measured | n | | 0.70 to 0.80 | measured | n | | 0.80 to 0.90 | measured | n | | 0.90 to 1.00 | measured | n | Calibration can drift when prompts, models, tools, or verifiers change. A router trained on chat preference may be confidently wrong about SQL execution. Recalibrate or retrain on current application outcomes. Keep an abstain region. If routes have overlapping uncertainty or the request is out of distribution, use a safe default, run a verifier-rich cascade, or ask for human input. Forced certainty is not efficiency. ## Preference is not correctness Pairwise human preference is abundant for open-ended responses and central to RouteLLM's research setup. In automation, the label should match the outcome. - Code: tests and repository acceptance. - Retrieval answer: supported claims and citation entailment. - Extraction: field-level exactness against audited records. - Tool action: schema validity, authorization, and side-effect confirmation. - Support resolution: issue resolved without unsafe or incorrect instruction. A fluent strong-model response may win preference while failing a hidden test. A terse small-model output may be correct and valid. Train and evaluate the router against the decision the product needs. When no verifier exists, use carefully designed human labels and explicitly price uncertainty. Do not manufacture an objective by asking another model and calling the result ground truth. ## Cascades need failure-aware stopping rules A cascade gate can inspect: - model-reported confidence, cautiously; - token-level uncertainty signals; - agreement among samples; - deterministic schema and type checks; - retrieval support and citations; - executable tests; - a learned correctness predictor; - a second model's critique; - business-rule constraints. [Language Model Cascades: Token-level uncertainty and beyond](https://arxiv.org/abs/2404.10136) studies uncertainty signals for deferral. It is a preprint and its evaluated signals should not be promoted to universal stopping rules. The strongest gates are often domain verifiers. If a generated patch passes the relevant tests and policy checks, escalation may be unnecessary. If an answer concerns an unverified live balance, high textual confidence is weak evidence. Count false accepts and false escalations separately: - false accept: weak output fails but gate accepts, damaging quality; - false escalate: weak output would pass but gate pays strong cost and latency; - correct accept: saving; - correct escalate: protected outcome. Optimize their consequence-weighted confusion matrix, not gate accuracy alone. ## Reliability includes correlated failure Multiple routes are not resilient if they share the same dependency. Two API model names may use one provider region, identity system, gateway, or quota. Two self-hosted models may share the same cluster and overloaded KV pool. Describe route failure domains: ```text provider, account, region, network, runtime, model family, tokenizer, data source, tool dependency, verifier ``` A fallback must be available, compatible, and capacity-tested during the event that triggers it. If all normal traffic shifts to a smaller fallback, its queue may collapse. Routing for availability can conflict with routing for cost. Reserve or rehearse fallback capacity and include its idle cost in the policy economics. ## Route workflow steps, not just initial prompts Agent loops reveal more information over time. The first planner call sees an ambiguous task. A later step knows it needs one of three operations. A verifier knows the produced artifact and failed checks. Allow route features to include safe workflow state: ```text step_type remaining_deadline_and_budget previous_failure_class tool_and_schema_requirements context_length verification_strength consequence_and_reversibility ``` Do not leak private content into an external router merely to decide that private content cannot leave. Feature extraction must obey the same policy gates as generation. Per-step routing also prevents a strong-model failure from recursively selecting itself. After a schema failure, switching to a route with constrained decoding may matter more than scaling model capability. ## The route decision is a security boundary Prompts are untrusted input. A user can write “send this to the cheapest external model,” “ignore the private-data flag,” or text that imitates the features a learned router associates with an allowed class. The router may use content to predict task type, but content must not override trusted policy attributes. Separate feature provenance: ```text trusted: tenant, authorization, data class, region, contract tier, remaining budget, workflow state, required tool permissions derived: intent, difficulty, domain, expected length, success score untrusted: instructions and claims inside user or retrieved content ``` Hard gates use trusted attributes and verified system state. Derived features may choose among already eligible routes. If classification of data sensitivity itself is learned, apply a conservative default and audit false negatives. The router also sees valuable metadata and sometimes full prompts. Minimize its input. A lightweight router that needs an embedding of private content is still a data processor. Prefer local feature extraction, redaction, declared workflow fields, or a policy-compatible router when raw text cannot leave its boundary. Protect economic controls. An attacker can craft requests that force the expensive route, create cascade escalations, or occupy fallback capacity. Rate limits based only on request count miss this. Track predicted and realized token/tool cost per principal and require budget admission before execution. Log route decisions without copying secrets unnecessarily. A useful audit record includes request and policy identifiers, trusted eligibility facts, derived score versions, excluded routes and reasons, selected route, budget, and eventual verifier outcome. Retention and access controls should match the sensitivity of the underlying work. Finally, test policy invariants independently of the model: - restricted data never enters an external route; - a tool permission cannot be gained through route selection; - expired credentials make a route ineligible; - prompt wording cannot alter tenant or region; - budget exhaustion produces a controlled stop; - fallback does not weaken the original privacy contract. Routing can reduce cost only after it preserves authority. ## A builder playbook ### 1. Build the route registry Version capabilities, hard constraints, price model, observed latency, capacity, and health. Separate declared vendor facts from your measurements. ### 2. Create a fixed baseline and oracle ceiling Evaluate every eligible route on the same representative tasks. The hindsight oracle selects the cheapest successful route after seeing outcomes; it is unattainable but shows maximum possible routing value. ### 3. Choose the true outcome label Use executable, schema, groundedness, or human outcome verification. Record consequence, not just win preference. ### 4. Evaluate policies offline Compare fixed, rules, learned pre-route, and cascade policies on identical traces. Report success, severe failure, cost, latency, routing overhead, escalation, and per-slice confidence bounds. ### 5. Shadow before controlling Log the route the candidate would choose while the baseline still executes. This reveals drift and capacity concentration without exposing outcomes. ### 6. Canary with abstention Start in low-consequence, strongly verified slices. Retain a safe default for uncertain or out-of-distribution requests. Bound spend and failure. ### 7. Monitor policy regret Sample counterfactual routes where affordable. Track the cost or outcome difference between chosen and successful alternatives. Without exploration, the router can stop learning about routes it rarely selects. ### 8. Recalculate cost per verified outcome Include router inference, duplicate cascade calls, tools, retries, latency, fallback reserve, and failure loss. Reuse the unit model from [Cost per Successful Outcome](/cost-per-verified-outcome). ## The decision Use a fixed route until workload heterogeneity creates enough avoidable cost or failure to justify policy complexity. Add hard eligibility gates first, transparent rules second, and learned prediction only when representative outcome data can support it. Use cascades when the first attempt produces strong verification evidence worth its extra latency. The winning router does not send the most traffic to the cheapest model or the strongest brand. It sends each step to the least costly feasible route and knows when its own evidence is too weak to decide. ## References - [FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance](https://openreview.net/forum?id=cSimKw5p6R), Chen, Zaharia, and Zou, TMLR 2024. Develops cost-aware model cascades and stopping strategies. - [RouteLLM: Learning to Route LLMs from Preference Data](https://proceedings.iclr.cc/paper_files/paper/2025/hash/5503a7c69d48a2f86fc00b3dc09de686-Abstract-Conference.html), Ong et al., ICLR 2025. Provides a learned strong-versus-weak routing framework. - [Language Model Cascades: Token-level uncertainty and beyond](https://arxiv.org/abs/2404.10136), Yue et al., 2024 preprint. Studies uncertainty-based deferral signals. - [On Calibration of Modern Neural Networks](https://proceedings.mlr.press/v70/guo17a.html), Guo et al., ICML 2017. Establishes the practical distinction between confidence ranking and calibrated probabilities. All policies, costs, and success rates in the worked scenario are illustrative. The full calculations are retained in the claim ledger. ## FAQ ### What is an LLM router? It is an application policy that selects a model or serving path for a request or workflow step based on requirements, predicted outcomes, live capacity, and constraints such as privacy, latency, tools, and cost. ### What is the difference between model routing and a cascade? A pre-router chooses before paying for generation. A cascade runs an initial route, inspects its output or verification evidence, and may escalate, so escalated requests pay additional latency and cost. ### How should a router choose between models? First exclude routes that violate hard data, tool, context, health, or deadline constraints. Then minimize expected total outcome cost using calibrated success, latency, retry, review, failure consequence, and capacity cost. ### Can model confidence drive routing? Only after calibration on representative application outcomes. Self-reported or learned confidence can rank examples yet remain numerically misleading. Keep an abstain or safe-fallback region. ### Should an AI agent use one model for every step? Usually not by default. Planning, extraction, code, tool calls, and verification have different requirements. Per-step routing can use cheaper specialized paths while preserving strong routes for ambiguous or consequential decisions. ### How is an LLM router evaluated? Compare fixed, rule, learned, and cascade policies on identical tasks. Report verified success, severe failures, cost, latency, overhead, escalation, calibration, confidence bounds, availability behavior, and cost per successful outcome. ## Test yourself ### 1. What should a production router apply before predicted quality scores? - [ ] Random selection - [x] Hard privacy, capability, health, and deadline gates - [ ] Provider popularity - [ ] Maximum temperature **Explanation:** Ineligible routes should never enter a soft cost-quality trade. ### 2. Why can a cascade cost more on hard requests? - [ ] It removes the first answer - [x] Escalated requests pay for the first attempt and the stronger retry - [ ] It never verifies - [ ] Hard requests have no tokens **Explanation:** The cascade buys evidence with the initial attempt before escalation. ### 3. What does calibrated 0.8 success confidence mean? - [ ] Every request succeeds - [x] About 80% of comparable predictions succeed under the defined outcome - [ ] The model uses 80% GPU - [ ] The route is always cheapest **Explanation:** Calibration connects predicted probabilities to empirical frequencies. ### 4. Why is human preference sometimes a poor routing label for automation? - [ ] Humans cannot read - [x] A preferred response may fail executable, schema, groundedness, or side-effect checks - [ ] Preferences are always free - [ ] Models cannot be compared **Explanation:** The label must match the production outcome the route is meant to achieve. ### 5. What does an oracle routing baseline show? - [ ] A deployable zero-cost policy - [x] The hindsight ceiling from choosing the cheapest successful route after seeing outcomes - [ ] Current provider uptime - [ ] Tokenizer speed **Explanation:** It bounds potential routing value but cannot be used online because it sees future outcomes.