--- title: The Loop Is the Runtime slug: the-loop-is-the-runtime canonical_url: https://blog.openfactoryai.com/the-loop-is-the-runtime published_at: 2026-07-14T20:52:00.289665+00:00 author: OpenFactoryAI tags: Loop Engineering, AI agents, agent runtime, tool use, state machines tldr: A prompt can propose the next move. An agent runtime must remember verified state, expose typed actions, authorize effects, observe receipts, recover from failure, and know when to stop. The minimum dependable loop is not “think, act, repeat.” It is an explicit state machine whose every transition has evidence, budget, authority, and a terminal condition. Loop Engineering begins where prompt engineering can no longer protect the system. key_takeaways: - The loop, not the prompt, owns task progress. - Model text is a proposal; verified events and typed state are the record. - Every transition needs preconditions, action authority, postconditions, and failure edges. - Stop rules are product logic, not a plea inside the system prompt. - Measure calls, tokens, tools, state growth, verification, and outcome together. --- Prompt engineering asks: what should the model say next? Loop Engineering asks harder questions: - What state is true now? - Which transition is legal? - What evidence would prove it happened? - How much more work may be spent? - What happens after a timeout or partial side effect? - When must the system stop? The moment a model calls a tool and waits for the world to answer, the prompt is no longer the system. It is one computation inside a runtime. An agent is therefore not “an LLM with tools.” It is a stateful program that uses inference to select or construct transitions under uncertainty. ::figure{template=state-machine caption="The minimum agent runtime turns proposals into verified state transitions" items="Accepted :: bind task and authority | Observe :: read trusted state | Decide :: propose next transition | Validate :: schema, policy, budget | Act :: invoke idempotent tool | Verify :: inspect receipt and postcondition | Update :: append event and reduce state | Stop :: success, limit, denial, or escalation"} ## A single call predicts; a loop operates A single inference call maps input context to output tokens: ```text y ~ model(context, policy, sampling) ``` A loop repeatedly combines model output with environment change: ```text state_t -> observation_t -> proposed_action_t -> validation_t -> environment_result_t -> verified_state_t+1 ``` The environment result may be a database receipt, test output, browser state, human approval, repository diff, API error, or timeout. It can contradict the model's prediction. This difference is the foundation. A model can write “the deployment succeeded.” Only the deployment system's receipt can establish that state. [ReAct](https://openreview.net/forum?id=WE_vluYUL-X) studies interleaving reasoning traces and actions, allowing observations to inform later reasoning. The research mechanism was important because it moved beyond producing a complete answer before interacting. A production runtime must add typed state, authority, durability, budgets, and recovery that are not established merely by an interleaved prompt format. ## Define the state before defining the prompt State is the smallest typed representation needed to decide the next legal transition and verify completion. For a repository-change agent: ```json { "task_id": "task_7", "goal": "fix failing CSV export test", "repo_revision": "4e91...", "phase": "failed_check", "changed_files": ["src/export.py", "tests/test_export.py"], "last_check": {"command_id": "check_19", "exit": 1}, "open_failures": ["test_escaped_newline"], "budgets": {"model_calls_left": 5, "wall_seconds_left": 240}, "authority": {"write_repo": true, "push": false, "deploy": false}, "event_cursor": 31 } ``` This is not the whole transcript. It excludes speculative thoughts and superseded plans. Each field has provenance in the event log. Model-visible context is assembled from state, current evidence, eligible tools, and recent events. The model may suggest a state update, but the reducer accepts it only after validating the associated event. ## Events are facts; reflections are hypotheses An append-only event can say: ```json { "event_id": "evt_32", "kind": "check.completed", "command_id": "check_19", "exit_code": 1, "artifact": "logs/check_19.txt", "started_at": "...", "finished_at": "..." } ``` A reflection can say: ```text The failure is probably caused by newline escaping in the CSV writer. ``` The event is an observed fact. The reflection is a useful hypothesis that may guide the next action. Do not store both in the same field under “memory.” [Reflexion](https://papers.neurips.cc/paper_files/paper/2023/hash/1b44b878bb782e6954cd888628510e90-Abstract-Conference.html) studies verbal feedback retained across trials. The mechanism can help an agent avoid repeating a mistake in evaluated settings. In production, a verbal reflection needs source links, expiry, and a confidence status. A wrong diagnosis repeated in later context can lock the loop into a failure attractor. ::figure{template=compare caption="A dependable loop separates observed events from model-authored interpretation" items="Verified event :: check 19 exited 1, log hash stored | Derived state :: open failure test_escaped_newline | Reflection :: likely newline escaping bug | Plan :: inspect writer then add boundary test | Authority :: may edit repo, may not push | Receipt :: patch hash and passing check"} ## A transition is a contract Represent each transition with: ```text name allowed source states preconditions typed action required authority budget charge timeout and retry class success postcondition failure states compensation or escalation ``` Example: ```text transition: apply_patch from: inspected | failed_check precondition: base revision unchanged; patch touches allowed paths authority: repository.write charge: one mutation attempt success: tool receipt contains new tree hash and changed paths failure: conflict | invalid_patch | policy_denied | timeout_unknown ``` `timeout_unknown` matters. A timeout does not prove no effect occurred. The runtime checks an idempotency key or current repository tree before retrying. This is why tools should be narrow typed operations rather than a universal shell with prose instructions. The action language defines what the loop can do. ## Plans are revisable, not executable truth [Plan-and-Solve](https://aclanthology.org/2023.acl-long.147/) separates generating a plan from executing subtasks in evaluated reasoning tasks. Planning is useful because it exposes missing steps and budgets before action. An agent plan should contain: - objective and success evidence; - ordered or dependent steps; - information gaps; - expected tool and model cost; - risky or irreversible transitions; - checkpoints and stop conditions. The plan is invalidated when observations change its assumptions. If inspection reveals a generated file, editing it directly may be the wrong step. If the repository revision changes, the patch plan needs rebase or restart. Do not grant authority to the whole plan. Authorize each transition against current state. A human approving “prepare a release” has not necessarily approved deleting a database discovered three steps later. ::figure{template=branch caption="A plan proposes a path; observations decide which branch remains legal" items="Goal :: repair export | Plan :: inspect, patch, test | Inspect :: generated file discovered | Branch A :: edit generated file, reject | Branch B :: edit source template | Test :: failure persists | Replan :: inspect escaping layer | Verify :: full check passes"} ## Tool use needs a control plane [Toolformer](https://proceedings.neurips.cc/paper/2023/hash/d842425e4bf79ba039352da0f658a906-Abstract-Conference.html) trains a model to decide which APIs to call, when, with what arguments, and how to incorporate results. That addresses capability selection inside model behavior. Production control remains external: ```text model proposes tool + arguments runtime checks schema runtime checks tenant, principal, object, budget, and workflow state tool executes with idempotency and timeout runtime validates receipt event log records effect model observes bounded result ``` The tool registry should describe: - input and output schemas; - read or write effect; - required permissions; - idempotency semantics; - timeout ambiguity; - rate and cost budget; - data classification; - side-effect reversibility; - health and version. A tool being technically callable does not make it eligible for the current step. ## Stop conditions are executable policy “Stop when done” is not a stop condition. It delegates the terminal decision to the same uncertain model that is generating work. Define terminal states: ### Verified success Named postconditions hold. Tests pass, target artifact exists, citations support claims, or transaction receipt is committed. ### Infeasible Required evidence, capability, permission, or dependency is unavailable. More retries cannot change it inside the task boundary. ### Budget exhausted Maximum model calls, tokens, tool cost, mutations, elapsed time, or human review is reached. ### Policy denied The requested or discovered action violates authority or safety rules. ### Repeated state The loop revisits an equivalent failure state without new evidence. A cycle detector stops or escalates. ### Deadline missed No remaining path can finish before the outcome deadline. ### Human escalation Consequence, ambiguity, or approval policy requires a person. ::figure{template=matrix caption="Stop rules distinguish successful completion from controlled non-completion" items="Postcondition proven :: verified success | Missing permission :: policy denied | Same failure three times :: repeated state | Calls exhausted :: budget stop | Deadline infeasible :: timeout stop | Irreversible ambiguity :: human escalation"} The loop returns a typed terminal result with evidence. A controlled `needs_approval` is not the same as task failure. ## Trace one complete task Consider an illustrative repository repair. The numbers are invented, not customer data. | Step | Transition | Calls | Input | Output | Tool time | Result | |---|---|---:|---:|---:|---:|---| | 1 | accepted -> scoped | 1 | 2,200 | 420 | 0.3 s | plan | | 2 | scoped -> inspected | 2 | 7,400 | 710 | 2.2 s | relevant files/tests | | 3 | inspected -> changed | 1 | 6,100 | 980 | 0.8 s | first patch | | 4 | changed -> failed_check | 1 | 4,800 | 260 | 11.0 s | one failing test | | 5 | failed_check -> changed | 2 | 9,600 | 880 | 1.3 s | diagnosis and fix | | 6 | changed -> verified | 1 | 5,200 | 210 | 12.4 s | checks pass | | 7 | verified -> committed | 0 | 0 | 0 | 0.4 s | receipt stored | Totals: ```text 8 model calls 35,300 input tokens 3,460 output tokens 28.4 seconds tool time 2 mutations 1 failed verification 1 verified terminal outcome ``` ::figure{template=trace caption="Illustrative loop trace attributes calls, tools, failure, and proof" items="Scope :: 1 call / 0.3s | Inspect :: 2 calls / 2.2s | Patch 1 :: 1 call / 0.8s | Check 1 :: fail / 11.0s | Diagnose :: 2 calls / 1.3s | Check 2 :: pass / 12.4s | Commit receipt :: 0 calls / 0.4s"} The first patch was not a failed task. It was a failed transition attempt that produced evidence. Whether that evidence was worth its cost depends on how efficiently the loop used it. ## State growth can quietly become the bottleneck If the full transcript is replayed on every call, input tokens grow roughly with accumulated history. A loop with ten calls can pay for early content ten times. Use: - stable prefix caching for policy and tool schemas; - typed current state instead of full replay; - event retrieval by relevance and provenance; - recent exact observations where wording matters; - summaries that link back to original events; - separate scratch space with short lifetime. Track context composition per call. If the same failed log is sent six times after its cause is resolved, the reducer is not maintaining state. The context rules from [The Window Is Not the Memory](/context-is-a-budget) become runtime memory rules here. ## Parallel branches are child runtimes, not extra thoughts An agent may launch three searches, two candidate patches, or several independent analyses. Parallelism can reduce critical-path latency and increase exploration. It also creates ownership questions: - Which branch may mutate shared state? - How is a winning branch selected? - What cancels the losers? - Can outputs be combined safely? - What budget does each child inherit? - What happens if a loser finishes after the task commits? Give every branch an identity, parent, budget, state snapshot, and merge rule: ```json { "branch_id": "br_7b", "parent_event": "evt_18", "base_revision": "4e91...", "purpose": "candidate patch B", "authority": {"sandbox_write": true, "shared_write": false}, "budget": {"calls": 3, "seconds": 90}, "merge_policy": "verified_winner_only" } ``` Read-only searches can often run concurrently. Mutating branches should work in isolated sandboxes or use optimistic concurrency against a declared base revision. Do not allow two model branches to write the same live object and hope the later prompt resolves it. The coordinator verifies candidates under the same test and policy contract. It does not choose the most persuasive explanation. When one candidate wins, the runtime cancels remaining work and records cancellation receipts. Measure tokens and tool time computed after cancellation, because late cancellation turns parallel exploration into invisible capacity waste. Merging is a transition with preconditions. Two individually valid patches may conflict or jointly fail tests. Two retrieved facts may refer to different revisions. Re-run verification on the combined artifact. ::figure{template=branch caption="Parallel exploration stays isolated until a verified merge transition" items="Parent state :: fixed revision and budget | Branch A :: candidate patch in sandbox | Branch B :: candidate patch in sandbox | Branch C :: additional inspection | Verifier :: same tests and policy | Winner :: one admissible artifact | Cancel :: stop losing work | Merge :: check base revision, apply, verify again"} ## Human-in-the-loop is an asynchronous state, not a modal dialog A workflow that needs approval can wait minutes or days. The process may restart, credentials may expire, prices may change, and another actor may modify the target while a person reviews. Represent approval as a durable transition: ```text proposed -> awaiting_approval -> approved | rejected | expired | superseded ``` The approval request binds: - normalized action and artifact hash; - evidence and verifier result shown to the reviewer; - principal requesting the action; - reviewer role required; - maximum amount or scope; - expiry time; - state or revision precondition; - whether changes require a new approval. An approval for patch hash `abc` does not authorize a regenerated patch `def`. Approval for a transfer at one balance snapshot may need revalidation before commit. The runtime checks current state after approval and before execution. Human feedback also has different meanings. “Looks good” may approve an effect, label an outcome, suggest a new requirement, or merely acknowledge a preview. Use typed decisions rather than inserting the sentence into context and guessing. Human latency belongs in the trace and ROI model. Automating 30 seconds of inference around a six-hour approval queue may have little lead-time value, though better evidence can reduce reviewer time. Measure: ```text time awaiting reviewer active review time approval and rejection rate requests returned for missing evidence expired or superseded approvals post-approval validation failures ``` Design escalation packets to minimize reconstruction: current state, proposed action, consequence, evidence, alternatives, budget spent, and the exact decision requested. A person should not need to read the full agent transcript. ## Verification is a transition, not a final ceremony Do not reserve verification for the end. Place cheap checks after transitions where error becomes expensive to carry. ```text parse after generation type-check after edit unit check after local change integration check before merge policy check before authority expansion smoke check after deployment outcome monitor after release ``` Verification depth should grow with consequence and accumulated work. Re-running a full suite after every character is wasteful; waiting until deployment to discover a syntax error is worse. Each verifier returns a typed result with scope. “Tests passed” is incomplete unless the runtime knows which tests, revision, environment, and time. A local unit test does not prove production health. A model critic is not independent evidence unless its limitations are accepted for that gate. Failed verification changes state. The next call receives the failure artifact and remaining budget, not a vague instruction to “try again.” If the same verifier fails on the same signature without new evidence, cycle detection stops the loop. ## Recovery is part of every action The loop can fail between any two observations: - model request times out; - tool accepts action but response is lost; - process crashes after effect but before event append; - state reducer fails; - human approval arrives after task deadline; - another actor changes the object concurrently. Every effect needs an idempotency key and a way to query its status. The runtime records intent before dispatch and receipt after observation. On recovery, it reconciles unknown attempts instead of blindly replaying. Durable execution receives a full treatment later in the AI-native development article. The minimum Loop Engineering rule is simple: if the runtime cannot answer “did this action happen?” after restart, the action is not safely automatable. ## A builder playbook ### 1. Draw the state machine List typed phases, legal transitions, terminal states, and failure edges. If the diagram is only a circle labelled think-act-observe, it is incomplete. ### 2. Define outcome evidence Write machine-checkable postconditions for success where possible. Name when human judgment is required. ### 3. Separate events, state, and reflection Events are immutable observations. State is a validated reduction. Reflections and plans are model-authored hypotheses with provenance and expiry. ### 4. Build the eligible tool registry Include schemas, permissions, effects, idempotency, timeouts, costs, and receipts. Select tools from trusted workflow state before exposing them to the model. ### 5. Install budgets and cycle detection Limit calls, tokens, mutations, tool cost, elapsed time, and repeated equivalent states. Stop before a retry storm becomes the product. ### 6. Trace transitions, not just model calls Record state before/after, evidence, proposal, validator result, authority decision, tool receipt, cost, latency, and terminal outcome. ### 7. Replay failures Run the runtime against stored event traces with model and tool stubs. Test timeouts before effect, timeouts after effect, concurrent state change, invalid receipt, denied authority, and crash recovery. ## The decision Use a single call when one bounded prediction can be independently checked. Build a loop when the task must gather new evidence, perform multiple transitions, or recover from intermediate failure. Then treat the loop as the runtime: typed state, append-only evidence, constrained actions, current authority, explicit budgets, verified postconditions, and executable stops. Prompts can make the next proposal better. Loop Engineering makes the whole task survivable. ## References - [ReAct: Synergizing Reasoning and Acting in Language Models](https://openreview.net/forum?id=WE_vluYUL-X), Yao et al., ICLR 2023. Establishes interleaved reasoning, action, and observation in evaluated language agents. - [Toolformer: Language Models Can Teach Themselves to Use Tools](https://proceedings.neurips.cc/paper/2023/hash/d842425e4bf79ba039352da0f658a906-Abstract-Conference.html), Schick et al., NeurIPS 2023. Studies learned decisions about API use. - [Reflexion: Language Agents with Verbal Reinforcement Learning](https://papers.neurips.cc/paper_files/paper/2023/hash/1b44b878bb782e6954cd888628510e90-Abstract-Conference.html), Shinn et al., NeurIPS 2023. Explores verbal feedback retained across agent trials. - [Plan-and-Solve Prompting](https://aclanthology.org/2023.acl-long.147/), Wang et al., ACL 2023. Separates planning from subtask execution in reasoning prompts. The workflow trace is illustrative. Exact invented inputs and totals are retained in the claim ledger. ## FAQ ### What is Loop Engineering? It is the design of the state, transitions, tools, authority, evidence, budgets, stopping, recovery, and observability around repeated model inference. It begins where improving the next prompt cannot guarantee the whole task. ### What is the difference between an AI agent and an LLM call? A call predicts output from context. An agent runtime uses calls inside a stateful loop that observes an environment, validates and authorizes actions, records receipts, updates verified state, and stops under explicit conditions. ### Should an agent store its full transcript as memory? Not as authoritative state. Keep immutable events, validated typed state, provenance-linked summaries, recent exact observations, and separate model reflections. Retrieve what the next transition needs. ### How does an AI agent know when to stop? The runtime checks explicit terminal conditions: proven success, infeasibility, policy denial, exhausted budget, missed deadline, repeated equivalent state, or required human escalation. The model may propose completion but does not prove it. ### How should agent tool calls be secured? Expose only eligible typed tools, validate arguments, bind authenticated identity and current authorization, enforce budgets, use idempotency for effects, verify receipts, and record events. Model text is a proposal, not permission. ### What should an agent trace contain? State before and after, evidence read, model proposal, validator and authority decisions, tool request and receipt, tokens, cost, latency, retry class, and the terminal outcome. This makes failures attributable and replayable. ## Test yourself ### 1. What is authoritative after a tool call? - [ ] The model's predicted result - [x] The validated tool receipt and resulting event - [ ] The original plan - [ ] The longest reasoning trace **Explanation:** Only observed and validated environment evidence establishes the effect. ### 2. Why is a plan not an execution grant? - [ ] Plans contain no words - [x] Current observations and authority can change before each transition - [ ] Tools ignore plans - [ ] Plans are always wrong **Explanation:** Each action needs authorization against current state, not blanket permission from a speculative plan. ### 3. Which is a valid terminal state? - [ ] Keep trying - [x] Repeated equivalent failure with no new evidence - [ ] The model sounds confident - [ ] Context is long **Explanation:** Cycle detection should stop or escalate a loop that revisits the same failure state. ### 4. What is the correct role of a reflection? - [ ] Immutable ground truth - [x] A provenance-linked model hypothesis that may guide later action - [ ] A transaction receipt - [ ] An authorization token **Explanation:** Reflections are useful interpretations but must remain distinct from observed events. ### 5. What makes a side-effecting action recoverable after timeout? - [ ] A longer prompt - [x] An idempotency key and queryable execution status - [ ] More temperature - [ ] Hiding the receipt **Explanation:** The runtime must reconcile whether the effect occurred before deciding to retry.