--- title: After the Demo slug: after-the-demo canonical_url: https://blog.openfactoryai.com/after-the-demo published_at: 2026-07-14T20:54:00.289665+00:00 author: OpenFactoryAI tags: AI-native development, durable execution, agent workflows, idempotency, software agents tldr: An agent demo can keep its plan in a prompt and restart when something breaks. An AI-native production system cannot. It needs durable state, append-only events, idempotent effects, checkpoints, compensation, versioned workflows, and receipts that prove what actually happened. The decisive design question is not whether an agent can write code. It is who owns state when the agent, worker, network, or human disappears halfway through the job. key_takeaways: - AI-native development designs for long-running autonomous work, not chat-shaped code generation. - Durable state lives outside the model context and can be reconstructed from verified events. - A timeout is an unknown outcome, not proof that an action failed. - Idempotency prevents duplicate effects; compensation handles effects that cannot be undone atomically. - Workflow versions, checkpoints, receipts, and replay tests are production features. --- The demo ends when the process dies. The factory begins when the work continues correctly. A coding agent can inspect a repository, edit a file, run a test, and announce success in one uninterrupted session. That is useful. It is not yet AI-native development. Real work lasts longer than one model context and crosses systems that fail independently: model gateways, sandboxes, Git hosts, package registries, CI queues, databases, humans, and deployment control planes. The hard question is not “Can the model finish this task?” It is: > Who owns the truth when the agent disappears halfway through it? The answer must be the runtime, not the transcript. ::figure{template=state-machine caption="An AI-native task survives process boundaries because verified state is durable" items="Accepted :: goal, identity, base revision | Running :: leased worker and budget | Waiting :: CI, tool, or approval | Reconciling :: timeout outcome unknown | Compensating :: reverse completed effects | Verified :: postcondition and receipts | Failed :: terminal reason preserved"} ## AI-native does not mean AI-shaped UI “AI-native” is often used to mean a product with a chat box, generated code, or a model call in the critical path. Those are interface and capability choices. They say little about the operating system underneath. AI-native development starts from a different assumption: inference is cheap enough and capable enough that software work can be attempted, observed, revised, verified, and resumed by machines over long periods. The unit of execution becomes a durable task, not a synchronous request. That changes the architecture. A request-shaped system assumes: ```text request arrives -> computation runs -> response returns ``` A durable task assumes: ```text task accepted -> many model and tool steps -> waits that may last seconds or days -> worker replacement -> retries and duplicate messages -> version changes -> one terminal, evidenced outcome ``` [AutoDev](https://arxiv.org/abs/2403.08299) and [SWE-agent](https://arxiv.org/abs/2405.15793) show why the environment and agent-computer interface matter for automated software work. Agents need repository navigation, editing, commands, tests, logs, and constrained execution. Their reported benchmark results are evidence about specific systems and tasks, not proof of production durability. A benchmark episode normally does not model a worker dying after a package was published but before the event was recorded. That missing interval is where production systems are made. ## Context is working memory, not the record An agent context may contain a plan, recent observations, summaries, and tool results. It is assembled to improve the next decision. It is not a durable source of truth. Contexts are truncated. Summaries omit details. Calls can time out. Models can restate hypotheses as facts. A process-local transcript disappears with the process. Even a stored transcript mixes observations, instructions, model proposals, and rejected actions in one untyped stream. Keep three planes separate: 1. **Event log:** immutable facts such as task accepted, lease granted, command started, command completed, patch stored, approval received, or deployment receipt observed. 2. **Derived task state:** a typed reduction of those events, such as `waiting_for_ci`, `base_revision`, `open_failures`, and `calls_left`. 3. **Model context:** a bounded projection of state and evidence for the next inference step. The event log rebuilds state. State assembles context. Context never silently rewrites the event log. ::figure{template=layers caption="The model sees a projection; the runtime retains the evidence" items="Model context :: bounded projection for the next decision | Typed task state :: current phase, authority, budget, revisions | Append-only events :: accepted facts and transition receipts | Artifact store :: patches, logs, test reports, evidence | External systems :: repository, CI, registry, deployment truth"} [Chandy and Lamport's distributed snapshot paper](https://www.microsoft.com/en-us/research/publication/distributed-snapshots-determining-global-states-distributed-system/) addresses how a distributed system can record a consistent global state while computation continues. An agent workflow is not an implementation of that algorithm by default, but the underlying warning transfers: there may be no single instant at which one process can simply “read the whole truth.” The repository, task database, CI run, tool queue, and worker can each reflect a different point in the causal history. This is why events need stable identifiers, causal links, and external receipt IDs. “Test passed” is weaker than “CI run 481 for commit `7ac1` completed with conclusion success at time T.” ## Every action has four moments For each side effect, distinguish: ```text intent recorded request dispatched effect committed externally receipt recorded locally ``` Failures between these moments are not equivalent. If the worker dies before dispatch, retrying is normally safe. If it dies after the external effect but before recording the receipt, local state says “unknown.” Blind retry can duplicate the effect. Marking it failed can lose a successful outcome. Consider an agent that opens a pull request: ```text 09:00:00 intent pr.opened_requested recorded with key task_42:pr:1 09:00:01 API request sent 09:00:02 Git host creates PR #812 09:00:02.5 worker loses network 09:00:30 lease expires; replacement worker starts ``` The replacement must reconcile before creating anything. It queries the Git host using the stable branch, head commit, task marker, or idempotency key. If PR #812 exists with the expected head, it records the missing receipt and advances. If not, it can dispatch safely under the same key. The correct state after timeout is `outcome_unknown`, not `failed`. ::figure{template=timeline caption="A timeout splits dispatch from knowledge of the external outcome" items="Intent :: durable key stored | Dispatch :: request leaves worker | Commit :: external system changes | Disconnect :: receipt is lost | Reconcile :: query by stable identity | Advance :: record existing receipt or retry"} ## Idempotency is a semantic property An operation is idempotent when repeating it has the same intended effect as applying it once. HTTP method names do not grant this property. The business operation, key scope, parameters, storage, and response behavior do. For an agent tool, define: ```text idempotency_key = tenant + task + logical_transition + attempt_generation request_fingerprint = hash(canonical_arguments) ``` On first use, the executor reserves the key and stores the fingerprint. A duplicate with the same fingerprint returns the original status or receipt. A duplicate key with different arguments is rejected. The key needs a retention period longer than any plausible retry or replay window. Reads are not automatically harmless. A command such as “download latest dependencies” can observe a different world on retry. Pin versions and record content hashes when reproducibility matters. Writes fall into four classes: | Effect class | Example | Recovery rule | |---|---|---| | Naturally idempotent | Set issue label to `verified` | Repeat after confirming the target identity | | Keyed creation | Create one build for task and revision | Return existing build for the same key | | Append or increment | Add usage charge | Deduplicate at the ledger boundary | | Irreversible or external | Send announcement, rotate credential | Require stronger preconditions, approval, and reconciliation | “Run this shell command again” is not a retry policy. It is an admission that the transition semantics were never designed. ## Exactly once is an outcome, not a delivery guarantee Queues commonly deliver at least once because losing work is worse than occasionally delivering a message again. Networks can lose acknowledgements. Workers can crash after effects. A system therefore reaches an effectively-once business outcome by combining duplicate delivery with deduplicated effects and durable receipts. That distinction matters for software factories: - the task message may be delivered twice; - the model may be called twice; - a test may be run twice; - only one branch, pull request, release, charge, or deployment should become authoritative. Do not try to make every computation singular. Make authoritative effects singular. This also changes cost accounting. Duplicate inference is waste, but duplicate mutation is risk. Track them separately: ```text replay compute waste = repeated model + tool compute cost duplicate-effect exposure = count of non-deduplicated mutation attempts reconciliation latency = time spent proving an unknown outcome ``` ## Long work needs checkpoints, but checkpoints can lie A checkpoint stores enough durable progress to resume without replaying every expensive step. It might include: - workflow version and current phase; - base and current repository revisions; - accepted plan version; - completed transition IDs and receipts; - artifact hashes and storage locations; - remaining budgets and deadlines; - pending external operations; - evidence cursors needed for context reconstruction. Checkpoint only after a transition boundary whose postcondition has been verified. Saving “deployment probably succeeded” turns uncertainty into false state. The snapshot must also identify the code that knows how to interpret it. A workflow changed after deployment may reorder steps, rename states, or alter retry behavior. Resuming an old task under new semantics can repeat a mutation or skip a new safety check. Use one of three explicit policies: 1. **Pinned execution:** existing tasks continue on the workflow version that created them. 2. **Compatible replay:** new code must preserve the deterministic decisions of old histories. 3. **Migration:** a reviewed state transformer moves selected tasks to a new version and records the migration event. Version the prompt, tool contract, reducer, policy, verifier, and workflow. “Model version” alone is not enough to reproduce behavior. ::figure{template=flow caption="Recovery rebuilds truth before it asks a model what to do next" items="Load history :: verified events only | Pin version :: workflow, tools, policy, model | Reduce state :: deterministic transition logic | Reconcile unknowns :: query external receipts | Validate artifacts :: hashes and base revisions | Assemble context :: bounded current evidence | Resume :: acquire lease and choose next legal step"} ## Compensation is not rollback A database transaction can make a group of local changes atomic. A multi-system workflow usually cannot lock Git, CI, a cloud provider, a registry, and a human inbox in one transaction. The 1987 [Sagas paper by Garcia-Molina and Salem](https://doi.org/10.1145/38713.38742) describes long-lived transactions as sequences of subtransactions with compensating transactions for partial execution. The useful lesson for agents is precise: after several committed effects, recovery may require new forward actions that semantically counter earlier actions. Deleting a created preview environment may compensate for creating it. Closing a pull request may compensate for opening it. Neither erases logs, notifications, costs, or the fact that observers saw it. Model the workflow as: ```text T1 create branch C1 delete branch if unmerged T2 open pull request C2 close pull request T3 create preview C3 destroy preview T4 publish release C4 no automatic compensation ``` The final step has no honest automatic inverse. That boundary should require stronger verification and possibly human approval before execution. Compensations must themselves be idempotent, authorized, observable, and retryable. They can fail. A task is not safely closed while required compensation remains unresolved. ::figure{template=branch caption="A saga commits forward steps and compensates in reverse until an irreversible boundary" items="Task :: prepare and release | Branch :: create branch / delete branch | Pull request :: open / close | Preview :: provision / destroy | Verify :: tests and policy | Release :: irreversible boundary | Failure :: compensate completed steps in reverse"} ## Worked example: the 4.1 percent duplicate that can become zero Suppose a factory executes 10,000 repository tasks per month. These values are illustrative, not customer telemetry: - 7 external side-effecting transitions per task; - 70,000 mutation requests in total; - 4.1 percent experience a timeout or duplicate delivery; - 60 percent of those ambiguous requests actually committed before acknowledgement was lost; - an unkeyed retry would therefore expose `70,000 × 0.041 × 0.60 = 1,722` already-committed effects to duplication. Assume keyed executors reconcile 99.7 percent of those cases and route the remaining 0.3 percent to review: ```text ambiguous requests = 70,000 × 4.1% = 2,870 already committed = 2,870 × 60% = 1,722 automatic reconciliation = 2,870 × 99.7% = 2,861.39 human exceptions = 2,870 × 0.3% = 8.61, rounded to 9 ``` The mechanism does not make the network exactly once. It moves duplicate-effect exposure from as many as 1,722 blind retries to roughly nine bounded exceptions, assuming stable keys, queryable external identity, correct fingerprints, and sufficient retention. The ROI is not token savings. It is avoided duplicate mutation, lower exception labor, and a smaller blast radius. Useful operating metrics are: | Metric | Why it matters | |---|---| | Tasks resumed after worker loss | Proves durability is exercised, not theoretical | | Unknown outcomes reconciled automatically | Measures external observability and key quality | | Duplicate deliveries / duplicate effects | Separates expected queue behavior from control failure | | Mean recovery point | Shows how much verified work must be replayed | | Compensation success and age | Exposes stranded external resources | | Replay divergence | Detects workflow versions that cannot reproduce history | | Cost per terminal verified task | Connects reliability overhead to business value | ## The counterargument: keep the agent simple Durable execution has real cost. Event schemas, idempotency storage, workflow versions, leases, compensations, and replay tests can outweigh their value for a two-second read-only task. [Agentless](https://arxiv.org/abs/2407.01489) is an important counterweight to reflexively complex agent architectures. Its authors report that a simpler localization, repair, and validation process performed strongly on their evaluated SWE-bench Lite setup. The exact benchmark result is scoped, but the design lesson is broader: autonomy and orchestration complexity need evidence. Use the smallest reliability class the effect requires: - **Ephemeral call:** short, read-only, cheap to repeat, no external state. - **Restartable job:** deterministic inputs and outputs, safe whole-job retry. - **Durable workflow:** long waits, multiple systems, valuable partial progress. - **Governed saga:** irreversible effects, money, credentials, production, or humans. Do not build a saga for autocomplete. Do not run a production migration like autocomplete. ## Builder playbook Start with the failure boundary, not the framework. 1. List every external read and mutation. Mark which can change between attempts. 2. Give each logical transition a stable ID, argument fingerprint, timeout class, and postcondition. 3. Record intent before dispatch and a validated receipt after commitment. 4. Treat timeout as unknown. Implement reconciliation before retry. 5. Store immutable events and derive typed state with deterministic reducers. 6. Checkpoint only at verified transition boundaries. 7. Pin or migrate workflow, prompt, tool, policy, model, and verifier versions. 8. Define compensation for every reversible committed effect and an approval boundary for effects without an honest inverse. 9. Kill workers at every point between intent, dispatch, external commit, and receipt. Replay the history and compare the terminal state. 10. Measure terminal verified outcomes, duplicate-effect exposure, exception labor, and recovery time. The most revealing test is deliberately cruel: ```text for every transition boundary: terminate the worker duplicate the last message delay the external acknowledgement change the lease owner resume from durable history assert one authoritative outcome ``` If the system cannot pass that test, the agent owns the task only while everything is healthy. ## After the demo, state is the product Models will improve. Tool use will become cheaper. More software work will cross the line from assisted to delegated and from delegated to autonomously managed. That shift increases the value of the parts the model does not own: identity, authority, event history, artifact integrity, reconciliation, verification, and termination. AI-native development is not code written by AI. It is work that remains correct when AI is only one unreliable participant in a durable system. The demo proves the agent can move. The runtime proves the work survives. ## References - [Chandy and Lamport, “Distributed Snapshots: Determining Global States of Distributed Systems,” ACM TOCS, 1985](https://www.microsoft.com/en-us/research/publication/distributed-snapshots-determining-global-states-distributed-system/). The classic consistent-state problem for active distributed computation. - [Garcia-Molina and Salem, “Sagas,” ACM SIGMOD, 1987](https://doi.org/10.1145/38713.38742). The primary paper on sequencing long-lived subtransactions with compensation. - [Tufano et al., “AutoDev: Automated AI-Driven Development,” 2024 preprint](https://arxiv.org/abs/2403.08299). An automated development framework with agents operating through constrained tools in Docker environments. - [Yang et al., “SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering,” 2024 preprint](https://arxiv.org/abs/2405.15793). Evidence that the interface between an agent and repository environment materially affects evaluated performance. - [Xia et al., “Agentless: Demystifying LLM-based Software Engineering Agents,” 2024 preprint](https://arxiv.org/abs/2407.01489). A useful simpler baseline and counterargument to unnecessary agent orchestration. - [Lamport, “Time, Clocks, and the Ordering of Events in a Distributed System,” Communications of the ACM, 1978](https://lamport.azurewebsites.net/pubs/time-clocks.pdf). The foundational account of causal ordering without a perfect global clock. ## FAQ ### What is AI-native development? It is the design of software work around model-driven, long-running, machine-operated tasks rather than adding AI to a request or user interface. Production AI-native systems make state, authority, effects, verification, recovery, and termination durable outside the model. ### Why should an agent not use its transcript as workflow state? A transcript mixes facts, proposals, instructions, and rejected hypotheses. It can be truncated or lost. Store immutable verified events, derive typed state, and assemble only the relevant projection into model context. ### What should happen when an AI tool call times out? Mark the outcome unknown and reconcile against the external system using a stable operation identity. Retry only after proving the effect did not commit, or use an idempotency key that returns the original result. ### What is idempotency in an agent workflow? It means repeating one logical transition produces one intended authoritative effect. The executor binds a stable key to canonical arguments and returns the original status or receipt for valid duplicates. ### What is compensation in a durable AI workflow? Compensation is a new forward action that semantically counters an earlier committed effect, such as destroying a preview environment. It is not time travel and may not erase notifications, cost, logs, or observation. ### When is durable execution unnecessary? A short read-only call with deterministic inputs, no valuable partial progress, and safe whole-call retry may remain ephemeral. Durability becomes valuable with long waits, multiple systems, side effects, expensive progress, or irreversible consequences. ## Test yourself ### 1. What is the correct state after a side-effecting request times out without a receipt? - [ ] Definitely failed - [ ] Definitely succeeded - [x] Outcome unknown, reconcile before retry - [ ] Delete the task **Explanation:** The external system may have committed the effect before the acknowledgement was lost. ### 2. Which artifact should reconstruct current workflow state? - [ ] The longest model reflection - [x] A deterministic reduction of verified events - [ ] The newest prompt - [ ] A screenshot **Explanation:** Events preserve accepted facts; a typed reducer derives the current state from them. ### 3. What makes an idempotency key safe? - [ ] It is random - [x] It is bound to one logical transition and canonical argument fingerprint - [ ] It expires immediately - [ ] The model remembers it **Explanation:** A duplicate key with different arguments must be rejected, while the same operation returns its original status. ### 4. Why is compensation not rollback? - [ ] It uses no tools - [x] It is a new action and cannot erase every prior observation or cost - [ ] It always fails - [ ] It only applies to reads **Explanation:** Distributed effects are already visible and committed; compensation semantically counters them through another action. ### 5. What is the strongest durability test? - [ ] Ask the model if it is reliable - [x] Kill workers and duplicate messages at every transition boundary, then assert one outcome - [ ] Increase context length - [ ] Disable timeouts **Explanation:** Fault injection tests whether durable history, reconciliation, and deduplication actually recover the task.