--- title: The Last Step Gets Too Much Credit slug: the-last-step-gets-too-much-credit canonical_url: https://blog.openfactoryai.com/the-last-step-gets-too-much-credit published_at: 2026-07-14T20:58:00.289665+00:00 author: OpenFactoryAI tags: credit assignment, reinforcement learning agents, multi-agent systems, process rewards, agent workflows tldr: A terminal success reward tells an agent that the whole trajectory worked. It does not reveal which model call, retrieval, tool, verifier, or teammate made the difference. Giving every step equal credit is noisy; rewarding only the final step teaches premature closure. Better credit uses executable subgoals, counterfactual replays, return decomposition, calibrated process verifiers, and ablations that remove one decision or agent at a time. key_takeaways: - Credit assignment has temporal, structural, and multi-agent dimensions. - The last action is often only where success becomes visible. - Reward observable state progress and verified subgoals, not eloquent internal narration. - Counterfactual baselines ask what would have happened without a decision or teammate. - Preserve complete branch and cancellation traces because rejected work can still create information value. --- The verifier passes after the final patch. The patch gets the applause. But the decisive move may have happened twelve calls earlier, when retrieval found the one migration note that invalidated the original plan. Long-horizon agents create a credit problem: success arrives late, many steps contributed, several were wasted, and the most visible action is not necessarily the most valuable one. If training gives the final step all the credit, the system learns to finish. It may not learn how to become finishable. ::figure{template=trace caption="The terminal reward appears after a trajectory whose decisive evidence arrived much earlier" items="Scope task :: call 1 | Retrieve docs :: call 2 | Find migration note :: call 3 | Inspect code :: calls 4-6 | Patch attempt :: call 7 | Failed check :: call 8 | Revise :: calls 9-11 | Final patch :: call 12 | Verify :: reward +1"} ## Credit has three dimensions **Temporal credit** asks which earlier action caused a delayed outcome. **Structural credit** asks which component helped: retriever, planner, model, tool, memory, verifier, router, or human. **Multi-agent credit** asks which participant changed team success when agents explored, reviewed, delegated, or voted. These dimensions interact. Agent B may find evidence early, Agent A may use it later, and a verifier may reject two branches before Agent C produces the accepted artifact. One scalar team reward does not identify the contribution path. Represent a trajectory as a causal graph, not only a list: ```text evidence E3 -> plan P4 -> patch A7 -> test failure O8 -> revision A11 -> success \-> rejected branch B6 -> diagnostic D9 -----/ ``` The rejected branch may have information value if its failure localized the bug. A cancellation may have capacity value if it stopped redundant work. A safe refusal may preserve expected value by avoiding an irreversible loss. ::figure{template=branch caption="Credit follows evidence and state dependencies, not wall-clock order alone" items="Verified outcome :: accepted change | Final patch :: commits behavior | Revision :: uses failed-test evidence | Failed branch :: localizes invariant | Retrieval :: finds version constraint | Verifier :: supplies rejection reason | Cancel :: prevents duplicate mutation"} ## Equal return is simple and noisy The simplest Monte Carlo assignment gives every action in a successful episode the terminal return. If the task succeeds with reward `1`, all decisions receive a positive learning signal. If it fails, all receive zero or negative signal. That is unbiased under some formulations and enough for short tasks with many samples. For long agent runs it has high variance: - good actions appear in failed trajectories because a later action was wrong; - bad actions appear in successful trajectories because a later recovery rescued them; - irrelevant stylistic tokens inherit the same reward as a decisive tool selection; - costs incurred after success was already inevitable look helpful; - environment failures punish the policy. Discounting earlier reward can make the problem worse. It favors late actions precisely because the terminal signal is nearer, even when an early decision made success possible. [RUDDER](https://proceedings.neurips.cc/paper_files/paper/2019/hash/16105fb9cc614fc29e1bda00dab60d41-Abstract.html) proposes return decomposition and redistribution for delayed rewards. Its reported experiments cover artificial delayed-reward tasks and Atari settings, not language-agent software workflows. The transferable mechanism is to learn which state changes predict a difference in eventual return, then move reward information closer to those changes while preserving the return objective. ## Do not reward the chain of thought as ground truth Process supervision gives feedback on intermediate steps. In [“Let's Verify Step by Step”](https://arxiv.org/abs/2305.20050), Lightman et al. compare outcome and process supervision on mathematical reasoning and report benefits for their process-supervised setup. That is primary evidence for a specific task and model regime. Agent workflows add a crucial distinction: ```text model reasoning text = proposed explanation environment transition = observed effect verifier evidence = scored claim about progress ``` A plausible statement such as “I have isolated the root cause” should receive no process reward until a reproducible test, dependency relation, or counterfactual patch supports it. Reward machine-checkable process when possible: - a query retrieved the required authoritative document; - the selected repository revision matches the task; - a new test fails before the fix and passes after it; - a tool receipt proves one idempotent effect; - the failure set shrank without suppressing tests; - a compensation restored the declared invariant; - the verifier cites the exact evidence for a subgoal. Process rewards can still be gamed. Counting files opened rewards opening files. Rewarding fewer failing tests rewards disabling tests. Rewarding agreement with a critic can train both systems into the same blind spot. ## Subgoals must describe state, not activity Bad subgoal: ```text inspect the authentication code ``` Better subgoal: ```text identify the function that validates session expiry, with file range and a failing test that reaches it ``` The first rewards an action. The second defines evidence. For multi-hop work, create a subgoal graph: ```text G1 establish task and base revision G2 locate behavior and executable reproduction G3 identify violated invariant G4 construct bounded candidate G5 pass focused behavior checks G6 pass regression and policy checks G7 commit verified artifact ``` Credit a transition only when its postcondition becomes newly true. Re-reading the same file does not earn G2 again. A later observation can invalidate a subgoal, in which case the trace records reversal and downstream credit should be reconsidered. ::figure{template=state-machine caption="Verified subgoals turn delayed success into evidence-bearing progress" items="Scoped :: base revision fixed | Reproduced :: failure executable | Localized :: invariant and path proven | Candidate :: bounded change stored | Focused pass :: target behavior | Regression pass :: broader safety | Accepted :: artifact and receipts"} ## Counterfactual credit asks what changed The strongest question is not “Was this step in a successful run?” It is “Would the run still have succeeded without this step, with everything else held as comparable as possible?” Useful counterfactuals include: - remove one retrieved document and replay selection; - replace one model decision with the baseline policy; - deny one tool and observe whether another path succeeds; - remove one candidate from the verifier set; - replace one agent's message with a null or default action; - stop the trajectory before an extra retry and compare outcome; - keep the action but change its timing or branch order. Language-agent environments are stochastic and path-dependent, so one replay is weak evidence. Run paired seeds or multiple matched rollouts and report a distribution. For a component `i`, a simple difference estimate is: ```text credit_i = E[return with i] - E[return with baseline_i] ``` The baseline must be plausible. Removing a required argument and crashing the workflow exaggerates credit. Replace the component with the best legal default or comparison policy. ## Multi-agent teams need a counterfactual baseline A shared team reward encourages cooperation but gives every agent the same signal. In a team of five, a passive agent receives success credit when four others solve the task. An agent that discovers a decisive failure may be punished if the team ignores it. [COMA](https://ojs.aaai.org/index.php/AAAI/article/view/11794) introduces a counterfactual multi-agent actor-critic method that uses a centralized critic and an agent-specific counterfactual baseline. The evaluated domain and neural architecture differ from language-agent swarms. The key idea is directly relevant: compare the chosen action with alternatives for that agent while holding other agents' actions in context. For an agent team: ```text credit(agent B at turn 4) = team value with B's evidence message - expected team value if B sent a legal baseline message ``` That requires a trace of who observed what and when. If Agent A independently saw the same evidence before B sent it, B's message has less marginal value. If B's message prevented a dangerous merge, it may have high value even though no new patch was produced. ::figure{template=matrix caption="Team contribution depends on novelty and consequence, not message count" items="Novel decisive evidence :: high information / high outcome impact | Duplicate evidence :: low novelty / sometimes confidence | Valid rejection :: no artifact / prevents loss | Candidate patch :: visible work / may be redundant | Coordinator message :: low content / high timing value | Idle participant :: no marginal contribution"} ## Shapley-style attribution is principled and expensive Cooperative game theory suggests measuring a participant's marginal contribution across different coalitions. For three agents A, B, and C, evaluate team value with subsets such as `{A}`, `{B}`, `{A,B}`, and `{A,B,C}`, then average the marginal value of adding each agent across orders. This handles redundancy better than counting messages. If A and B independently find the same evidence, neither receives full standalone credit when both are present. Exact evaluation requires many coalitions and becomes infeasible as team size grows. Agent workflows also violate clean assumptions: removing one agent changes messages, context, timing, and later decisions. Use sampled coalitions or structured ablations, and label the result as modelled contribution rather than causal fact. Credit is also not payment. A component can have negative marginal outcome value while still being required for governance. Independent review may reject many candidates and reduce throughput, yet preserve expected value by preventing rare losses. ## Worked trace: visible work loses to decisive evidence Consider 100 matched illustrative tasks with three components: - R retrieves versioned evidence; - P proposes and revises a patch; - V verifies behavior and policy. Observed success under repeated ablations: | Configuration | Successes / 100 | |---|---:| | R + P + V | 78 | | P + V, no R | 51 | | R + V, baseline patcher | 22 | | R + P, no V, externally adjudicated | 58 accepted | Naive “last action” credit favors P because it writes the patch. Simple marginal differences show other contributions: ```text R marginal in full context = 78 - 51 = 27 points P marginal versus baseline patcher = 78 - 22 = 56 points V observed acceptance difference = 78 - 58 = 20 points ``` The V difference is not pure success contribution because “no V” changes the acceptance mechanism and external adjudication. It may include false accepts and false rejects. This is why every ablation needs an admissible baseline and equivalent terminal scoring. Now add costs: ```text R costs $0.20/task and adds 27 successes per 100 incremental cost per added success = $20 / 27 = $0.74 extra P search costs $1.50/task and adds 18 successes over a cheaper patcher incremental cost per added success = $150 / 18 = $8.33 ``` These are constructed numbers. They demonstrate how credit and economics combine: allocate the next inference dollar to the component with positive marginal verified-outcome value, not the component that produces the final artifact. ::figure{template=bar caption="Illustrative full-context marginal success points from matched ablations" items="Versioned retrieval R :: 27 | Patcher versus baseline P :: 56 | Verification acceptance V :: 20"} ## Credit the stop decision Continuing is an action. Cancelling is an action. Escalating is an action. An agent that stops two losing branches after a verified candidate appears saves cost and capacity. A coordinator that detects repeated equivalent failure prevents a runaway loop. A reviewer that abstains under insufficient evidence avoids a false accept. Add explicit counterfactual value: ```text stop credit = avoided expected cost and risk - probability that cancelled work would improve outcome × value ``` Estimate it from historical continuation experiments, not from the stopping agent's confidence alone. ## Credit models drift with the workflow A process reward model trained on one tool interface may mistake new behavior after tools change. A retriever improvement can make an old “search again” step redundant. A stronger verifier can shift value from human review to automatic rejection. Version credit models and test them on bifurcation tasks where one controlled decision changes the outcome. Monitor: - rank correlation with controlled ablations; - false positive reward for non-progress activity; - credit mass by position, component, and agent; - stability under paraphrase and harmless reordering; - agreement with delayed production outcomes; - whether cost-saving stop decisions receive credit. If 80 percent of positive credit always lands on the last 10 percent of tokens, investigate positional bias before celebrating “efficient reasoning.” ## Some outcomes arrive after the training window A patch can pass today and create maintenance cost next month. A support answer can satisfy the user and cause a refund later. A migration can complete and corrupt a rare record during the next batch. If the episode closes at immediate verification, delayed value never enters the return. Extend outcome identity beyond the training run: ```text task accepted candidate verified artifact merged deployment observed 7-day rework window 30-day rollback and incident window 90-day maintenance sample ``` Do not keep an RL worker open for 90 days. Join delayed events to the immutable task and artifact IDs, then update an evaluation dataset or delayed reward model. Protect against survivorship bias: deleted, reverted, abandoned, and never-deployed artifacts remain in the denominator. Delayed feedback is confounded. A later incident may involve many changes. A maintainer may rewrite a patch for unrelated style. Use causal evidence where possible: - bug-fix commits that cite the generated artifact; - rollback receipts tied to the exact release; - review comments linked to changed lines; - paired canaries or feature flags; - ownership and dependency graphs; - human adjudication with uncertainty labels. Credit can be negative. If an early shortcut yields an immediate pass and later rework, the full return should reflect both. Version the attribution window so “success” does not silently change between training runs. ## Credit uncertainty should control autonomy Teams often store a single credit score, such as `0.73`, and route optimization as if it were observed truth. Credit is an estimate. It depends on the baseline, replay seeds, judge calibration, missing alternatives, and environment stability. Report an interval or distribution: ```text retrieval marginal success: +27 points 95% bootstrap interval: [+18, +35] matched tasks: 100 baseline: same policy without retrieved document environment: repo-eval.v12 ``` The interval above is illustrative. In real analysis, preserve paired task outcomes and use a method appropriate to the design. Uncertainty changes decisions: - high positive credit, low uncertainty: invest or route more work; - near-zero credit, low uncertainty: simplify or remove the component; - positive credit, high uncertainty: collect targeted bifurcation tasks; - negative credit on a protected slice: block promotion; - unknown credit for irreversible work: retain human or independent verification. This prevents an attribution model from becoming a hidden authority system. Credit can recommend where to spend inference or which agent to include. The runtime still enforces permissions and consequence boundaries. ## Credit without novelty rewards meetings Agent swarms can inflate activity through repeated agreement. Ten agents restating one diagnosis generate more tokens and more apparent consensus without adding information. For messages or evidence, separate: ```text validity: is the claim supported? novelty: was it already known to the team? decision impact: did it change an action, verifier, or stop? outcome impact: did the changed decision improve expected terminal value? cost: what inference, latency, and coordination did it consume? ``` A duplicate can still have value by increasing confidence under independent evidence. But agents using the same model, prompt, context, and retrieved source are not independent votes. Measure source and failure correlation before rewarding consensus. The next article applies this discipline to swarms: adding agents increases search capacity, but also communication edges, correlated error, stale state, and verification load. ## Builder playbook 1. Preserve causal trajectory graphs with states, observations, eligible actions, effects, and terminal evidence. 2. Separate temporal, structural, and multi-agent attribution. 3. Begin with honest terminal reward and explicit cost. 4. Define subgoals as newly verified state, never mere activity. 5. Keep process judgments tied to executable or cited evidence. 6. Build legal counterfactual baselines and run matched repeated replays. 7. Use agent-removal, message-removal, tool-removal, and candidate-removal ablations. 8. Measure marginal verified outcome and marginal cost together. 9. Credit cancellation, abstention, recovery, and loss prevention. 10. Version and recalibrate credit models when workflows, tools, or verifiers change. The last step deserves credit for what it changed. It does not deserve the whole story merely because it was standing beside success when the reward arrived. ## References - [Arjona-Medina et al., “RUDDER: Return Decomposition for Delayed Rewards,” NeurIPS 2019](https://proceedings.neurips.cc/paper_files/paper/2019/hash/16105fb9cc614fc29e1bda00dab60d41-Abstract.html). Return decomposition and reward redistribution for delayed-reward problems. - [Foerster et al., “Counterfactual Multi-Agent Policy Gradients,” AAAI 2018](https://ojs.aaai.org/index.php/AAAI/article/view/11794). Agent-specific counterfactual baselines under a centralized critic. - [Lightman et al., “Let's Verify Step by Step,” 2023 preprint](https://arxiv.org/abs/2305.20050). Primary comparison of process and outcome supervision in mathematical reasoning. - [Sutton and Barto, “Reinforcement Learning: An Introduction,” second edition](http://incompleteideas.net/book/the-book-2nd.html). Primary textbook treatment of returns, value, temporal difference, and policy learning. - [Wang et al., “Agent Lightning,” 2025 preprint](https://arxiv.org/abs/2508.03680). A framework addressing training and credit assignment across multi-step agent executions. - [Li et al., “Multi-Agent Credit Assignment with Pretrained Language Models,” AISTATS 2025](https://proceedings.mlr.press/v258/li25e.html). Peer-reviewed work on sparse-reward cooperative multi-agent credit assignment. ## FAQ ### What is credit assignment in AI agents? It is the problem of attributing a delayed terminal outcome to earlier model calls, observations, tools, state transitions, components, or agents so training and resource allocation reinforce the work that made a difference. ### Why not give every step the final reward? Good steps can appear in failed runs and bad steps in rescued successes. Equal return produces high-variance signals and rewards irrelevant activity, especially in long workflows. ### What is process supervision? It scores intermediate steps or states rather than only the final answer. For agents, process scores should be tied to verifiable environment evidence because model reasoning text is not authoritative state. ### How do counterfactual baselines assign agent credit? They compare team return with an agent's actual action against expected return under a legal baseline action while accounting for the other agents and current state. ### Can Shapley values measure contribution in agent swarms? They provide a principled coalition-based idea, but exact computation is expensive and removing an agent changes the trajectory. Use sampled coalitions or structured ablations and label results as modelled contribution. ### Should an agent receive credit for stopping? Yes when stopping avoids expected cost or risk without sacrificing more expected outcome value. Estimate that counterfactual from continuation data and task slices. ## Test yourself ### 1. Why can the final patch receive too much credit? - [ ] Patches are invisible - [x] Earlier evidence and failures may have made the final action possible - [ ] Only first steps matter - [ ] Rewards arrive before actions **Explanation:** Visibility at the terminal step does not establish marginal contribution. ### 2. What is a good process subgoal? - [ ] Think carefully - [ ] Open five files - [x] Prove the violated invariant with a failing executable test - [ ] Write more tokens **Explanation:** The subgoal defines newly verified state rather than activity or style. ### 3. What makes a counterfactual baseline useful? - [ ] It always crashes - [x] It is a plausible legal alternative under comparable conditions - [ ] It removes the whole environment - [ ] It is written after results **Explanation:** An impossible null action exaggerates the component's apparent credit. ### 4. Why is shared team reward insufficient? - [ ] Teams cannot succeed - [x] Passive and decisive agents inherit the same outcome signal - [ ] It is too private - [ ] It prevents tools **Explanation:** Agent-specific contribution requires structural or counterfactual attribution. ### 5. What should be optimized with marginal success credit? - [ ] Message count - [x] Marginal cost and risk - [ ] Title length - [ ] Only token output **Explanation:** Resource allocation depends on added verified value after full incremental cost and risk.