Blog · arXiv Analysis · Published: August 12, 2026 · Modified: August 12, 2026 · Last reviewed: August 12, 2026

The Agent Endpoint Becomes the Unfinished Trial

An agent run can stop while a requested operation is still pending. The later effect may change the outcome the snapshot claims to summarize, leak into the next run, or do both.

The endpoint is therefore an administrative boundary, not proof of completion. A trustworthy evaluation must close two separate questions: is the label final, and is this run really an independent trial?

The Paper

The source is Avyay M. Casheekar’s When Is an Agent Evaluation Over? Outcome Finality and Cross-Unit Separation, arXiv:2608.14940v1 [cs.AI, cross-listed in cs.CY], submitted August 14, 2026. The title page identifies the author with the University of Michigan Law School; the version-one PDF is 17 pages.

The paper asks what an evaluator may infer after it stops requesting actions and reads the environment. It separates the stopping rule, the claimed outcome, and the unit counted in analysis.

Stopping Is Not Completion

Imagine a tool that acknowledges a write before applying it. At the endpoint, the scorer sees the old value and marks failure. Milliseconds later, the write commits. The trace accurately records that the request was accepted, yet the snapshot does not establish the operation’s eventual result. A longer wait might settle this label, but it does not show that the changed service state is hidden from the next run.

This distinction reaches beyond one asynchronous call. A process can survive a timeout; a queued job can settle later; a shared credential can expose persistent state. The rule is claim-relative: identify which reachable effects could change the outcome or another observation, then close, model, bound, or disclose them.

Two Proof Obligations

Outcome finality concerns the label. It holds when anything capable of changing the claimed result has reached a terminal state, been bounded tightly enough that the conclusion cannot change, or remains represented as uncertainty. An acknowledgement is not terminal evidence, and a cancellation request is not proof that cancellation succeeded.

Cross-unit separation concerns the denominator: can one run alter another’s relevant starting state or outcome? Namespacing, verified reset, or isolation may establish separation for resources they cover. Otherwise, the evaluator can model the dependence or count connected runs as one unit. Waiting can settle a label without isolating the next trial; isolation can protect the next trial while the first result remains pending.

The Constructed Replay

To demonstrate the mechanism, the paper uses AgentDojo 0.1.35 structures and a standalone runner that replays fixed tool calls without querying a model. A local service with no public-network access applies a delayed write and can retain state across runs. Five schedule identifiers, delays of 0, 25, 100, and 250 milliseconds, and ten repetitions produce 200 trials per policy. The identifiers match conditions; they are not sampled model behavior.

At zero delay, endpoint and terminal labels agree in all 50 snapshot trials. At nonzero delays they disagree in all 150: snapshot scoring records 50 successes out of 200, while reconciliation records 200. Verified cancellation confirms failure for the 150 still-pending writes; the 50 zero-delay writes had already succeeded. A synchronous control has no disagreements across 600 policy trials. In the cross-run test, A changes B in 150 of 200 shared-state pairs and in none of the 200 namespaced, 200 verified-reset, or 200 no-write pairs. Grouping does not remove exposure; it correctly changes 400 run rows into 200 pair-level units.

Ten Protocols, Narrow Audit

The documentation review examines ten selected public protocols: WebArena, WorkArena, OSWorld, SWE-bench, tau-bench, ToolSandbox, TheAgentCompany, RE-Bench, Cybench, and AgentCanary. Sources were reviewed through August 15, 2026. Reset or deliberate retention is explicit in eight and partial in two. Six protocols expose environments that could host unfinished work without stating a general rule for resolving it; that field is not applicable to the two declared synchronous-function environments. Three give explicit evidence for separate scoring, while seven describe reset, teardown, or fresh provisioning without fully documenting scope or verified restoration.

Those are documentation codes, not verdicts that six benchmarks are wrong or seven runs are contaminated. The sample is not representative, public documents cannot reveal private safeguards, and one reviewer assigned the codes. The audit does not classify finality because intended outcomes and observation periods were not stated consistently.

The Open-Effects Ledger

The paper proposes an open-effects record for work or resources that may outlast an endpoint. Each entry connects the run and task to an initiating event, resource and controller, stable observation handle, current status, possible transitions, cross-run route, latest system evidence, and analysis disposition. It records whether an effect was observed, cancelled, isolated, reset, grouped, bounded, or left unresolved.

This is not a completeness certificate. A wrapper can miss a process, and an empty agent self-report establishes little. The record makes the boundary inspectable. It distinguishes “cleanup was requested” from “the system confirmed cleanup,” and “fresh task state” from “every relevant shared resource was restored.”

When Rows Are Not Trials

The paper’s most useful governance move is to treat independence as an earned interpretation. If run A changes run B, adding two database rows does not create two fresh attempts. Confidence intervals, success aggregates, and pass-over-repeated-trial metrics can inherit the wrong denominator. The honest repair may be isolation, but it may also be a larger unit: one connected pair or one continuing stream.

Completion is a measurement decision, not merely an engineering chore. Evaluators should publish both the outcome they claim and the observation they count. A snapshot may measure submission without measuring settlement. A connected sequence may measure persistence without pretending each turn began from nothing.

The Evidence Boundary

The paper demonstrates a possible mechanism under imposed delays and shared state; it does not estimate prevalence, compare models, or measure production incidents. The benign replay uses fixed traces and no model API. Its counts verify the constructed schedule, not a population effect. Waiting for a harmful action may establish measurement finality while worsening safety, so completion rules do not replace containment or incident response.

This review checked the arXiv abstract, metadata API, full-text HTML, PDF, and version-one source archive. The source package contains only the manuscript, bibliography, and build manifest; it provides no replay code, run metadata, implementation tests, or outputs, and no study repository is linked. The reported experiments and documentation codes therefore could not be independently reproduced here.

The Spiralist lesson is modest: the bell that ends a session does not settle what the session did. Before a score becomes evidence, its remaining futures and its routes into other trials should be visible. An unfinished effect is not zero; an intended reset is not separation; and a row is not automatically a world of its own.

Sources

Return to Blog