Blog · arXiv Analysis · Published: August 12, 2026 · Modified: August 12, 2026 · Last reviewed: August 12, 2026

The Task Order Becomes the Hidden Curriculum

A memory-based agent does not learn from a bag of tasks. It learns from a sequence, and early accidents can become instructions for everything that follows.

A new re-evaluation shows why an improvement score needs its order, repetitions, and memory history attached.

The Paper

The source is Qinyuan Ye, Yu Li, Yada Pruksachatkun, Jiaxin Zhang, and Chien-Sheng Wu’s On the Fragility of Self-Improving Agents: Variance, Task Order, and Underspecification, arXiv:2608.18066v1 [cs.AI, cross-listed cs.CL and cs.LG], submitted August 18, 2026. The 28-page version-one paper lists Salesforce AI Research as the authors’ affiliation and links public code and trajectory data. It studies inference-time textual memory, not model-weight training or a live user deployment.

A Score With a Sequence

The two evaluated methods write reusable text after each task. Agent Workflow Memory summarizes workflows from successful trajectories; ReasoningBank writes broader items from successful and failed trajectories, then retrieves relevant items for later tasks. The study places them on WebArena’s 812 tasks, VisualWebArena’s 910 tasks, and 267 SCUBA tasks retained after invalid cases were removed. Its reported setup uses GPT-5-mini for the agent and memory constructor, supplies the ground-truth task reward to memory construction, repeats each experiment of interest three times, and compares the default sequence with two shuffled orders.

Three Runs, Divergent Histories

The variance table first shows noise without memory: the best and worst baseline runs differ by 4.44 percentage points on WebArena’s 180-task GitLab subset. Adding a memory method increases the estimated standard deviation in 17 of 24 method-domain comparisons; 11 relative increases exceed 50 percent. ReasoningBank’s best-worst gaps reach 7.78 points on GitLab, 8.26 on Map, and 10.42 on the 48-task Multisite subset.

Those are warning signals, not precise population estimates. Every standard deviation is built from three runs, and a large relative increase can start from a very small baseline. The useful finding is that a single observed score can select one branch of a stateful process and conceal materially different memory histories.

The Hidden Curriculum

The default benchmark order follows increasing task identifiers. In the paper’s order analysis, no-memory pass rates begin around 75 percent and fall below 40 percent after task 150 in two plotted panels; the third has a more irregular pattern. The authors interpret this as an implicit easy-to-hard curriculum. On WebArena, the reported baseline is 54.8 percent, ReasoningBank reaches 56.3 percent in the default order, yet its score is 49.8 percent under Shuffle-1; Agent Workflow Memory scores 49.1 percent there.

Two shuffled sequences do not estimate every possible ordering, and task identifier is not a universal difficulty scale. They do establish a narrower point: task order is part of the treatment whenever earlier outcomes write context for later work. An evaluation that publishes the final average but hides the sequence has omitted a causal input to its own system.

Learning the Wrong Interface

The authors’ manual inspection finds memories recommending APIs even though the browser environment cannot execute API calls. Other memories recommend asking for user confirmation, which the benchmark also cannot provide. In map tasks, a Haversine-distance shortcut occasionally receives reward, enters memory, and is retrieved more often when it appears earlier. Ambiguous prompts and evaluator false negatives provide additional routes from one questionable lesson to later tasks.

This is qualitative analysis of a subset, not a measured taxonomy of all failures. Its governance value is still concrete: a memory item should be treated as an unverified policy proposal, with its originating task, environment constraints, reward, evaluator, and downstream retrievals preserved.

A Partial Repair

The repair experiment gives memory construction post-task rubrics and scores, environment error feedback, and a prompt that excludes unsupported strategies. Combined, these changes move ReasoningBank on Shuffle-1 from 49.8 to 52.7 percent and add 1.1 points on Shuffle-2 while maintaining its default-order result. Both shuffled results remain below the no-memory baseline. Better specification helps; it does not make the memory process order-invariant or prove the three interventions will transfer to another domain.

The Evidence Boundary

The default-order WebArena gain is modest: the paper reports ReasoningBank at plus 1.5 points over baseline with an unpaired three-run t-test p-value of 0.23. The study covers two textual-memory methods, one web-task family, one reported backbone configuration, three runs, and two shuffle seeds. Its limitations say the manual memory review was non-exhaustive and that newer memory methods and other domains were not tested. No result here supports a claim about all continual-learning agents or deployed harm rates.

I inspected the Apache-2.0 repository at commit f79fbb14 and the public trajectory dataset at revision 4367a214. The repository contains the repeated-run and ordinal/shuffle configurations plus reproduction entry points; the dataset exposes trajectories, results, logs, and memory files. I did not rerun the full cloud-, credential-, and model-dependent experiment.

The Order-Sensitivity Report

For consequential evaluation, an order-sensitivity report should bind every score to the ordered task identifiers, permutation method and seed, initial memory state, model and harness revision, environment reset, reward source, rubric visibility, memory-write rule, retrieval rule, per-run outcomes, dispersion measure, invalid-task exclusions, evaluator corrections, and every memory item quarantined or removed. It should report baseline capability, default-order change, and shuffled-order change separately.

The Spiralist lesson is that a stream is also a teacher. Once past results can rewrite future context, history is part of the mechanism. Calling the outcome self-improvement without publishing that history turns a schedule into an invisible coauthor.

Sources

Return to Blog