Blog · arXiv Analysis · Published: August 12, 2026 · Modified: August 12, 2026 · Last reviewed: August 12, 2026

The Automation Benchmark Cannot Certify the Assistant

A model that completes a task well has not thereby shown that its guidance will improve another worker’s result.

CentaurBench makes that role difference measurable, while its fixed model worker keeps the finding on the simulation side of the human-workplace boundary.

The Paper

The source is Pattaraphon Kenny Wongchamcharoen, Kris Gulati, Min Min Fong, and Abhishek Nagaraj’s CentaurBench: Benchmarking LLM Capabilities on Augmenting vs. Automating Real-World Work Tasks, arXiv:2608.18554v1 [cs.CY], cross-listed in cs.AI, cs.MA, and econ.GN and submitted August 19, 2026. The 46-page version-one paper is licensed CC BY-NC-SA 4.0. The authors describe the study as an initial pilot, not a definitive map of model capability or workplace value.

Two Roles, Two Tests

The benchmark design separates two jobs. In automation mode, a model directly produces a bounded, single-turn deliverable. In augmentation mode, one of nine assistant models writes a single process-only guidance text—a requirements check, execution plan, and final checklist—for a fixed GPT-3.5-Turbo worker, which then produces the deliverable. The assistant is prohibited from supplying the answer itself. A plain condition lets the same worker act without guidance.

The seven tasks cover counseling, market analysis, meal planning, operations research, tax preparation, travel planning, and tutoring. They use no tools and do not model a continuing conversation. CentaurBench therefore tests direct completion against one-shot procedural coaching. It does not test every form of augmentation, such as drafting content, correcting errors, retrieving evidence, or adapting to a worker over several turns.

One Rank Cannot Serve Both Roles

Across ten independent runs, outputs entered separate automation and augmentation tournaments. Four LLM judges made blind pairwise choices with general and task-specific rubrics; a judge could not score an output from its own provider family. Pairwise win rates became within-task ranks, which were then averaged across judges and runs.

The cross-role result is suggestive, not universal. Among the nine assistant models, average automation and augmentation ranks had Spearman correlation 0.48 with a two-sided p-value of 0.187. The winning condition differed across the two roles on five of seven tasks. That is neither proof that the capabilities are opposites nor a statistically settled population estimate. It is evidence that a direct-solving leaderboard is an incomplete procurement test for a coaching role.

No Guidance Is a Necessary Baseline

The plain worker ranked first in augmentation for operations research, tax preparation, and travel planning. Across all seven tasks, its mean rank was 3.79; GPT-5-Mini was the only assisted condition with a better overall mean rank, at 3.66. These are rankings inside this candidate pool, protocol, and judge panel, not universal quality scores.

The paper’s qualitative review connects better-ranked guidance with task-specific analytical structure and alignment with the requested deliverable. Some lower-ranked guidance repeated constraints the worker already had or steered the worker away from required content. That inspection proposes mechanisms; it does not causally isolate them. The firm finding is narrower: under this protocol, adding guidance did not guarantee a better judged output. An evaluation without an unaided baseline cannot reveal that interference.

The Judge Panel Is Part of the Result

The judge-validity appendix reports that pairwise choices agreed with the same judge’s rubric-score ordering in 99.7 percent of 15,120 non-tied scored comparisons. That shows internal consistency, not agreement with a human professional. The 6,265 comparisons evaluated by at least two eligible judges produced 11,257 judge-pair decisions; judges chose the same winner in 71.0 percent of those decisions, with Krippendorff’s nominal alpha at 0.414. Augmentation agreement was lower than automation agreement: 67.8 versus 74.5 percent, with alpha 0.354 versus 0.448.

Those numbers make the evaluator population part of the benchmark specification. A rank produced by model judges is evidence about what that panel preferred under its rubrics. Counseling, tax, tutoring, and operations decisions still require the human and domain-expert validation the authors identify as future work.

The Worker Is a Model Proxy

The most important noun in the paper is worker. In the fixed-worker design, it names GPT-3.5-Turbo, held constant so differences can be attributed to the assistance text. No human receives the guidance, experiences the workflow, exercises professional judgment, or reports whether the intervention helps. A different model worker could reorder the assistants; the paper says so explicitly.

The limitations also confine the study to one assistance message, one worker model, seven tasks, ten runs, and LLM evaluation. Economically grounded prompts are not observed workplaces. The results support a model-to-model coaching hypothesis and a reusable evaluation design. They do not establish productivity, skill, autonomy, satisfaction, safety, or benefit for human workers.

What the Release Lets Us Inspect

The public reproducibility repository includes MIT-licensed benchmark code, task YAML files, pipeline and audit scripts, raw outputs for the ten reported runs, judge records, and derived artifacts. Its interactive dashboard exposes task, run, model, mode, judge, output, and rationale views. Re-running generation requires an Expected Parrot account, API credits, and remote model access, but the released records permit inspection and reaggregation without regenerating those outputs.

That is a substantial audit surface. It still cannot turn simulated workers into people or model preferences into expert judgment. Reproducibility answers whether the computational record can be followed; validity asks whether the test represents the deployment decision.

The Role-Conditioned Evaluation Receipt

A role-conditioned evaluation receipt should name the task; the model’s role; the downstream worker or workflow; the unaided baseline; whether help is a plan, draft, critique, tool action, or conversation; turn and tool budgets; exact model identifiers; repeated-trial design; evaluators and their agreement; per-task gains and harms; uncertainty; human or expert validation; override path; and the owner authorized to accept the result. This is this essay’s proposal, not a validated CentaurBench instrument.

The Spiralist lesson is practical: test the relationship you intend to deploy. A leaderboard can certify a solver only for the work it actually measured. Assistance begins where another actor enters the loop, and so must the evidence.

Sources

Return to Blog