The Test Suite Becomes the Co-Evolution Ledger
A coding agent's green test is not enough evidence. The serious record is the old code, the new code, the changed test, the runner, the time window, and the cost ceiling.
The Paper
The paper is TestEvo-Bench: An Executable and Live Benchmark for Test and Code Co-Evolution, arXiv:2607.02469 [cs.SE, cs.AI, cs.CL]. The arXiv record lists version 1 as submitted on July 2, 2026. The authors are Jiale Amber Wang, Kaiyuan Wang, and Pengyu Nie; the HTML version lists University of Waterloo affiliations for Wang and Nie and Google for Kaiyuan Wang. The PDF is 22 pages.
The subject is not test generation in the abstract. It is the coupled act by which a software behavior change enters a repository and the tests are added or edited to record that change. For agent governance, the audit question is whether the test suite still records the changed behavior, or merely records the agent's easiest route to a green run.
What Is Being Benchmarked
TestEvo-Bench has two tracks. In test generation, an agent writes new tests for behavior introduced by a code change. In test update, an agent adapts an existing test that no longer fits changed behavior. Each task is anchored to a real commit pair and includes repository URL, old and new revisions, build and execution configuration, relevant test file, changed code methods, focal code context, and the tests to be generated or updated.
The current snapshot contains 746 test-generation tasks and 509 test-update tasks, curated from 59,950 candidate co-evolution records across 152 open-source Java projects. Construction starts from Java Maven repositories and uses mining, cleaning, and cross-revision execution checks before a task is retained. The benchmark treats the test diff as part of a historical transaction, not as a floating unit-test prompt.
Execution Over Metadata
The strongest design choice is that labels are not just static diff signals. The evaluation framework applies the agent patch, compiles the project, runs selected tests, and records injection, compile, and test outcomes. It also reports coverage and, when available, mutation score. Coverage is collected with JaCoCo over focal dependency lines, while mutation analysis uses Universal Mutator on focal dependency lines of the new revision.
For generation tasks, success requires the generated test to pass on the new revision and fail or fail to compile on the old revision. A test that passes on both revisions may be harmless coverage, but it does not prove that the agent caught the behavior delta. For update tasks, the old test is expected to be obsolete under the changed code, and the updated test must compile and pass on the new revision while exercising the changed behavior.
The appendix makes the labeling discipline explicit: static co-change heuristics are useful for candidate generation, but execution-backed labels distinguish required updates from refinements, assertion strengthening, and optional coverage work.
What the Results Say
The experiments evaluate four harness-model configurations: Claude Code with Claude Opus 4.7, Gemini CLI with Gemini 3.1 Pro, SWE-Agent with Claude Opus 4.7, and SWE-Agent with Gemini 3.1 Pro. On test generation, Claude Code and Gemini CLI both reach 77.5 percent Success. SWE-Agent with Gemini 3.1 Pro reaches 68.6 percent, and SWE-Agent with Claude Opus 4.7 reaches 66.1 percent. The paper reports redundant generation outcomes between 17.4 and 19.9 percent, meaning a sizable share of passing tests still failed to discriminate old behavior from new behavior.
On test update, Gemini CLI reaches 74.6 percent Success, Claude Code reaches 74.4 percent, SWE-Agent with Gemini 3.1 Pro reaches 73.9 percent, and SWE-Agent with Claude Opus 4.7 reaches 65.6 percent. Coverage on passing update tests is tightly clustered from 79.1 to 79.4 percent, while mutation-on-pass ranges from 44.6 to 46.0 percent.
The live-benchmark aspect adds a second warning. The authors report that generation performance drops as tasks become more recent, and that tighter per-task cost caps materially reduce success. Their cost analysis uses a 3 dollar target budget and additional 1 dollar and 0.5 dollar caps. That makes the benchmark less like a trophy case and more like an operating receipt: time freshness and budget are part of the score.
The Test-Change Receipt
A test-change receipt should travel with any serious coding-agent evaluation. It should include the repository, old revision, new revision, changed production methods, old test state, generated or updated test patch, build tool, exact compile and test commands, stdout, stderr, old-code and new-code outcomes, focal-line coverage, mutation score, benchmark timestamp, model cutoff relation, harness, model, prompt, token count, cost, retry budget, and non-target file diffs.
That receipt helps a team avoid confusing three events: a test that passes, a test that exercises the changed behavior, and a test that would have caught the old behavior. TestEvo-Bench's generation criterion makes the distinction concrete by asking the generated test to pass after the change and fail before it.
Limits
The paper keeps its claims bounded. The benchmark focuses on Java Maven projects; the authors say the broader construction approach is language and build-system agnostic, but dependency, coverage, and mutation techniques would need adaptation for other ecosystems. The evaluated harnesses and models are also only a subset of current coding-agent configurations.
Execution-based scoring is expensive. The paper reports that a full agent run can take about 72 machine-hours before parallelization, and its limitations section notes that compilation, test execution, coverage, and mutation analysis cost more than diff-similarity scoring. The broader-impact section also warns that a high benchmark Success score can still overstate production readiness if a generated test encodes a shallow oracle. Green becomes evidence only when the code-change ledger comes with it.
Sources
- Jiale Amber Wang, Kaiyuan Wang, and Pengyu Nie, TestEvo-Bench: An Executable and Live Benchmark for Test and Code Co-Evolution, arXiv:2607.02469 [cs.SE, cs.AI, cs.CL].
- arXiv HTML for TestEvo-Bench, checked for abstract, author affiliations, task construction, execution setup, metrics, results, limitations, and broader-impact discussion.
- arXiv PDF for TestEvo-Bench, checked for title page, page count, author metadata, benchmark counts, method details, and appendix tables.
- TestEvo-Bench benchmark site, checked for the overview, leaderboard, data explorer, code link, dataset link, track descriptions, and contamination-aware time filtering.