Blog · arXiv Analysis · Last reviewed July 11, 2026

The Benchmark Becomes the Judge Agent

AgentBeats is useful because it treats agent evaluation as an interface standardization problem, not only as a leaderboard problem.

The benchmark becomes a judge agent. Subject agents receive tasks over A2A, use tools over MCP, and can be evaluated without every benchmark rewriting a harness for every agent.

The Paper

The paper is AgentBeats: Agentifying Agent Assessment for Openness, Standardization, and Reproducibility, arXiv:2606.13608 [cs.AI, cs.LG]. arXiv lists version 1 as submitted on June 11, 2026 and version 2 as revised on June 14, 2026, with DOI 10.48550/arXiv.2606.13608. The arXiv page links a CC BY-NC-ND 4.0 license.

The authors include Xiaoyuan Liu, Jianhong Tu, Yuqi Chen, Siyuan Xie, Sihan Ren, Tianneng Shi, Gal Gantar, Evan Sandoval, Donghyun Lee, Daniel Miao, Peter J. Gilbert, Nick Hynes, Mauro Staver, Warren He, David Marn, Andrew Low, Xi Zhang, Elron Bandel, Michal Shmueli-Scheuer, Siva Reddy, Alexandre Drouin, Alexandre Lacoste, Ramayya Krishnan, Elham Tabassi, Yu Su, Victor Barres, Chenguang Wang, Wenbo Guo, and Dawn Song.

AAA

The paper's central proposal is Agentified Agent Assessment, or AAA. Instead of forcing every benchmark and every agent to integrate with one another through bespoke glue code, AAA turns the benchmark into a judge agent and asks all participants to communicate through standardized protocols.

The two named protocols are A2A for task management and MCP for tool access. The judge agent owns the evaluation logic, dataset, workflow, environment provisioning, and scoring. The subject agent receives a self-contained task and environment access through the protocol boundary.

The claimed integration shift is from N times M bespoke benchmark-agent integrations to N plus M protocol-level integrations. That is the right target. Agent systems differ by harness, memory, tool execution, browser control, terminal access, and deployment model. Evaluation should not require every benchmark author to understand every agent's internals.

Five Modes

AgentBeats is the concrete system realization. It defines five operation modes: Local, Remote, Hosted, Proxy, and CI. The modes are less interesting as product taxonomy than as governance handles.

Local mode supports one developer running all agents on one host. Remote mode supports public assessments of private agents registered as live A2A services. Hosted mode lets a platform instantiate submitted repositories or container images. Proxy mode connects local development to remote participants through a platform tunnel. CI mode runs assessments through public continuous-integration infrastructure such as GitHub Actions.

Those modes encode tradeoffs between openness, privacy, and reproducibility. A public hosted run is inspectable but may require platform trust. A remote private agent protects implementation details but can make reproducibility weaker. A CI run improves auditability but constrains infrastructure.

Field Study

The field study comes from the AgentX-AgentBeats competition, run with the Agentic AI MOOC community. The paper reports roughly 40,000 registered learners, 298 judge agents across 12 categories, and 467 subject agents from independent participants after five months of a six-month competition.

Many existing benchmarks were agentified, including Tau2-Bench, MedAgentBench, FinanceAgent, OfficeQA, PersonaGym, and OSWorld among others. The paper reports that Tau2-Bench received 347 assessments from 42 unique subject agents, with 16 developers submitting 10 or more agent versions.

The code-submission analysis is also useful. The paper reports average Python code size of 5.3k lines for judge agents and 3.8k lines for subject agents, with prompts detected in 78.3 percent of judge-agent repositories and 87.1 percent of subject-agent repositories. Python dominated, but the protocol boundary allowed TypeScript and Rust implementations too.

Coding Agents

The coding-agent case study evaluates four model-harness pairs on three agentified benchmarks: DevEval, SWE-Bench Pro, and Terminal-Bench 2.0. The pairs are Claude Opus 4.7 with Claude Code, GPT-5.4 with Codex CLI, Gemini 3.1 Pro with OpenCode, and Qwen3.5-397B-A17B with mini-SWE-agent.

The paper reports 1,222 verified DevEval instances, 731 SWE-Bench Pro public-split instances, all Terminal-Bench 2.0 instances, and an experiment cost around $6,000. In Table 6, GPT-5.4 with Codex leads DevEval at 94.8 percent solve rate and $0.20 average cost per instance. Claude Opus 4.7 with Claude Code leads SWE-Bench Pro at 69.1 percent and Terminal-Bench 2.0 at 68.5 percent.

The key finding is not a single winner. It is that public records are incomplete and harness-dependent. The paper argues that agentified evaluation preserves much of the public-record fidelity while surfacing missing head-to-head comparisons. It also reports harness co-adaptation: native pairings perform best in five of the reported harness-swapping comparisons, and given sufficient time Opus 4.7 with Claude Code reaches 79.8 percent while Opus 4.7 with Codex reaches 73.6 percent in the relevant swapped setting.

Benchmark Receipt

An agentified benchmark should ship a receipt. The receipt should name the judge agent, subject agent, A2A protocol version, MCP tool list, environment image, task source, task split, hidden files, scoring script, model, harness, system prompt, reasoning setting, timeout, retry rule, cost accounting, transcript, tool calls, artifacts, parser failures, benchmark version, and public-record comparison rule.

The receipt should also separate protocol validity from benchmark validity. A2A and MCP can make evaluations reproducible and interoperable. They do not decide whether the task distribution is representative, whether the metric is meaningful, whether the judge is fair, or whether a platform-run private agent is independently auditable.

This belongs beside AI Agents, Model Context Protocol, AI Agent Observability, The Harness Becomes the Runtime Contract, The Reliability Scorecard Becomes the Agent Gate, The Judge Change Becomes the Measurement Drift, The Agent Environment Becomes the Discovery Lab, and The Agent Fix Becomes the Knowledge Debt.

Limits

The paper is strongest as infrastructure design. It shows that heterogeneous agent evaluation can be coordinated through agent-native protocols and that a field competition can get developers to adopt the pattern. It does not prove that every judge agent is valid, every benchmark becomes fair, or every platform mode is equally auditable.

There is also a standardization risk. A shared interface can make bad measurements easier to run at scale. AAA standardizes the route by which agents and benchmarks interact; the community still has to govern task choice, hidden data, model leakage, judge prompts, metric design, and leaderboard incentives.

The right conclusion is practical. AgentBeats gives agent evaluation a portable contract. The next fight is making the contract evidence-bearing enough that a score is not just reproducible, but interpretable.

Sources

Return to Blog