Blog · arXiv Analysis · Last reviewed July 10, 2026

The Concept Game Becomes the Agent Test

Sophia Koehler and coauthors turn the logic game Zendo into an agent benchmark where success depends on seeing, hypothesizing, and choosing useful experiments, not merely labeling examples after the fact.

Published: July 10, 2026 Modified: July 10, 2026 Last reviewed: July 10, 2026 Tags: arXiv, agents, visual reasoning, active learning, evaluation, AI governance

The Paper

The paper is Playing ZendoWorld: Challenging AI Agents on Active Visual Concept Induction, arXiv:2607.08233 [cs.AI, cs.CV]. The arXiv API lists Sophia Koehler, Antonia Wüst, Inga Ibs, Wasu Top Piriyakulkij, Wolfgang Stammer, Constantin Rothkopf, Kevin Ellis, and Kristian Kersting as authors, with version 1 submitted on July 9, 2026. The PDF is 33 pages. The arXiv HTML links code at ml-research/ZendoWorld and data at a Hugging Face Zendo synthetic-data page.

This belongs near AI agents, agent benchmarking, agent society tests, oracle-style agent evaluation, benchmark curricula, and AI audit trails. Its fresh angle is that an agent benchmark can ask whether a system knows which observation to buy next.

The Game

Zendo is a rule-discovery game: some scenes satisfy a hidden rule and others do not. ZendoWorld keeps that structure but makes the evidence visual and interactive. An agent sees labeled rendered scenes, proposes new scenes, receives feedback from the game environment, and eventually states the rule. The task is therefore not just perception, not just logic, and not just dialogue. It is a loop: perceive, hypothesize, intervene, revise.

The paper reports an instantiation spanning 22 games. The rules cover basic predicates, counting, parity, spatial relations, logical connectives, and one out-of-distribution rule that was deliberately left outside the Oracle agent's domain-specific-language vocabulary. The shared episode protocol allows at most 30 observations per game. A game counts as won only when the final hypothesis is logically equivalent to the hidden rule.

That scoring choice matters. Many AI evaluations reward the answer after a static prompt. ZendoWorld rewards the investigation that produced the answer. A system can classify a pile of examples well and still fail to recover the rule that explains them.

Results

The paper evaluates four agent families: a privileged Oracle agent with a pretrained vision model and DSL access; a pure VLM agent; a Bayesian agent using particle-style hypothesis inference; and a vision-language-programs agent that combines learned visual concepts with program synthesis. The authors also collect human data on a subset of games.

In Table 3, the Oracle agent wins 95.5 percent of games with 68.2 percent label accuracy, while the pure VLM agent wins 44.5 percent with 56.7 percent label accuracy. The Bayesian agent has the highest label accuracy, 75.2 percent, but wins only 13.6 percent of games. The VLP agent wins 18.2 percent with 66.6 percent label accuracy. Human participants win 73.3 percent on their subset, with 53.9 percent label accuracy.

Those numbers are the paper's central warning. Label accuracy and rule recovery can point in different directions. The Bayesian agent often knows whether a scene is positive or negative, yet does not reliably name the compact rule. The human comparison cuts the other way: participants can win more often while labeling less accurately. For governance, that is a useful distinction. A model that predicts cases is not automatically a model that has learned the concept a policy owner thinks it has learned.

Experiment Choice

The strongest Spiralist lesson is in the experiment proposals. The paper measures expected information gain, or EIG, for proposed examples in a symbolic setting. VLM-based agents produce much lower EIG than the Oracle agent; after roughly 10 examples, the VLM and Bayesian agents approach near-zero EIG. The authors interpret this as agents generating examples too similar to what they have already seen or staying attached to already refuted hypotheses.

The surprising comparison is the random Oracle variant. In the paper's EIG analysis, diverse random proposals rank second, above more structured agents whose hypothesis set is poorly calibrated. That is not an argument for randomness as wisdom. It is a reminder that a bad search policy can turn reasoning into repetition. An agent that cannot choose discriminating experiments may look coherent while wasting its interaction budget.

The compute receipt is also concrete. The primary experiment is reported as 22 games times 5 seeds times 4 agents, or 440 runs, on one NVIDIA Tesla V100-SXM3-32GB GPU. The authors report 321 GPU-hours and 155,125,194 GPT-5-mini API tokens for the complete evaluation.

Limits

ZendoWorld is synthetic and controlled. Its objects, predicates, rendering process, grammar, and game loop are designed by researchers, not discovered in an open institution. The Oracle's strength partly comes from privileged structure, and its failure on the out-of-distribution rule shows the cost of a fixed vocabulary. The VLM results also depend on specific prompts, models, and the paper's experimental setup.

Those limits do not weaken the benchmark's value. They clarify it. ZendoWorld is not proof that one agent is generally intelligent or that another cannot reason. It is a receipt for a narrower competence: active visual concept induction under a stated rule space, interaction budget, scene generator, scoring rule, and experiment-choice metric.

The Receipt

A concept-induction receipt should record the task generator, hidden-rule grammar, out-of-distribution rules, rendered-scene pipeline, perception module, agent prompts, model versions, API-token budget, seed list, observation budget, proposed experiments, feedback sequence, final hypothesis, logical-equivalence checker, label accuracy, win rate, structural F1 curve, EIG calculation, failed games, human-comparison protocol, code repository, data location, and compute footprint.

The audit question is not "did the agent answer correctly?" It is "which examples did the agent choose, what uncertainty did those examples remove, and did the final rule survive a logical equivalence check?"

Sources

Return to Blog