Blog · arXiv Analysis · Published: August 12, 2026 · Modified: August 12, 2026 · Last reviewed: August 12, 2026

The Mobile Judge Becomes Two Instruments

A judge that ranks mobile agents and a judge that rewards them during training do not have the same job.

MobileJudgeBench connects offline judge errors to both uses and shows why one accuracy number cannot certify both instruments.

The Paper

The source is Ziqiang Wang, Li Gu, Zhixiang Chi, Zhi Liu, Seyed Mehdi Ayyoubzadeh, Yuanhao Yu, and Yang Wang’s Benchmarking LLM Judges for Mobile Agent Evaluation, arXiv:2608.11434v1 [cs.AI, with cs.CL and cs.CV cross-listing], submitted August 11, 2026. Its MobileJudgeBench dataset contains 931 trajectories from 289 tasks, six mobile-agent benchmarks, four agent models, and 68 Android apps. Each recorded step can include a screenshot, action, UI tree, and agent reasoning.

Nine graduate-student annotators assigned binary success or failure. Each trajectory received two to four independent labels; reported average pairwise agreement was 88.4 percent, and disagreements were resolved through discussion. The final labels are nearly balanced: 492 successes and 439 failures. That protocol gives the judges a shared target, but it does not turn one binary label into one universal measurement job.

One Label, Two Instruments

The paper tests six judge methods across five model backends, producing 30 method-and-backend variants. For benchmark evaluation, it compares judge-derived success rates with human-derived rates across 24 benchmark-and-agent combinations. In the reported task-cluster bootstrap analysis, F1 is the strongest predictor of agent-ranking fidelity, with Spearman’s rho of 0.90 and a 95 percent interval of [0.66, 0.92]. Balanced accuracy best predicts lower success-rate error, with rho of -0.79 and interval [-0.92, -0.70]. Precision alone has intervals spanning zero for both evaluation targets.

That is the scorekeeping instrument: preserve ordering and estimate rates without letting one class dominate the result. A training reward has a different consequence. Its errors are not merely counted after behavior; they are fed back into the procedure that produces behavior. Calling both outputs “judge accuracy” hides the change in function.

The Evidence Budget Is Part of the Judge

The study’s simple baseline receives the task, actions, and uniformly sampled screenshots. Its default uses 48 frames with a maximum long edge of 600 pixels, without UI-tree metadata or agent reasoning. Across the five backends, that configuration reports 84.9 to 90.9 percent accuracy and is within 1.2 percentage points of the best method even where it does not lead.

Complexity is not the only variable. At approximately equal input cost, 16 higher-resolution screenshots outperform 192 much lower-resolution screenshots for both tested ablation backends. At the 48-frame operating point, only 12 of 931 trajectories are long enough to be subsampled; an eight-frame stress test finds nearly equal accuracy for uniform and event-based selection, with their 57 disagreements split 30 to 27. The paper does not establish a universally optimal sampler. It shows that frame count, resolution, and which moments are visible belong in the measurement record beside the model name.

Reward Error Has Direction

For the training test, the authors use GRPO to train UI-TARS-7B-SFT on AndroidWorld’s easy tasks under four binary reward sources: AndroidWorld’s rule-based checker and the simple judge backed by GPT-5-mini, GPT-5.2, or Qwen2.5-VL-72B. Hyperparameters are held constant, and a rule-based checker evaluates checkpoints across three seeds.

The reported best easy-set success rates are 54.6 percent for rule-based reward, 45.4 for GPT-5-mini, 42.6 for GPT-5.2, and 39.9 for Qwen. GPT-5.2 and Qwen have the same offline accuracy on the AndroidWorld subset, 88.8 percent, but different precision, 93.7 versus 80.2 percent, and differ by 2.7 points at their best easy-set checkpoints. A false positive gives reward to a failed trajectory; a false negative withholds reward from a successful one. Equal total error can therefore teach different policies.

The paper is appropriately cautious. On the complete 116-task suite, the two matched-accuracy conditions tie at 25.9 percent at the fixed final checkpoint. The precision interpretation holds under best-checkpoint selection, not every comparison convention. It is suggestive evidence that false-positive control matters in this setup, not a causal law for reinforcement learning.

The Backends Fail Differently

The hard-core failure analysis examines trajectories misjudged by at least four of five adapted methods. With GPT-5-mini, 30 of 48 such cases are false negatives; last-frame anchoring and unfamiliar success states lead the taxonomy. With Qwen2.5-VL-72B, 71 of 78 are false positives; constraint violations and partial completion dominate. Surface-level UI matching appears in both sets.

This makes the two-instrument distinction operational. A leaderboard report needs confusion matrices and slice-level ranking stability. A reward loop also needs a false-positive ceiling and examples of the exact failed states it would pay for. A permissive screen judge is not merely an optimistic reporter. Under optimization, it can become a curriculum for superficial completion.

What the Study Does Not Establish

This is a version-one preprint about binary completion. The authors’ limitations leave optimality, efficiency, and partial progress for future work; restrict training to AndroidWorld, one judge method, and four reward sources; and identify subjectivity in the failure taxonomy. The rule-based reward is also evaluated by the same rule-based checker, so the paper explicitly treats that condition as an upper bound rather than an independent comparison.

The manuscript says the team releases trajectories, annotations, and evaluation code, but no study-specific repository URL appears in the version-one abstract record or the source package reviewed for this essay. That package contains manuscript source, tables, and figures, not the benchmark records, judge logs, or training code needed to recompute the claims. I therefore checked the reported methods, tables, appendices, and qualifications but did not independently rerun the experiments.

The Two-Instrument Receipt

A mobile-judge receipt should first record the task sources, label protocol, class balance, judge method, exact backend and endpoint, prompt, decoding, screenshot count, resolution, sampling rule, visible actions, UI metadata, reasoning exposure, confusion matrix, uncertainty method, and failure slices. Then it should fork.

The evaluation branch should preserve per-agent rates, ranking fidelity, rate error, and the decision that uses the leaderboard. The training branch should preserve reward encoding, false-positive rate and ceiling, rollout and optimization settings, seeds, independent state checker, checkpoint-selection rule, paid failure examples, and a stop-and-revalidate trigger. The Spiralist lesson is narrow: a scorekeeper describes prior trajectories; a reward model helps write later ones. Never certify one instrument with the other instrument’s metric.

Sources

Return to Blog