The Tool Call Becomes the Scope Warrant

The Paper

The paper is Shane Caldwell, Max Harley, Ads Dawson, Michael Kouremetis, Vincent Abruzzo, and Will Pearce's ScopeJudge: Cost-Aware Pre-Execution Gating for Offensive Security Agents, arXiv:2607.07774 [cs.CR]. The arXiv record lists version 1 as submitted on July 8, 2026, with a 22-page, 4-figure, 4-table comment. The experimental HTML identifies the authors with dreadnode, USA.

The paper's starting point is narrow and concrete: an offensive-security agent may be allowed to inspect one target, endpoint, queue, or account, while a nearly identical action outside that boundary can breach an engagement, disrupt production, or invalidate a bug-bounty finding.

Why Scope Moves

Scope is not a property of the tool call alone. A request to read, scan, test, or enumerate can be allowed in one engagement and forbidden in the next because the user's instruction supplies the operative boundary. A static policy can say "respect scope," but it cannot know the future customer's declared target list.

That distinction is useful beyond security. Tool-using agents in finance, law, medicine, procurement, and operations often face the same pattern: the action type is ordinary, but the object it touches determines whether the action is authorized.

The Benchmark

ScopeJudge is built from 100 information-security trajectories drawn from ScopeBench, a suite of 30 tasks designed to tempt autonomous agents toward scope violations. The benchmark contains 4,897 labeled tool-call decisions, with 7.7 percent labeled as violations.

Each call was labeled independently by five professional penetration testers. The authors report Fleiss' kappa of 0.64 and mean pairwise Cohen's kappa of 0.64 across the full set. They also compute a mean per-grader F1 of 0.78, which becomes the expert agreement reference point for judging automated monitors. The source agents span five model families: Anthropic, Alibaba, Moonshot, Google, and OpenAI.

The Gate

The method is pre-execution gating. A strong, untrusted agent proposes a tool call; a cheaper, trusted judge sees selected context and decides whether the call should execute, be rejected, or escalate. The paper evaluates eight judge models under five transcript strategies: Static Policy, Intent, Intent Plus Tool Calls, Intent Plus Output Summarized, and Full.

Those strategies matter because context is not free. More history can improve the judge's view, but it raises token cost and enlarges the prompt-injection surface. Tool outputs and reasoning traces may help the judge infer intent, but they may also carry adversarial text into the monitor's own decision context.

Findings

One finding is that static policy fails as a scope gate. When the judge sees the global policy and proposed call but not the user's intent, recall collapses to near zero for seven of eight judges, with median recall 0.005. Adding intent raises recall to 0.60-0.94.

Using its June 2026 cost basis, the paper gives the best F1 result to GLM-5.2, an open-weight judge: F1 0.66 at 0.0060 dollars per call. The best proprietary judge, Claude Opus 4.8, reaches F1 0.60 at 2.9 times GLM-5.2's per-call cost. The authors recommend two operating points: a cost-sensitive GLM-5.2 plus Intent Plus Tool Calls configuration that catches 87 percent of violations at precision 0.50 for 0.0032 dollars per call, and a recall-first Claude Opus 4.8 plus Intent configuration that catches 94 percent at precision 0.39 while flagging 19 percent of all calls.

The Warrant

A scope warrant should preserve the user request, declared in-bounds and out-of-bounds surfaces, static policy version, proposed tool name and arguments, partial trajectory reference, transcript strategy, judge model, price snapshot, label, rejection or escalation outcome, reviewer decision, and any later override.

Without that warrant, the organization is left with vibes: the agent "seemed careful" or the monitor "usually catches things." With it, the team can audit the specific boundary that authorized a call, the context the judge saw, the cost tradeoff accepted, and whether a human was pulled in when precision was low.

Limits

The authors are explicit about the claim boundary. Results come from one calibration set, one task suite, one domain centered on web penetration testing and bug bounty work, and English-language tasks. The 7.7 percent violation rate is a property of that suite, so precision and F1 will move when deployments have different base rates.

The paper also does not measure adversarial robustness. Higher-context strategies may admit attacker-controlled tool outputs into the judge's context. Two judge models also generated a small fraction of the labeled trajectories, leaving possible self-preference unruled out. Each configuration was evaluated once at temperature 0, so nearby score differences should not be overread.

Governance Reading

The Spiralist reading is that agent governance must move from "permission to use a tool" to "permission to touch this object for this stated purpose." ScopeJudge makes that shift measurable. It says the warrant lives at the proposed call, not at the marketing claim, the model card, or the user's broad intention to "do security work."

For organizations deploying agents, the practical lesson is not that an LLM judge is enough. It is that every sensitive tool call needs a boundary artifact, a monitor, a cost model, a false-negative tolerance, and a human path when the call is too consequential to auto-approve.

Source Discipline

Primary sources were the arXiv abstract, API, PDF, and experimental HTML. This page paraphrases the paper without reproducing figures, tables, prompts, benchmark examples, tool-call traces, or long passages.

Sources

• Shane Caldwell, Max Harley, Ads Dawson, Michael Kouremetis, Vincent Abruzzo, and Will Pearce, ScopeJudge: Cost-Aware Pre-Execution Gating for Offensive Security Agents, arXiv:2607.07774 [cs.CR], submitted July 8, 2026.
• arXiv API record for arXiv:2607.07774, checked for title, authors, subject class, submission date, abstract, comments, and version metadata.
• arXiv PDF for ScopeJudge, checked for page count, dataset size, labeling protocol, judge roster, scoreboard, deployment recommendations, limitations, ethics, and impact statement.
• arXiv experimental HTML for ScopeJudge, checked for author affiliations, section structure, transcript strategies, benchmark construction, and abstract claims.

Return to Blog