What Does an Agent Attack Success Rate Measure?
A critical review of research on how measurement choices shape attack success rates in agentic AI security evaluations.
The paper
Attack Success Rate Is Not a Number: On Measurement Validity in Agentic AI Security Evaluation — Pathade, Chetan; Pawar, Prathamesh; Patil, Shubham.
arXiv:2609.25173v1 · Cryptography and Security (cs.CR). First submitted 2026-09-21; reviewed version dated 2026-09-21. This review addresses the arXiv manuscript; arXiv hosting alone does not establish peer review.
Contribution
Source-derived synopsis. In Attack Success Rate Is Not a Number: On Measurement Validity in Agentic AI Security Evaluation, arXiv:2609.25173v1, Chetan Pathade, Prathamesh Pawar, and Shubham Patil examine whether attack success rates in agent-security studies can be compared. They identify six choices that shape the metric: the unit counted, the success criterion, the number of trials, attacker adaptivity, attacker knowledge, and the treatment of partial success. They combine a review of 259 arXiv papers with an analytical study of statistical power and ranking errors. They propose a ten-item reporting checklist but do not test whether adopting it improves comparability. The authors do not claim that a particular defense is ineffective.
Original analysis. The title is provocative, but the useful point is precise: a percentage needs a definition and a test procedure before it can support a comparison. An agent may encounter several malicious instructions during one task. Counting each instruction answers a different question from counting whether the task was compromised at least once. Both can be useful, but a reader cannot assume that two percentages describe the same risk merely because both are called attack success rates.
The choice of success criterion is equally consequential. A text response that contains an attacker’s desired phrase is observable, but it is different from a tool action that changes a record or exposes information. The paper’s distinction is relevant to human agency: an evaluation should specify whose intended task is being protected and which actions would count as a loss of control. The broader AI Evaluations page places that question within the governance of a complete system and its deployment setting.
Evidence
Original analysis. The reporting study began with twelve topical arXiv queries, screened for empirical agentic-security work, and analyzed 259 papers with a detected numerical attack success rate. The authors used full-text pattern detectors, checked and revised them against manually inspected passages, and hand-coded a random sample of 50 papers for the headline uncertainty measures. In that sample, 58% reported neither a variance estimate nor repeated runs, with a 95% Wilson interval of 44.2% to 70.6%. The automated estimate for the full corpus was 65.3%. Because the authors revised the detectors using the hand-coded sample, agreement on that sample is optimistic; the two estimates should not be treated as independent confirmations.
For 64 papers confirmed to use an LLM judge as the success oracle, 29.7% reported an agreement check against human labels. This is a finding about reported checks, not proof that the other judges were inaccurate. The paper also reports that only 30.9% of the corpus disclosed any of the decoding information it sought. Most other automated rates rely on small validation samples and lack measured recall, making their exact values less secure than the headline hand-coded estimate.
The second study is analytical rather than an end-to-end attack experiment. Under its stated independent-binomial assumptions, at a benchmark size of 100 units and a baseline attack success rate of 30%, the calculated minimum detectable difference is about 18.2 percentage points at 80% power and a 5% significance level. A normal approximation puts the chance of reversing the order of two defenses separated by five percentage points at about 21% in the illustrated setting; the authors report a Monte Carlo check of about 23.7% for that cell. These figures describe modeled conditions, not an observed frequency of wrong rankings in deployed systems.
The paper also models how an imperfect success oracle could reverse a ranking when its errors differ between systems. Its example requires a substantial difference in false-negative rates; it does not establish that existing judges actually produce such reversals. This distinction matters for AI Red Teaming: a plausible failure mechanism identifies something to investigate, while an observed failure would require evidence from the tested systems.
Limitations and implications
Original analysis. The corpus is limited to English-language arXiv papers found by the authors’ queries, and some screened papers lacked available HTML. The inclusion rule also depended on detecting a number near an attack-success mention; the authors identified 88 additional papers that mentioned the metric without a detected numeral, including cases showing signs of conversion loss. Whether that loss is independent of reporting practice remains unverified. All manual adjudication was by one coder, so the reported agreement between detector and adjudicator does not measure agreement between independent human reviewers. The authors have not published an artifact repository.
There is also a limit to the paper’s broad warning about cross-paper comparison. Its analysis gives good reasons to question comparisons made without aligned definitions and uncertainty estimates; it does not inspect every possible pair of studies or demonstrate that every published comparison is invalid. Similarly, the checklist addresses identifiable reporting gaps, but reporting a setting does not by itself make two settings equivalent. Agreement on units, threat models and decision-relevant outcomes would still require coordination and judgment.
Hypothetical example. An organization choosing an assistant to update case records sees two vendor scores. One counts every injected instruction; the other counts compromised cases. Even if both scores are accurate within their tests, a procurement decision cannot safely rank the systems from those figures alone. The organization would need comparable tests of the actual workflow, a clear definition of unauthorized changes, and a way for people to review and correct a disputed action.
Proposed tests. Independent reviewers could code the same sampled papers using a published rubric, report disagreements, and test the automated detectors on a separate held-out sample. Researchers could also run paired defense evaluations with the same tasks and attack budgets while varying one measurement choice at a time. Such work could show when rankings change in practice and how much the checklist helps reviewers interpret them. These are proposals, not results of this paper.
Original analysis. Better evidence has costs: repeated agent runs, human adjudication and retained records require resources. The supplied paper does not measure their energy use or total infrastructure cost. Evaluators could record compute, reviewer time and storage alongside the decisions supported, so institutions can weigh evidence quality against those costs. The paper contributes to accountable AI governance by making the basis for a security claim inspectable; it does not investigate consciousness or establish a spiritual doctrine.
AI-use disclosure
This review was drafted by GPT-6.1 Terra with high reasoning and checked in a separate GPT-6.1 Sol session with medium reasoning. Automated checks are not human peer review; the paper’s claims have not been independently verified for this article.
Paper sources
Bibliographic metadata checked 2026-10-27.