What a Refusal Can Hide: Reviewing Harmful Reachability in Language Models
A critical review of bounded harmful reachability, its evidence from jailbreak evaluations, and the limits of using it for AI governance.
The paper
Refusal Is Not Robustness: Measuring Harmful Reachability in Autoregressive Language Models — Kapoor, Parv; Fortes, Teresa; Mamede, Claudia; Abreu, Rui; Aldrich, Jonathan; Kang, Eunsuk.
arXiv:2610.22754v1 · Machine Learning (cs.LG) ; Logic in Computer Science (cs.LO). First submitted 2026-10-15; reviewed version dated 2026-10-15. This review addresses the arXiv manuscript; arXiv hosting alone does not establish peer review.
Contribution
Source-derived synopsis. In Refusal Is Not Robustness: Measuring Harmful Reachability in Autoregressive Language Models, arXiv:2610.22754v1, Parv Kapoor and colleagues propose bounded continuation analysis (BCA) to examine possible model outputs beneath an observed refusal. Their method builds a tree of high-probability token continuations, labels terminal branches with a harm evaluator, and sums their retained probabilities. A second measure, generative-attack susceptibility (GAS), tracks the largest bounded harmful mass found as an attack budget grows. The authors test these measures on instruction-tuned models and JailbreakBench behaviors, and use BCA to guide adaptive attack search. They report differences hidden by greedy refusal scores, positive associations with separately sampled harmful responses, and partially distinct failures found by BCA-guided and judge-guided searches.
Original analysis. The paper's central contribution is a more precise question for safety evaluation: what alternatives does a model assign probability to when one observed answer is a refusal? That question matters for human oversight because a single reassuring interaction can be a poor basis for judging how a system behaves across repeated or altered interactions. BCA gives evaluators another diagnostic, but its number describes a specified model, prompt, bounded tree and harm judge. It is not a general probability that a deployed product will cause harm.
Evidence
Original analysis. The construction preserves the model's original probabilities on retained branches; discarded mass goes to an unknown state rather than being redistributed. The calculation is exact for the constructed, labeled tree, while the tree is an approximation to the full distribution. The authors retain high-probability next tokens subject to a maximum branching factor and a finite generation horizon. If the branching cap prevents the retained tokens from reaching the chosen probability threshold, the paper's stated lower bound on retained mass no longer applies. This distinction makes the formal calculation valuable without making its answer complete.
On 100 direct JailbreakBench behaviors, all four tested models had 0% attack success under greedy decoding. Their reported GAS at zero attack budget nevertheless ranged from 0.002 for Llama-3.1-8B-Instruct to 0.211 for Mistral-7B-Instruct-v0.3. Those scores show a difference under the paper's bounded evaluation, not a measured difference in real-world harm. Under 18 templates, the paper's empirical success measure reached 100% for three models, while their maximum bounded scores still differed. The two measures answer different questions: whether any template produced a judged harmful completion and how much judged harmful mass was found in a bounded tree.
The authors also compared bounded scores with 100 independently sampled completions for each of 540 behavior–template pairs per model. This validation used 30 behaviors, a Qwen model to judge BCA leaves and HarmBench-Mistral-7B to score sampled completions. Reported Spearman correlations ranged from 0.534 to 0.733 across the four targets. The separate sampling and different judges give the comparison substance, but correlation does not establish that either judge accurately captures every relevant harm, nor does it prove that the score transfers to other populations or settings.
In adaptive TAP search, BCA-guided and outcome-judge-guided searches found partially different failures across 120 runs per target. Their combined coverage exceeded either alone in the reported comparisons. Yet using BCA both to guide search and select the final candidate produced a substantial drop from any-leaf success to selected-candidate success on every target. For example, the DeepSeek target fell from 73.3% to 31.7%; selecting with the outcome judge instead yielded 73.3%. This is an important qualification: a signal useful for exploring a search tree need not be a good rule for choosing its final output.
Limitations and implications
Original analysis. Semantic judgments of short terminal continuations are a consequential measurement choice. A prefix may be ambiguous, and a classifier's definition of harm may differ from the practical severity, accuracy or usefulness of a complete response. The supplied evaluation does not establish human agreement for these particular labels. Nor does the reported correlation isolate the effects of tree truncation, judge error and decoding choices. Comparisons therefore need the horizon, branching parameters, retained mass, judge and sampling procedure alongside the score.
Access is another boundary. Constructing the tree requires token-level probabilities, which the tested open-weight models provide. A product can also add interfaces, filters, tools or human review that the model-level tree does not represent. The broader distinction between a model and its deployed workflow is developed in AI Evaluations. A release or procurement decision needs evidence about the actual system and the people who will use or be affected by it.
Hypothetical example. A service operator sees a model refuse a prohibited request in a demonstration. A bounded analysis finds judged harmful mass on other continuations. That finding would justify investigating the specific paths and deployment conditions; it would not by itself tell the operator how often a user will see a harmful answer or whether an additional product safeguard will intercept it. A decision should identify who can inspect the finding, request a mitigation and require a retest.
Proposed tests. Independent evaluators could compare several tree depths and branching caps while reporting retained and unknown mass, then assess whether prompt rankings remain stable. A separate, blinded human review of a representative sample of full continuations could measure agreement with the automated judges and examine disagreements by harm category. Tests on held-out behaviors and actual product configurations could ask whether the diagnostic predicts failures that matter in those settings. These are proposals, not results reported by the paper.
Original analysis. The method also has material costs. Tree size can grow exponentially with depth, and the authors cite computational cost as the reason their sampling validation used a subset of behaviors. Recording compute, evaluator time and the added failures found would help organizations weigh the diagnostic against other uses of their testing budget. The discussion of AI Red Teaming places that choice within a wider campaign of scoped testing and remediation. This research informs accountability for AI systems; it does not investigate consciousness or establish a spiritual doctrine.
AI-use disclosure
This review was drafted by GPT-6.1 Terra with high reasoning and checked in a separate GPT-6.1 Sol session with medium reasoning. Automated checks are not human peer review; the paper’s claims have not been independently verified for this article.
Paper sources
Bibliographic metadata checked 2026-10-27.