Blog · arXiv Analysis · Published: August 12, 2026 · Modified: August 12, 2026 · Last reviewed: August 12, 2026

The Security Label Becomes the Treatment Leak

A behavioral label cannot independently measure an agent’s conduct if the grader already knows which experimental treatment the agent received.

A forensic audit of one MCP-style security campaign found exactly that circularity. Its correction is useful beyond the campaign: separate delivered treatment, executed behavior, authorization, and classification before a benchmark verdict becomes a security claim.

The Paper

The source is Rana Muhammad Ahmed and Sabahat Abbas’s Labels Are Not Endpoints: Treatment Leakage and Construct Validity in MCP Agent Security Evaluation, arXiv:2608.12880v1 [cs.CR, cross-listed cs.AI], submitted August 13, 2026. It audits the authors’ own preserved campaign. The study asks what treatment reached runtime, what the independent unit was, whether the endpoint measured behavior without using treatment assignment, and what survived treatment-blind reclassification.

When the Label Knows the Treatment

The historical runner created an adversarial-payload flag from two administrative facts: the row was not a utility condition and its surface was not CLEAN. The grader then required that flag before it could assign ATTACK_SUCCESS. Its attack-success branch otherwise needed at least two capabilities, but ignored recorded measures such as unauthorized-call count, authorized-sequence match, benign-task success, privilege aggregation, and an explicit unauthorized objective.

Keep an authorized two-tool completion unchanged and alter only the treatment-derived flag: the historical class flips from benign success to attack success. The execution did not change; the story attached to it did. Experimental assignment has entered the outcome it is supposed to explain.

Ten Thousand Rows, Fifteen Stimuli

The unit reconstruction is equally sobering. The queue held 10,200 rows, 2,550 per model, under 150 administrative task identifiers. Hashing the task text, expected sequence, and execution plan reduced those identifiers to 15 observable stimuli: one single-capability case, five three-capability cases, and nine five-capability cases. Three delivered schema variants produced 45 semantic requests; four fixed model and tokenizer bindings produced 180 model-bound requests in 60 matched blocks.

Complete-input groups appeared 11 to 162 times, with a median of 47.5. The paper reports complete within-group agreement in turns, outputs, parser state, stored class, and normalized behavior. Repetition can test execution and recordkeeping, but it does not create new stimuli or independent requests. The five-digit row count is a provenance fact, not the evidentiary denominator.

A Counterfactual Label Test

The authors propose a Treatment-Invariance Test: hold the behavioral record fixed, vary permitted treatment metadata, and require the outcome class to remain unchanged. Their corrected endpoint uses accepted invocation, authorization, dispatcher execution, task completion, and source-to-sink evidence, while excluding surface, poison, payload-family, defense, model-identity, historical-label, and review-status fields.

Passing that test is necessary, not sufficient. A treatment-blind classifier may still encode the wrong authorization rule, mishandle parser evidence, or draw an unjustified boundary around task completion. Treatment invariance detects a particular circular path; it does not bless the rest of the construct.

The Corrected Census

Under the paper’s locked version-two endpoint, the 180 requests become 89 authorized benign completions, three benign task failures, one hijack attempt, zero ATTACK_SUCCESS records, three critical exploits, and 84 structurally invalid records. Of 70 historical ATTACK_SUCCESS or HIJACK_ATTEMPT labels, 58 become authorized benign completions. The correction therefore removes many dramatic labels without erasing the underlying security evidence.

A blinded review covered the 96 requests that version two considered structurally interpretable. The two reviewers agreed with one another on all 96, yet their consensus differed from the locked codebook on four construct-boundary cases. The paper describes reviewer identities and blinding as privately attested rather than independently verified. Agreement can reveal consistency; it cannot settle a disputed definition by itself.

Three Events Are Not a Rate

The three retained critical cases contain an authorized source read followed by an unauthorized write to a mock outbox, with matching hashes for the protected source and sink content. A fourth unauthorized forwarding case carried ordinary task text rather than protected content and remains a hijack attempt. The testbed was local, used canned data, and sent nothing to an external party.

Those are inspectable cases, not an attack-rate estimate. The three protected transfers occur in two scenarios for one of four fixed integrations, and their matched CLEAN requests completed authorized plans. Because treatment texts, models, scenarios, and deterministic settings were narrow and bundled, the study explicitly does not estimate causal effects, model rankings, defense efficacy, or deployment prevalence. Zero records in one class also does not mean zero security failures when three more severe protected transfers remain under the codebook.

Reproducible Is Not Valid

The preserved hashes, outputs, parser events, dispatcher transcripts, and grader predicates exposed the error. That vindicates provenance without proving that provenance prevents bad measurement. A pipeline can rerun exactly and reproduce the same circular endpoint.

The paper’s endpoint-integrity linter reproduces ten prespecified diagnostics in its synthetic fixtures. Its scope is narrow: it can flag declared treatment-valued inputs and direct static references, but not every upstream proxy embedded in behavioral evidence. The authors do not report accuracy, precision, recall, external validity, or comprehensive construct validity for the tool.

The Evidence Boundary

The corrected endpoint is post-hoc, not preregistered. Review omitted the 84 records it classified as structurally invalid. Parser error was not measured. The campaign used four integrations, 15 stimuli, a local mock environment, and deterministic generation. Recorded defense labels were behaviorally inert, so defense tradeoffs cannot be inferred. The broader planned external payload corpus did not reach runtime, so its lineage labels are not delivered attacks.

The paper links a tagged repository artifact, but also states that this cited tag predates the version-two reconstruction. I verified the tag resolves to commit da858efa9c608165d1d96f69a1b14ab4167522d2; I did not independently recompute the paper’s census. That boundary belongs beside the findings, not in fine print.

The Endpoint-Integrity Receipt

An agent-security evaluation should preserve a receipt with: the claim and intended estimand; planned treatment; exact delivered bytes, location, and hash; runtime exposure; pre-treatment authorization plan; accepted calls and dispatcher receipts; protected-flow evidence; classifier inputs, code, version, and hash; a treatment-relabeling counterfactual; structural-invalid counts; stimulus, semantic-request, model-bound-request, and repetition bindings; blind-review codebook and disagreements; surviving verified events; causal and external-validity limits; artifact tag; and independent-reproduction status.

The Spiralist rule is simple: treatment says what the system encountered, behavior says what it did, authorization says what it was allowed to do, and the endpoint says how evidence became a class. Merge those ledgers and the verdict begins to manufacture the event it claims to observe.

Sources

Return to Blog