The Analyser Becomes the Safety Case
A July 2026 arXiv paper asks a sharp systems question: if an LLM-assisted tool writes a hazard analysis, who analyses the tool? Constitutional Meta-STPA turns the safety analyser into the target system and binds its governance rules to validators, manifests, model records, and export gates.
The Paper
The paper is Samuel Tetteh, Udip Shrestha, Joshua R. Waite, and Cody Fleming's Who Analyses the Analyser? Self-Validating LLM Hazard Analysis with Constitutional Meta-STPA, arXiv:2607.08054. The arXiv API lists version 1 as submitted on July 9, 2026, with primary category cs.LG and a cs.AI cross-list. The PDF is 26 pages, and the title page lists Iowa State University affiliations across Mechanical Engineering, Electrical and Computer Engineering, and the Translational AI Center.
This belongs beside the site's work on AI safety cases, AI evaluations, agent governance roadmaps, runtime safety cases, and portable action receipts. Its fresh angle is reflexive safety engineering: the safety-analysis tool itself becomes the system under analysis.
What It Builds
Constitutional Meta-STPA wraps Systems-Theoretic Process Analysis around an LLM-assisted STPA tool. The tool produces losses, hazards, controllers, unsafe control actions, and constraints, but a deterministic harness owns much of the structure. After the model proposes losses, hazards, and a control structure, the engine creates one unsafe-control-action slot for every controller, control action, and STPA UCA type. The model fills slots; it does not decide which slots exist.
The paper's governance layer is a derived constitution: 21 Tool Principles for the analysis the tool writes and 8 Meta-Safety Principles for the machinery around it. The meta layer covers audit logging, run manifests, model and version pinning, hash-only logging by default, constitution pinning, semantic-matching voting, validator-gated export, and prompt-injection resilience. The paper says each principle is bound to a code enforcement point.
The tool also runs eight validators over the finished artifact, including checks for completeness, all four UCA types, evidence, no fabricated standards, measurable constraints, responsibility assignment, overconfidence, and limitations. Markdown and JSON are still written for inspection when validation fails, but redistributable artifacts such as PDF and email are gated on a clean validator report or an explicit recorded override.
The Benchmark Signal
The self-derivation experiment asks whether the tool, when analysing itself, recovers the principles that the authors derived from meta-STPA. A stronger ensemble, claude-opus-4.8 plus claude-sonnet-4, recovers 18 of 21 canonical Tool Principles and all 8 Meta-Safety Principles. A weaker pair, gpt-4o-mini plus gpt-4o, recovers 12 of 21 and 3 of 8. The authors repeat the loop on a second, independently authored AI-assisted code and security review tool and report that all 8 governance principles re-emerge there too.
The behavioural experiment uses 20 adversarial probes and scores responses from 0 to 2. Adding the 21 Tool Principles raises the mean safety score from 1.03 to 1.85, a reported 79 percent lift, with Wilcoxon p=0.0004 and Cohen's d=1.29. A generic helpful, harmless, honest preamble does not produce the same effect, and adding the 8 Meta-Safety Principles does not materially change the held-out probe score because those principles govern logging, pinning, voting, and export rather than the text of a single probe answer.
The paper also reports ordinary STPA runs across five vendors, using gpt-4o, claude-sonnet-4, gemini-2.5-flash, llama-3.3-70b, and deepseek-v3 on automatic emergency braking, infusion pump, and UAV autoland systems. All five complete all three systems, but thoroughness varies from 45 to 183 unsafe control actions per system.
Why It Matters
The paper's most useful move is to refuse the outsourced-safety loop. If an LLM helps draft the safety analysis, the wrapper around that LLM is now part of the safety-relevant system. It can hallucinate standards, omit UCA types, suppress uncertainty, keep no audit trail, or export a failed analysis as if it had passed. Those are not just model errors. They are tool-control errors.
That distinction matters for procurement and oversight. A vendor can say that a safety assistant "uses STPA" while leaving open whether prompts are pinned, model versions are recorded, validators are deterministic, model disagreement is surfaced, failed reports are blocked from distribution, or sensitive system descriptions are logged in full. Constitutional Meta-STPA turns those questions into explicit control points.
What It Does Not Prove
This is still a research paper, not an assurance standard. The authors are explicit about limitations: the coverage metric relies on lexical scanners and an LLM-judge cross-check that agree only moderately, with Cohen's kappa reported as 0.39. Practising safety engineers have not independently scored the same analyses or adversarial probes, so the paper cannot claim human-expert calibration.
The generalization claim is also bounded. The class-level result rests on two AI-assisted tools, three hardware negative controls, one analysis paradigm, English-language probes, and hosted models reached through a single router. The paper reports manifests and hashes to make drift detectable, but it does not make hosted model drift impossible.
Governance Reading
The Spiralist concern is the safety report that arrives without a chain of custody. A polished hazard analysis can look institutional while hiding the route from system description to losses, hazards, controllers, UCAs, constraints, validator failures, model disagreement, and reviewer override. The document can become a mask for the process that made it.
A defensible AI safety assistant should therefore ship two artifacts: the analysis and the analyser receipt. The analysis says what might go wrong in the target system. The receipt says what might have gone wrong in the tool that produced the analysis, and what controls were active when it ran.
The Receipt
An analyser receipt should include the system description hash, constitution hash, tool version, git SHA, model endpoint, model version, temperature, seed, prompt hash, response hash, token count, latency, validators run, validator errors, standards reference list, multi-model vote settings, pairwise disagreement report, export-gate decision, override text if any, generated artifact hash, reviewer identity, reviewer edits, and release destination.
The practical rule: a safety-analysis model is not outside the safety case. Once it drafts the hazard record, the analyser belongs inside the record too.
Sources
- Samuel Tetteh, Udip Shrestha, Joshua R. Waite, and Cody Fleming, Who Analyses the Analyser? Self-Validating LLM Hazard Analysis with Constitutional Meta-STPA, arXiv:2607.08054, submitted July 9, 2026.
- arXiv API record for arXiv:2607.08054, checked for title, authors, subject class, submission date, update date, and abstract.
- arXiv PDF for arXiv:2607.08054, checked for page count, affiliations, pipeline design, principles, validators, experiments, model names, quantitative results, limitations, and reproducibility claims.
- arXiv experimental HTML for Constitutional Meta-STPA, checked for section structure, figures, table text, method details, appendix principles, and threat discussion.