The Safety Monitor Becomes the Alarm Threshold
A deployment guardrail is not only a classifier. It is a threshold, a risk budget, and a decision about when generation must stop.
The Paper
The paper is Online Safety Monitoring for LLMs, arXiv:2607.02510 [cs.AI], cross-listed in Computation and Language, Machine Learning, Applications, and Machine Learning statistics. The arXiv record lists Mona Schirmer, Metod Jazbec, Alexander Timans, Christian Naesseth, Maja Waldron, and Eric Nalisnick as authors, with version 1 submitted on July 2, 2026. The record notes the ICML 2026 Hypothesis Testing Workshop, and the 12-page PDF lists affiliations with UvA Bosch-Delta Lab at the University of Amsterdam, University of Wisconsin Madison, and Johns Hopkins University. The paper also points to a public code repository at monasch/llm-monitor.
Why Online Matters
Much AI safety testing is pre-deployment or post-hoc: run evaluations before release, or classify the completed answer after it appears. This paper studies the more operational problem of watching an output as it unfolds. At each step, the generator has produced only a partial sequence. A monitor receives a safety signal, such as a probability from an external verifier, and must decide whether to remain silent or raise an alarm.
That is a different governance surface. If the monitor waits until the end, the harmful or incorrect output may already have reached the user. If it interrupts too early, it blocks ordinary use and teaches people to route around the safety layer. The system therefore has two visible failure modes: false alarms on safe outputs and missed detections on unsafe ones.
The Threshold Is the Policy
The authors study a deliberately simple monitor. It takes a stream of verifier scores and raises an alarm the first time the score falls below a calibrated threshold. The threshold can be selected through conformal risk control, which controls risk in expectation, or through a high-probability upper-confidence-bound procedure using the Hoeffding-Bentkus bound. The paper focuses on false-alarm risk in the main text and extends the same framing to missed-detection risk in an appendix.
This is useful because it makes the safety promise legible. A deployment team is not just saying that a safeguard exists. It is choosing which risk to bound, what tolerance level to use, what calibration data to trust, and what intervention follows the alarm: halting generation, escalating to a stronger verifier, or sending the case to human review.
What the Experiments Show
The first experiment monitors factuality on the MATH dataset. The authors compare two generators with different mathematical capacity: Claude Haiku 4.5, reported in the paper as solving 90 percent of problems correctly, and Mistral-7B-Instruct-v0.3, reported as solving 26 percent. OpenAI's o3-mini supplies final correctness labels, and Qwen2.5-Math-PRM-7B supplies step-level process-reward scores.
Against e-valuator baselines based on sequential hypothesis testing, the simpler risk-control monitors remain competitive. The paper reports that high-probability monitors generally controlled false-alarm rate, with e-valuator-anytime violating the bound at epsilon 0.05 and 0.1 on Mistral. It also reports that conformal risk control satisfied its in-expectation guarantee. The most deployment-relevant result is timing: the threshold methods detected incorrect reasoning earlier than the e-valuator monitors, which can reduce user exposure and token cost when an answer is going wrong.
The second experiment monitors harmfulness on Anthropic Red Teaming conversations and FineHarm. The Red Teaming setup uses Llama Guard as the verifier signal. The FineHarm setup uses a Qwen2.5-1.5B safeguard verifier fine-tuned for token-level harmfulness. False-alarm control holds across the reported monitors and datasets, but power varies sharply with the signal.
Signal Quality Is Safety Quality
The paper's strongest lesson is that the monitor can only be as good as the signal it receives. On FineHarm, where the verifier is tuned to the online detection task, the paper reports power close to 1. On Anthropic Red Teaming, Llama Guard's power stays below 0.8. In the MATH ablation, replacing the external process-reward model with the generator's own token log-probabilities lowers monitoring power substantially; near epsilon 0.3, the PRM variants exceed 0.9 power while the log-probability counterparts sit around 0.5.
That result matters for procurement. A cheap internal confidence score may be attractive because it avoids an extra verifier pass. But if the signal is weak, the monitor's statistical wrapper cannot rescue it. The budget question is not only compute cost. It is the cost of false security.
The Alarm Receipt
An online safety monitor should produce an alarm receipt. It should record the generator, verifier, signal definition, threshold, calibration set, target risk, confidence parameter if used, observed score path, alarm step, and downstream intervention. It should also state whether the system was controlling false alarms, missed detections, or another risk.
Without that receipt, an interruption is just a black-box refusal or a silent safety failure. With it, developers can audit whether alarms cluster around certain prompts, whether a verifier has drifted, whether a threshold is too conservative, and whether a human escalation path is being used as designed.
Limits
The study is a workshop paper using selected factuality and harmfulness benchmarks. It does not prove that a single threshold is sufficient for every deployment. The authors state two key limits: a time-invariant threshold inherits the verifier's limits, and it ignores temporal structure in the score sequence. They point to future work on combining signals, calibrating accuracy-cost tradeoffs, triggering targeted checks, and using per-step thresholds.
The practical conclusion is modest and strong. Real-time monitoring is not solved by naming a guardrail. The actual safety object is the calibrated alarm system: the signal, the threshold, the guarantee, the delay, the intervention, and the record left behind.
Sources
- Mona Schirmer, Metod Jazbec, Alexander Timans, Christian Naesseth, Maja Waldron, and Eric Nalisnick, Online Safety Monitoring for LLMs, arXiv:2607.02510 [cs.AI].
- arXiv PDF for Online Safety Monitoring for LLMs, checked for affiliations, workshop note, problem setting, methods, experiments, results, limitations, and acknowledgments.
- Public code repository monasch/llm-monitor, checked for existence, project description, homepage, and public repository status.
- arXiv recent listing for Artificial Intelligence, checked for current submission metadata and subject listing.