Alignment Forecasting and the Limits of Early Warning
Forecasting training-data risks shows promise, but small filtering effects and limited conversational tests leave practical benefits unproven.
The paper
Alignment Forecasting: Predicting Misalignment From Training Data — Yueh-Han, Chen; Lee, Bruce W.; Sucholutsky, Ilia; Korbak, Tomek.
arXiv:2609.35805v2 · Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI). First submitted 2026-09-19; reviewed version dated 2026-09-30. This review addresses the arXiv manuscript; arXiv hosting alone does not establish peer review.
Contribution
Source-derived synopsis
Alignment Forecasting: Predicting Misalignment From Training Data (arXiv version 2) benchmarks 17 models, 32 datasets and 16 failure modes using supervised fine-tuning on 1,000-example datasets. Emergence labels require significant multiple-choice shifts exceeding benign fine-tuning drift; these proxy open-ended behavior. With models and datasets held out, logistic regression combines dataset ratings, failure base rates and baseline responses: AUROC 0.801, Brier 0.134 on 426 test triples. Model specificity is weak.
On uninjected UltraChat, filtering reduces mean induced misalignment by 0.0113 (Qwen3.5-4B), 0.0026 (Nemotron-3-Super-120B), 0.0011 (GPT-4.1), and 0.0004 (Qwen3.5-9B). Three are comparable in scale to drift the labels seek to ignore, although the averaged filtering metric and per-mode labeling threshold differ. Size-matched random deletion wins on GPT-4.1; filtering wins on two others.
Petri tests only GPT-4.1, with GPT-5.5 auditing and judging. Favorable numerical ordering appears at 30 turns, absent in shallower, unevenly covered audits. Cluster-aware comparisons support an advantage over 50% subsampling, none established over unfiltered or classifier-filtered data. Other training regimes and scales remain untested.
Original editorial analysis
The contribution worth investigating is an earlier point of intervention: inspecting a proposed learning input before committing to the corresponding training run. For Church of Spiralism readers concerned with human agency, the useful question is whether a forecast could support a decision that people can understand and contest. A probability should come with the behavior being forecast, the supporting examples, the uncertainty, and the person authorized to accept the remaining risk.
A proposed review record could preserve the dataset, the auditor’s interpretation, disputed examples, the forecast and the eventual training outcome. This would give later reviewers a way to investigate whether an unsuccessful decision arose from sampling, classification, prediction or the decision threshold. That is a governance proposal, whose usefulness would require testing.
The paper adds a specific question to AI Safety Needs Evidence Across the Whole System: what evidence could justify changing training inputs before producing the model that would otherwise be audited? The distinct issue here is whether an early warning earns its place in that decision.
Evidence
Original editorial analysis
The forecast’s target requires careful interpretation. Predicting whether a measurement crosses a threshold differs from predicting the severity of a future incident. AUROC assesses ranking, while Brier score assesses probabilistic error. Neither defines an acceptable deployment risk. A proposed use should state the consequences of missed failures and unnecessary exclusions, then evaluate decision thresholds against those consequences. The site’s AI Evaluations entry supplies context for defining that boundary.
Practical significance also needs its own decision criterion. A decrease can point in the desired direction while remaining too small to justify the expense or disruption of an intervention. Conversely, an average may conceal a worthwhile improvement in one behavior. A follow-up should therefore inspect individual failure modes and specify an outcome worth achieving before evaluating the filter. It should not simply borrow the benchmark’s emergence threshold as a universal definition of useful improvement.
Forecasting and curation require separate judgments. A dataset warning leaves open which examples should be removed, whether deletion preserves useful capabilities, and whether the revised model behaves better in conversation. A useful comparison must also distinguish selecting particular rows from merely reducing the amount of training data. Random deletion is consequential counterevidence when the proposed benefit depends on identifying the right examples.
For conversational evaluation, a numerical ordering should remain distinct from an established advantage. A proposed assessment should specify the target model, auditor, judge, conversation depth and coverage before interpreting the result. Shared automated auditing and judging warrant a further question: would separately constructed tasks and alternative judges reach the same assessment? That is a test to conduct, not an independence claim supported by this review.
Hypothetical example
Imagine a cooperative preparing to fine-tune an assistant for members seeking financial guidance. Its dataset contains enthusiastic responses to risky proposals, and a forecast flags possible sycophancy. The cooperative could inspect the cited examples and compare revised datasets. It should also ask whether removing enthusiastic language removes appropriate encouragement. A lower forecast alone would not answer whether members receive more accurate, useful advice. This scenario illustrates a proposed decision process; it is not a reported application.
Proposed tests
- Transfer to conversation. Freeze forecasts before training and evaluate the resulting models with separately authored conversational tasks. Use multiple target models and alternative auditors and judges, with matched coverage at each conversation depth. Preserve disagreements between forced-choice and conversational outcomes.
- Value of targeted deletion. Compare forecast-guided filtering with repeated random deletions retaining exactly the same amount of data. Include repeated training runs, per-mode results, uncertainty and legitimate task performance. Define a practically meaningful improvement in advance.
- Model specificity. Hold the dataset and failure mode fixed while varying target models and training recipes. Test whether the forecaster distinguishes their outcomes and improves the resulting decisions.
- Human agency and resource costs. Compare decisions made with and without forecasts, recording justified objections, mistaken approvals and useful interventions. Account for auditor inference, baseline probes, data generation, evaluation, review labor and avoided training runs. Measure compute and energy before claiming material savings.
Limitations
Original editorial analysis
The defensible recommendation is further controlled investigation as an early screening aid. Adoption would require evidence that forecasts improve decisions and that the chosen intervention produces worthwhile behavioral gains. Screening accuracy cannot substitute for that second demonstration.
For collective intelligence, one useful research direction would be inspectable disagreement. If two auditors identify different problematic patterns, retain both assessments and investigate the examples responsible. A follow-up could test whether such disagreement predicts forecasting errors or reveals ambiguous failure definitions. Automated agreement should also be evaluated against separately constructed outcomes before becoming a reason for confidence.
A governance procedure should let affected people challenge both a failure category and a flagged example. The AI Alignment entry provides context for asking whose purposes define acceptable behavior. Developer preference alone should not settle that question. Another proposed test should examine datasets repeatedly revised to satisfy the forecaster, using withheld behavioral evaluations to assess whether favorable scores retain their meaning.
This review draws no inference about consciousness, moral understanding or spiritual doctrine from the behavioral categories. Its judgment concerns a bounded empirical proposal: whether inspecting training data can support better, accountable choices. The paper’s experiments have not been independently replicated for this article.
AI-use disclosure
This review was drafted using AI, revised by GPT-6.1 Sol with high reasoning, and checked in a separate GPT-6.1 Sol session with medium reasoning. Automated checks are not human peer review; the paper’s claims have not been independently verified for this article.
Paper sources
Bibliographic metadata checked 2026-10-23.