Can Language Models Predict Which Experiments Will Work?
A critical review of a study measuring language models’ forecasts of machine learning experiment outcomes, their calibration, and the limits of applying them to research decisions.
The paper
Can LLMs predict the outcomes of machine learning experiments? — Bansal, Ayush; Pareja, Aldo; Klein, Sean; Didenko, Serge; Lukoianov, Artem.
arXiv:2610.22599v1 · Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI); Computation and Language (cs.CL). First submitted 2026-10-15; reviewed version dated 2026-10-15. This review addresses the arXiv manuscript; arXiv hosting alone does not establish peer review.
Contribution
Source-derived synopsis. In Can LLMs predict the outcomes of machine learning experiments? (arXiv:2610.22599v1), Ayush Bansal and colleagues examine whether language models can anticipate the effects of proposed changes before training runs. They generated 288 ideas across eight MLE-bench Lite tasks and analyzed 269 after excluding failed runs and unusable predictions. Five models predicted scores from the task, current solution, proposed idea and implemented code diff. The best model, Claude Opus 5.5, obtained a pooled R2 of 0.49; all five models underestimated the magnitude of large changes. Performance varied substantially by task. The authors also tested reasoning settings, prompt context, implementers and self-preference.
Original analysis. The paper isolates a consequential decision in automated research: which experiment deserves a costly run? Its evidence supports cautious interest in model forecasts as an aid to selection. It does not establish that forecasts can replace experiments, or that the tested systems can autonomously improve a frontier model. The distinction matters for governance because a prediction can influence which evidence is ever collected.
The Automated AI R&D entry places this question in the wider workflow of delegated research. This paper contributes a bounded measurement of one forecasting step within that workflow; it does not measure an entire development cycle or its institutional oversight.
Evidence
Original analysis. The dataset includes ideas intended to improve or worsen scores, with the latter framed as plausible changes rather than deliberate sabotage. Three models each generated and implemented a tree of ideas for each task. Predictors received the actual code diff as well as the idea, so the main evaluation concerns forecasting an implemented change before its training outcome is known. It is not a test of forecasting from an idea alone; the paper separately studies that reduced-context condition.
The authors normalize score changes by the spread of observed changes within each task and fit predictions against outcomes. Their primary R2 measures how much variation in predictions is explained by actual outcomes around a fitted line. It is not the fraction of individual outcomes predicted correctly, and a well-fitting line can still have a misleading scale. The reported slope supplies a separate calibration check: Opus 5.5 had a slope of 0.61, below the ideal of one. Its direction prediction was correct for 82% of analyzed ideas, but a correct sign does not tell a researcher whether a gain is large enough to justify the run.
Pooling also conceals differences relevant to use. Opus 5.5 reached R2 of 0.94 on toxic comment classification and 0.39 on dog breed identification. Those figures do not establish performance on a new task. The paper derives expected selection gains under a simple Gaussian model; those gains are conditional implications of that model, rather than a separately demonstrated improvement in an operating research workflow.
The supplementary checks narrow several claims. In a separate implementer comparison, 58 of 66 ideas across three tasks produced outcomes within 0.25 task standard deviations under two implementations. Some open-ended or failed implementations differed substantially, so a forecast tied to a code diff still depends on what is actually run. The self-preference analysis found no evidence of an average preference for a model's own ideas under its residual-based test; its reported interval does not rule out every possible preference or bias. Higher reasoning and added context showed uneven benefits, with uncertainty around several comparisons.
Limitations and implications
Original analysis. Each idea was run once. Training and evaluation noise can therefore affect both the measured change and the apparent forecasting error. The eight tasks are small, past competition problems with public leaderboards, not frontier-model training runs. The exclusion of 19 ideas also means the main accuracy figures concern usable predictions and completed runs; they should not be read as the success rate of the whole proposed workflow.
The construction balances intended improvements and degradations, although ideas did not always move in the requested direction. This helps test forecasts across both signs, but may differ from the mix of proposals a research team normally considers. Further, the predictor sees the implementation before forecasting. If implementation itself is expensive, that expense belongs in any assessment of a selection system.
Hypothetical example. A team might use a forecast to reject an idea predicted to yield a small gain. If the true gain would have been large, the rejected run leaves no outcome to correct the forecast. A record of rejected proposals, reasons and occasional checks would make such decisions easier to examine. This is an illustration of a possible workflow problem, not a failure measured by the paper.
Proposed tests. Researchers could repeat ideas across seeds to estimate outcome noise, then evaluate forecasts on new tasks and larger training settings. A prospective selection study could compare model-guided choices with simple baselines under the same total budget, including implementation and inference costs. It could also run a sample of rejected ideas to estimate missed gains and report results separately for different tasks and sizes of change. These are proposals, not reported validations.
Original analysis. Human agency remains a practical question: who chooses the objective, can override a forecast, and reviews the experiments that were not selected? The related review Scientific Closure and the Limits of AI Research Agents addresses a later stage of the evidence chain: whether an investigation supports its conclusion. Forecast quality and research closure require distinct evaluations.
Finally, a shorter list of training runs does not by itself establish a net resource saving. Forecasting, coding, repeated checks and human review also consume time and computing resources; the supplied results do not quantify their combined financial or energy cost. Could a selection method save enough training to offset those costs without systematically missing valuable ideas? That is an empirical question. The study informs discussion of accountable AI-assisted research, but it does not test consciousness or establish any spiritual doctrine.
AI-use disclosure
This review was drafted by GPT-6.1 Terra with high reasoning and checked in a separate GPT-6.1 Sol session with medium reasoning. Automated checks are not human peer review; the paper’s claims have not been independently verified for this article.
Paper sources
Bibliographic metadata checked 2026-10-27.