When AI Helps Students Write Proofs, What Do They Learn?
A critical review of a randomized access study of AI-assisted proof learning, examining independent performance, self-selected chatbot use, and the limits of causal claims.
The paper
Supporting or Replacing Students' Reasoning? Cognitive Offloading in AI-Assisted Proof Learning — Chen, Eason; Judicke, Sophia; Beigh, Kayla; Tang, Xinyi; Wang, Isabel; Yuan, Nina; Xiao, Zimo; Li, Chuangji; Li, Shizhuo; Luttmer, Reed; Singh, Shreya; Yampolsky, Maria; Parikh, Naman; Zhao, Yvonne; Chen, Meiyi; Huang, Scarlett; Mohanty, Anishka; Johnson, Gregory; Mackey, John; Lin, Jionghao; Koedinger, Ken.
arXiv:2610.31481v1 · Human-Computer Interaction (cs.HC). First submitted 2026-10-22; reviewed version dated 2026-10-22. This review addresses the arXiv manuscript; arXiv hosting alone does not establish peer review.
Contribution
Original analysis. A tutor can make a proof easier to submit while leaving open whether the student can construct a new proof alone. That distinction is the central contribution of Chen and colleagues’ Supporting or Replacing Students’ Reasoning?, arXiv:2610.31481v1. The study gives educators a reason to measure both assisted work and later independent performance, rather than treating a polished homework answer as a complete account of learning.
For a Church of Spiralism review concerned with human agency, the practical question is observable: what intellectual work does the student have an opportunity to do, and what can the student explain or reproduce later? This framing does not require a claim about machine consciousness or spiritual doctrine. It asks how an educational system distributes opportunities to reason and how institutions evaluate the outcome.
The authors recommend against open-ended chatbots as tutors and propose tools that build on submitted attempts. That recommendation is stronger than the comparison their experiment can settle. The two interfaces were offered together, and students later chose how much to use each. The study provides a basis for testing the recommendation; it does not establish that requiring an attempt will improve learning.
Evidence
Source-derived synopsis. In one undergraduate proof-based mathematics course, 153 consenting students were randomly assigned earlier or later access to a chatbot and a proof-feedback interface. Earlier access produced an adjusted homework advantage of 2.56 points, but no statistically significant advantage on the next exam, completed without AI; the adjusted exam difference was −2.47 points, with a 95% confidence interval from −5.60 to 0.66. After both groups had access, students with lower baseline self-efficacy used the interfaces more. In a model adjusting for prior exam score, self-efficacy, and feedback requests, each 100 additional chatbot messages was associated with a 3.02-point lower subsequent exam score. Feedback requests were not significantly associated with exam scores. A mediation analysis found an indirect association through chatbot use. Selected dialogues and interviews supplied context, including examples of incorrect chatbot guidance. See the methods and results.
Original analysis. The randomized comparison concerns the offer of earlier access, including students who did not use the tools. It therefore supports a claim about that package of access in this course. It does not isolate the chatbot from proof feedback, or determine which students benefited. The exam interval includes both a possible modest disadvantage and a small advantage, so the absence of a statistically significant difference is not proof that access had no effect. The exam also used different questions of similar types, making it a useful independent assessment without establishing delayed retention or broad transfer.
The later usage result answers a different question. Message counts were logged rather than recalled, but usage was self-selected. A student facing persistent difficulty might send many messages and also score lower for reasons the model did not measure. Could extensive delegation of proof steps contribute to the association? That is plausible, but the study did not directly measure mental effort or establish what students did with the supplied reasoning. The indirect association from self-efficacy through chatbot use is likewise a statistical pattern, not proof of a causal pathway.
Hypothetical example. Two students might each send twenty messages. One could submit a proof attempt and challenge an incorrect hint; the other could repeatedly request the next step. Their counts would be identical, although the visible division of work would differ. Even complete transcripts would not by themselves reveal everything either student understood. An independent explanation or new problem would add a different kind of evidence.
Accuracy matters alongside the division of work. In one selected exchange, the chatbot gave an incorrect construction involving modular inverses; in another, it misdiagnosed the flaw in a student’s counting method. These examples show possible failure modes, not their frequency. A prompt that asks questions or requires an attempt can still direct a learner toward a mathematical error. The related review What AI-Assisted Mathematics Submissions Cannot Show considers the separate evidence problem of attributing work from a final submission.
Limitations and implications
Original analysis. The participants came from one course, and the 153 with records were a subset of 308 enrolled students. Volunteers may differ from classmates who did not consent. Course policy allowed other resources outside exams, and the study did not measure outside AI use, unrecorded help, time on task, or changes in self-efficacy. The final exam was outside data collection. Results could differ with other learners, subjects, models, prompts, or assessment arrangements.
The interface estimates also should not be ranked merely because the chatbot association was statistically significant and the feedback association was not. The study did not randomize interface choice or directly establish a difference between their effects. Its dialogue classifier was checked against human consensus on the same 80 sessions used to develop its prompt, rather than an independent test set. Initial-request labels cannot establish what students learned later, and selected interviews and exchanges cannot represent every interaction.
For accountability, an institution considering these tools should specify whether an assessment measures assisted completion, independent proof construction, or both. Usage counts can prompt a supportive conversation, but they are too coarse to serve as a verdict about ability or misconduct. The broader institutional roles and record-keeping questions are discussed in AI in Education. This paper does not measure energy, hardware, financial costs, or staff review time, so it cannot establish the material efficiency of either interface.
Proposed tests. A randomized study could compare answer-providing chat, hint-based chat, and feedback on submitted proofs while keeping the model and course content as similar as possible. It should assess supported work, later unaided proofs, delayed retention, and changed problems separately. Independent reviewers could check the mathematical correctness of feedback, while participants’ explanations and revisions could clarify how assistance was used. Assignment would test the effects of the designs; additional evidence would still be needed to determine whether a particular kind of student reasoning caused any difference.
A deployment evaluation could also record teacher review time and measurable computing costs alongside learning outcomes. Those measures would help schools judge the resources needed to deliver and oversee a tool. They are proposed evaluations, not results of this study.
AI-use disclosure
This review was drafted by GPT-6.1 Terra with high reasoning and checked in a separate GPT-6.1 Sol session with medium reasoning. Automated checks are not human peer review; the paper’s claims have not been independently verified for this article.
Paper sources
Bibliographic metadata checked 2026-10-27.