Making AI Tutoring Productive: Evidence from a Mastery-Based Math Practice Experiment
中文摘要
大语言模型（LLM）可以提高辅导的可扩展性，但前提是学生利用它们推理并理解错误，而非逃避努力。我们通过一项随机现场实验研究这一问题，实验对象为汉密尔顿县学校的6,000多名初中生，使用的是基于研究开发的计算机辅助学习（CAL）平台NUMI。学生被随机分配接受人工智能支持或仅接受CAL支持，采用掌握式或非掌握式进阶方式，并学习两个数学主题之一。一周后，他们完成了一项涵盖已练习和未练习材料的延迟测评。接受人工智能支持的学生进度较慢，尝试作答的问题较少，但在进入作答环节的条件下，答题准确率更高。最明确的作用机制出现在答错之后：人工智能提高了下一次作答的正确率，减少了恢复正确作答所需的尝试次数，同时增加了学生在每道获得结构化支持的问题上投入的时间。掌握式进阶提高了连续答对三题的达成率，但其本身并未改善延迟学习效果。最令人鼓舞的延迟测试证据出现在将人工智能嵌入掌握式学习流程时，边际显著的增益集中在练习过的练习1材料上。结果表明，相较于标准CAL，大语言模型辅导能够带来额外价值，但其价值取决于能否通过结构化设计将错误转化为富有成效的学习时刻。
Abstract
Large language models (LLMs) can make tutoring more scalable, but only if students use them to reason through mistakes rather than avoid effort. We study this question in a randomized field experiment with more than 6, 000 middle-school students in Hamilton County Schools using NUMI, a research-based computer-assisted learning (CAL) platform. Students were randomized to AI versus CAL-only support, mastery versus non-mastery progression, and one of two math topics. One week later, they completed a delayed assessment covering practiced and unpracticed material. AI students progressed more slowly and attempted fewer questions, but answered more accurately conditional on reaching an attempt. The clearest mechanism appeared after mistakes: AI improved next-attempt correctness and reduced attempts needed to return to a correct answer, while increasing time spent on each question receiving structured support. Mastery increased three-correct-in-a-row attainment but did not by itself improve delayed learning. The most encouraging delayed-test evidence appears when AI is embedded in the mastery workflow, with marginally significant gains concentrated on practiced Exercise 1 material. The results suggest that LLM tutoring can add value over standard CAL, but its value depends on structure that turns mistakes into productive learning moments.