A cluster randomized trial conducted across 18 Tennessee middle schools over two academic years provides some of the first large-scale experimental evidence on generative AI tutoring in K-12 settings. Researchers Philip Oreopoulos and Nina Low, writing for the Annenberg Institute at Brown University, studied what happened when students in existing daily remedial mathematics sessions were given access to Khan Academy augmented with Khanmigo—Khan Academy's AI tutor configured to coach rather than supply answers. The study found that assignment to the AI-assisted condition raised math achievement by 1.3 national percentile ranks per term, equivalent to roughly 0.06 to 0.08 standard deviations over a school year. For students who actively participated throughout a full year, the implied effect reaches about 0.14 standard deviations.
These gains, however, are statistically indistinguishable from the improvements seen when students use Khan Academy's traditional practice exercises without AI assistance. The reason, the authors argue, is not that the tutor is ineffective but that students scarcely engaged with it. Ninety-six percent of students tried Khanmigo at least once, yet the median student sent messages to the tutor on only one-third of the days they practiced. In exercise sessions where a student made a mistake, Khanmigo was consulted in just 17 percent of cases. When students did interact, their messages were overwhelmingly bare answers or clicks on suggested prompts rather than substantive mathematical dialogue. The researchers conclude that the binding constraint on AI tutoring's promise is not access but engagement: realizing the technology's potential will require getting students to use it meaningfully, not merely making it available.
The study's design—a cluster randomized trial embedded in regular school routines—strengthens its ecological validity, but several limitations temper the conclusions. The measured effects reflect an intent-to-treat analysis (assignment to the condition, not actual usage), so the 0.14 standard deviation figure for active participants is an implied estimate, not a directly observed causal effect. The sample covers middle-school remedial math in one state; whether similar patterns hold for other grades, subjects, or student populations remains untested. Additionally, the Khanmigo configuration studied here explicitly withheld direct answers to encourage coaching; different prompting strategies or tighter integration with classroom instruction might yield different engagement patterns. What the evidence does establish is that, under real-world conditions where students control whether and how to invoke an AI tutor, utilization is low and superficial, and the resulting achievement gains are modest and comparable to non-AI practice tools.