AI Tutoring Evidence: Design, Not Access, Drives Learning Gains

A weekly digest catches a field in the middle of a correction

Research on artificial intelligence in education now arrives faster than most instructors can read it. A single week can bring more than a hundred new studies. The second issue of the AIEd Research Weekly Update, the newsletter of the open AI in Education Knowledge Base, came out on October 31. It added 121 studies to a collection that now holds 1,926 article summaries, 222 concept pages, 42 FAQs and 37 resources.1 The newsletter sorts its findings by audience, with sections for instructors, developers, researchers and administrators. Its main highlights converge on one uncomfortable claim: what AI does inside a course matters far more than whether the course has AI at all.1

That theme is not limited to one newsletter. Over the past several weeks, randomized trials, a Stanford evidence review, an OECD report and a stream of campus integrity disputes have pointed the same way. Taken together, the coverage describes a field moving away from easy optimism. AI is no longer being treated as a learning upgrade you can simply switch on. The working assumption is now closer to this: AI changes how students spend their effort, and that can help or hurt.

The tutoring results are real, and also narrow

The best-known positive result is still the Harvard physics trial by Kestin and colleagues. In that study, students using a purpose-built AI tutor learned more in less time than classmates in an active-learning class session.1315 The OECD's Digital Education Outlook 2026 gave the result fresh attention this week. Its coverage cited an effect size of 0.63, which the OECD calls large, and noted higher student motivation and engagement.10

Sources disagree on some details. Coverage of the OECD report describes a trial with 316 students10. Several research summaries put the analyzed sample at 194111316. Reported effect sizes also run from 0.63 in a linear model up to a range of 0.73 to 1.3, depending on the method used.1316 The more careful reviews agree on the limits. The authors built the tutor themselves. The study covered two lessons, not a whole course. Learning was measured only by an immediate post-test, so retention was never tested.1215

The other positive results also come with conditions. Stanford's Tutor CoPilot coached human tutors instead of replacing them. Students were 4 percentage points more likely to master topics, and students of lower-rated tutors gained 9 points, at a cost of roughly $20 per tutor per year.1517 A teacher-supported program in Nigeria produced gains of about 0.23 standard deviations in English and 0.31 on a broader assessment.11 In each case, the AI sat inside a structure built and run by people.

When AI is simply added, results flatten or reverse

The most important new evidence is negative. In a semester-long randomized trial at the University of Maryland, 2,379 undergraduates and 30 instructors took part. Course sections given an integrated AI tutor ended up with final grades 0.37 standard deviations lower. Recorded participation in online course activities fell by 0.90 standard deviations.1819 Losses for first-generation students were more than twice as large as for their peers.1619 One commentator blamed the setup: most instructors left the tool in a mode that hands out answers.19 Another account added a caveat. Only about 15% of students in the tutor sections used it even once, and the outcome was course grades, not an independent test.18 Even so, a tool that most students never opened was linked to lower grades and less participation. Read together, these accounts suggest the main problem was displaced effort, not bad answers from the tool.

That reading fits other recent studies. In the high-school math experiment by Bastani and colleagues, unrestricted GPT-4 raised practice scores by 48% but lowered unaided exam scores by 17%. A hint-first version avoided the drop but did not beat having no AI.1215 A two-year Khanmigo trial in 18 Tennessee middle schools found gains of about 0.06 to 0.08 standard deviations a year. That is similar to Khan Academy practice without AI, and well below the 0.288 average for human tutoring.1217 Students used the tutor only now and then.14

The weekly update adds a mechanism to this picture. Across four graduate courses, an AI teaching assistant answered first in 97% of student-started forum threads. Its median response time was 0.92 minutes, compared with 490 minutes for classmates. As the share of follow-ups coming from peers fell from 91% to 45%, self-regulatory language in the threads dropped from 49% to 36%.1 The newsletter's advice is to delay the assistant's first reply rather than remove it.1 This is the Maryland pattern at a smaller scale: a fast, helpful machine crowds out the slower human exchanges where reasoning happens.

Published averages are shrinking under scrutiny

Optimistic meta-analyses are also losing ground. The weekly update describes a meta-analysis of 58 studies whose 0.86 effect lost two-thirds of its size after correction for publication bias. Its advice to researchers is to plan for a small effect, not the published average.1 A separate preregistered meta-analysis of 49 generative-AI studies in STEM likewise found that little of its positive effect survived bias correction. One distinction held up: AI that extends students' own work helped, while AI that takes the work away could harm.16 One widely shared meta-analysis reporting large gains from ChatGPT was retracted on April 22, 2026.12 Other recent pooled estimates, such as an average g of 0.67 across 35 studies, come with heterogeneity near 91%. That means a single average says little about what will happen in any given classroom.11

The Stanford SCALE Initiative's K-12 review makes the gap concrete. Of 818 papers, only 20 met What Works Clearinghouse standards for strong causal inference. None of those high-quality causal studies looked at US K-12 students.3 The review's central finding is that AI boosts performance while students use it, but results are mixed or negative once it is taken away. Tutoring-specific tools consistently do better than general-purpose chatbots.3 The evidence is not uniformly bleak: a TU Berlin lab experiment found a 0.23 standard-deviation gain on an unaided test.16 Still, this is the pattern a careful reader should expect.

Academic integrity: the measurement problem has become a fairness problem

The integrity debate rests on the same tension between performance and learning. The most rigorous estimate comes from a study in Science of 95,513 students at 20 US public research universities. Using a list-randomization method, it put the share who had used generative AI to cheat at about 9%. Among daily AI users the figure rose to 26%, compared with 7% for monthly users.2024 In the UK, the Higher Education Policy Institute found that the share of undergraduates who put AI-generated text directly into assessed work rose from 3% in 2024 to 12% in 2026.2029 At Oxford, investigations into suspected AI misconduct rose 46% in a year, though the university says the rate is still about 1.3 per 1,000 students.29

Detection is losing ground as an enforcement tool. Reporting by AFP says Yale and Cornell have banned or discouraged faculty from treating detector output as the main evidence of cheating. UW-Madison's guidance calls the tools "imperfect at best" and notes their bias against non-native English speakers.26 Harvard College dean David Deming said the university should get "out of the AI-detection business."22 In a faculty survey, 55% judged detectors ineffective and only 23% thought they worked.22 Students accused of AI use are increasingly turning to lawyers. In one case, a New York court annulled an Adelphi University finding that relied on a 100% Turnitin score.23 The credibility problem reaches administrators too. Dartmouth's provost, who had written publicly about AI cheating, resigned after a review found he had not disclosed his own significant AI use, including in that op-ed.27

The replacements now emerging are oral exams, AI-use logs and grading of the work process rather than only the final product.22 These follow the same logic as the tutoring evidence: make the student's own thinking visible. A review of 50 innovative US universities found that only six frame their main AI page as "policy". Their integrity processes are moving from automated gatekeeping toward stated expectations, evidence of process and due-process protections.5 The weekly update reports one small study in which teaching AI-assisted writing as a genre, instead of policing it, produced post-test score distributions with no overlap with the control group.1

The reading

The tutoring trials, the bias-corrected meta-analyses and the integrity data all point to one conclusion. Generative AI reliably raises the measures most visible to institutions, such as practice scores, homework grades and finished essays. Those are also the measures least able to show whether a student learned anything. The tools that help are the ones built to keep effort with the learner: hint-first tutors, AI that coaches a human tutor, and assignments that require visible reasoning. Tools that remove that effort, whether a chatbot answering in the course sidebar or a student pasting in a generated essay, damage both learning and the credibility of grades.

The practical advice for institutions follows from that. Before licensing a tutor, they should test it against closed-book performance rather than usage counts or satisfaction surveys, and set its defaults with care.31219 They should also check what data the tool sends to third parties. Among 42 chatbot platforms tested, half sent conversation titles to trackers.1 The evidence does not support the claim that AI tutoring has failed. It does undercut the idea that giving students access to AI is, on its own, a teaching strategy.