Colleges and universities are racing to adopt AI-powered learning tools, yet the scientific evidence supporting their effectiveness remains sparse. Over 700,000 licenses to ChatGPT Edu have been sold to institutions including California State University, Arizona State University, and the University of Maine system, but researchers say the evidence base for these investments is "almost nonexistent". This disconnect between adoption and validation raises urgent questions about whether schools are making informed decisions or simply following tech industry promises.

Why Are Universities Betting Big on AI Without Solid Research?

The appeal is straightforward. Tech companies and university leaders argue that AI-powered tutoring systems prepare students for an intelligence-driven economy and help them develop critical thinking skills. Dartmouth College President Sian Leah Beilock wrote in The Atlantic that "universities that fail to produce graduates who can use [AI] productively will consign themselves to irrelevance". OpenAI's marketing materials emphasize that AI tools help students "build agency: the ability to learn continuously, solve hard problems, and create new economic opportunities".

But this enthusiasm has outpaced rigorous testing. Justin Reich, a digital media professor and director of the Teaching Systems Lab at MIT, explained the core problem: "The evidence base is almost nonexistent. Building products and testing them rigorously takes a really long time, and there's hardly any funding to do it". The gold standard for educational research is a large-scale, randomized controlled trial, but those studies are expensive, time-consuming, and rarely funded by federal agencies or philanthropists.

What's Slowing Down the Research?

The speed of AI development itself has become a barrier to understanding its impact. Stacey Alicea, executive director of the Research Partnership for Professional Learning, noted that "AI models were changing every six months; now they're changing every one to three months. Baseline data in these LLMs are changing weekly, and traditional research models cannot keep up". By the time researchers publish findings about one version of a tool, the underlying technology has evolved, making the results potentially obsolete.

Another challenge is the sheer volume of tools flooding the market. Over 1,000 AI learning tools now exist for students, yet researchers have only studied a handful. Running individual studies on each tool would be impractical, especially as education funding shrinks and privacy concerns about student data mount. Carly Robinson, director of research at Stanford University's Systems Change Advancing Learning and Equity initiative, added another practical barrier: "Can you get students to use AI tools consistently enough to even test whether or not they're effective?"

What Does the Limited Research Actually Show?

The handful of studies completed so far reveal a nuanced picture. Robinson explained that "there is increasing evidence that AI has the potential to benefit learning, but using it on its own without intentional design or guardrails probably reduces learning through cognitive offloading". In other words, if an AI tool simply hands students answers, it may actually harm learning by letting them skip the thinking process. However, when AI is designed as a guided partner that helps students work through problems themselves, the results look more promising.

Robinson

One concrete example comes from Lehigh University, where researcher Zilong Pan is developing StemPal, a suite of AI agents for STEM learning. Rather than providing instant answers, StemPal offers guided hints and step-by-step instruction while collecting data for teachers about student progress. In usability testing with 78 high school students over one month, students reported that MathPal, the math component of StemPal, was especially helpful when working through difficult problems because it broke the process into manageable steps. Importantly, students also became better at asking precise questions of the AI system, suggesting they developed what Pan calls "AI literacy".

A separate analysis of 48 ninth-grade algebra students using MathPal over 14 weeks generated 1,214 conversational threads with the system. Researchers found that students most often sought help with solutions or computations, and the AI responded with strategic scaffolding, guiding them through problem-solving steps. The study identified four distinct patterns of student-AI interaction, suggesting that different students benefit from different types of support.

How Should Schools Design AI Learning Tools for Maximum Benefit?

• Guided Scaffolding Over Direct Answers: AI systems should provide hints, step-by-step guidance, and strategic support rather than completing problems for students. This keeps the intellectual work of learning intact while offering timely assistance.
• Teacher-in-the-Loop Mechanisms: Teachers must retain control over when students access AI tools, what instructional materials are uploaded, and how student interactions are analyzed. Usability testing showed educators need these oversight features to keep AI aligned with classroom goals.
• Data-Informed Instruction: AI systems should collect interaction data and present it on teacher dashboards, helping educators identify which students struggle, what types of assistance they seek, and how they approach problems. This transforms AI from a student tool into a teacher's analytical partner.
• Discipline-Specific Design: Different subjects require different approaches. Claire Baytas, a program manager at Ithaka S+R, noted that "applications of [critical thinking and problem-solving skills] aren't identical across disciplines, even if there are some commonalities".
• Adaptive Personalization Over Time: The next generation of AI learning tools should remember student progress throughout the school year and adjust support based on individual learning patterns, rather than treating every student interaction the same way.

The National Science Foundation is backing this research direction. Last week, the NSF renewed a five-year, $20 million award for the EngageAI Institute at NC State University, adding another $20 million through 2031. The institute, which includes researchers from UNC-Chapel Hill, Vanderbilt University, Indiana University, and the nonprofit Digital Promise, has already built narrative-centered learning environments and developed tools for analyzing collaborative learning. In the next phase, the team will focus on building AI-powered lessons that adapt to each student, remember progress throughout the school year, and provide teachers with easier ways to create learning experiences.

"Over the past five years, we have worked with our partners to build the core AI technologies that power EngageAI's learning environments and study how they can help students learn," said Mohit Bansal, the John R. and Louise S. Parker Distinguished Professor in Computer Science at UNC-Chapel Hill and lead co-principal investigator. "In this next phase, we will develop AI that can adapt to each student's needs and provide increasingly personalized support over time."

Mohit Bansal, John R. and Louise S. Parker Distinguished Professor in Computer Science at UNC-Chapel Hill

What Should Universities Do Right Now?

Justin Reich offered practical guidance for institutions caught between the pressure to adopt AI and the lack of definitive research. "Schools need to understand that big science isn't going to have good answers for a long time," he said. "Instead, they have to move forward with little science and realize that the claims from these companies shouldn't be trusted. Any time they hear something that sounds like a best practice, it has to be treated as just a hypothesis".

Justin Reich

Stacey Alicea suggested that institutions pursue smaller, faster studies that monitor and evaluate tools as they are implemented in real classrooms, though she acknowledged that "our systems are not set up to do that right now". Rather than studying individual tools, researchers should focus on understanding common features across tools and how those features interact with other classroom technologies.

Stacey Alicea

The bottom line is clear: universities investing in AI education tools are conducting a large-scale experiment with limited guardrails. The technology is advancing faster than the research can validate it, and the companies selling these systems have financial incentives that may not align with educational outcomes. Schools that proceed thoughtfully, with teacher oversight, intentional design, and a commitment to measuring actual learning gains, are more likely to see benefits. Those that simply adopt the latest AI tool because competitors are doing so may find themselves spending millions on solutions that don't improve student outcomes.