Artificial intelligence is entering classrooms faster than evidence can keep up, and a growing body of research suggests that short-term performance gains from AI tools may mask serious long-term learning deficits. While 34 states now have official AI guidance for K-12 schools, the underlying assumption that AI improves student learning is more complicated than policymakers realize.

A University of Pennsylvania research team conducted a field experiment involving nearly 1,000 high school students and found a troubling pattern: students who used ChatGPT demonstrated a 48% improvement in grades while working with an AI tutor, but when AI access was removed, they scored 17% worse than students who had never used AI at all. A recent report from Stanford University's AI Hub for Education confirmed the pattern, showing that AI tools often boost academic performance while students have access, but those gains weaken or vanish when students are assessed on their own without using AI tools.

Why Does AI Feel Like It Works When It Actually Doesn't?

The problem lies in how learning actually happens in the brain. When students use AI to summarize a reading, draft an essay, or work through a problem, they may produce polished outputs, but they are not building lasting knowledge. The shortcut bypasses the very processes that develop real understanding. Durable learning requires cognitive engagement, productive struggle, and repetition, none of which happen when a tool does the heavy lifting for the student.

"Short-term performance gains can mask long-term learning deficits. AI is a crutch," explained Norman Eng, a lecturer at the school of education at CUNY's Brooklyn College and founder of EducationXDesign Inc.

Norman Eng, Lecturer at CUNY's Brooklyn College

This does not mean AI has no place in K-12 classrooms. It means its place in schools must be defined carefully, with student learning as the primary criterion, not efficiency or improved test scores. The research points toward a clear principle: design lessons around the conditions known to produce durable learning first, and integrate AI only when it genuinely supports rather than substitutes for those conditions.

What Are the Four Questions Teachers Should Ask Before Using AI?

Experts have identified four core conditions that produce lasting learning. Before adding any AI tool to a lesson, teachers and instructional leaders should work through each question to determine whether the technology genuinely supports learning or undermines it.

• Will students have to use, recall, and demonstrate core content knowledge? Higher-order thinking is built on a foundation of domain knowledge. If an AI tool actively engages students with foundational content through retrieval practice, targeted feedback, or elaborative questioning, it may be worth integrating. If it lets students bypass that content, it is almost certainly counterproductive.
• Will students have to apply their learning to a new context? Transfer, or applying knowledge to a new situation, is one of the most reliable signs of genuine understanding. If an AI tool scaffolds that transfer while preserving cognitive effort, it may add value. If it does the transfer for the student, learning is short-circuited.
• Will students have to think independently and defend their own reasoning? Critical thinking requires students to make judgments and defend them. The productive struggle of brainstorming, drafting, and revising is where the learning happens. AI-generated scaffolding can short-circuit this process entirely, like taking a shortcut at the gym and skipping the part that actually builds strength.
• Will meaningful human interaction be preserved? Peer feedback, collaborative problem-solving, and teacher-student dialogue do more than support academic learning; they develop the social and intellectual habits that define educated citizens. Before adding any AI tool, ask whether it complements or competes with these interactions.

How to Evaluate AI Tools for Your Classroom

The decision to integrate AI into instruction should follow a structured approach based on learning science rather than enthusiasm from technology vendors. Here are practical steps educators can take to ensure AI serves learning rather than undermines it:

• Start with learning conditions: Before considering any AI tool, identify which of the four core conditions your lesson already supports. Only then should you ask whether AI can enhance those conditions without replacing student effort.
• Test the substitution problem: After completing an assignment using AI, can the student explain in their own words why they made the choices they made? If the answer is no, the tool is likely substituting for thinking rather than supporting it.
• Examine the burden of proof: Remember that AI developers have profit motives that have nothing to do with improving student outcomes. The enthusiasm of technology companies is not evidence of pedagogical effectiveness. The burden of proof should be on the technology and the policymakers pushing it, not on the teacher who questions it.

Across the country, school districts are moving quickly to adopt AI, with Chicago public schools' guidebook touting AI's potential to elevate educational delivery and New York City public schools recently releasing guidelines aimed at establishing "a foundational vision for how to use AI going forward". Nevada's state-level guide explicitly "embraces AI in its schools." Yet this rapid adoption is outpacing the evidence base for effectiveness.

What Does the Global Conversation Look Like on AI in Education?

The conversation about AI in education extends beyond the United States. Rwanda is positioning itself at the center of a new dialogue about how artificial intelligence and emerging technologies can reshape education through data-driven decision-making. EdTech Mondays Rwanda, a monthly talk show that has grown into one of the country's most influential spaces for national dialogue on technology and learning, recently explored how AI, machine learning, automation, and analytics can improve learning outcomes and build a future-ready workforce.

The discussion brought together diverse perspectives from research, policy, and innovation, including insights on how data generated through AI systems can support evidence-based policymaking while addressing ethical concerns linked to data governance, privacy, and the localization of AI models. One of the strongest messages emerging from that conversation was that AI should not replace teachers, but strengthen education systems through better decision-making, personalized learning, and improved classroom support.

Globally, UNESCO estimates that 6 billion people, nearly 74 percent of the world's population, were using the internet in 2025, while AI tools are increasingly entering schools and universities. Yet the organization also warns that only 40 percent of primary schools worldwide are connected to the internet, exposing a persistent digital divide that continues to shape educational inequality.

For educators caught in the middle, pressured to integrate tools their districts endorse and their students already use, the message is clear: ask hard questions before adopting AI. The four conditions that produce durable learning, not the promises of technology companies, should guide the decision.