Educational technology companies have long measured success by counting clicks, app minutes, and student engagement, but a growing body of research shows these metrics mask a troubling reality: students can be highly engaged with an app while learning almost nothing. The American Psychological Association's latest expert report warns that generative AI tools pose particular risks by improving students' immediate performance without building their underlying knowledge or skills.

Why Does Engagement Feel Like Learning But Isn't?

When a student spends 30 minutes on an educational app, clicking through animations, collecting rewards, and responding to prompts, it looks like learning is happening. Teachers see focus. Parents see screen time that feels productive. But research shows this engagement tells us almost nothing about whether the student can actually apply what they encountered when they close the app.

The distinction matters enormously because it shapes how schools spend their budgets. A recent analysis of 150 education interventions across 46 countries found that more than half had no measurable learning gain at all, despite whatever engagement metrics they generated. The problem is not unique to AI; it affects educational apps, games, intelligent tutoring systems, and digital learning platforms across the board.

"AI can help a student produce a better essay, but the improved final product does not necessarily mean the student has become a better writer. Just as we need to separate engagement from learning, we will increasingly need to separate performance from learning," explained Nicole Barnes, executive lead psychologist for education at the American Psychological Association.

Nicole Barnes, Executive Lead Psychologist for Education, American Psychological Association

What Does Real Learning Actually Look Like in Cost Terms?

To cut through the noise, researchers have developed a single metric called LAYS, short for learning adjusted years of schooling. One LAYS represents a year of school in a high-performing system, measured by what a child actually learns rather than how long they sit in a classroom. When you divide the learning gain by the money spent, you get LAYS per $100, a cost per learning outcome that works across tutoring, textbooks, radio, and apps alike.

The benchmark matters: targeted instruction and structured pedagogy, where teachers work from structured lesson plans with linked materials and monitoring, each return around 3 additional LAYS per $100 on average. That is the bar that educational technology needs to clear.

When researchers applied this metric to actual EdTech programs in Kenya, the results were mixed. Digital personalized learning in pre-primary classrooms returned 4.61 LAYS per $100, while low-tech personalized learning for girls returned 1.68 LAYS per $100. Educational television, surprisingly, led at 6.77 LAYS per $100, but not because it teaches best; it leads because it costs only $1.67 per child compared to $6.88 and $10.79 for the other programs.

A randomized trial of an AI tutor in Nigeria, which assigned 657 secondary students by chance to the program or to their normal lessons, returned between 0.6 and 1.9 LAYS per $100 at a marginal cost of $9 per additional student. That performance falls below the benchmark, though researchers noted that electricity and school infrastructure costs were not included in the calculation.

How to Evaluate Whether an EdTech Tool Actually Works

• Demand Independent Evidence: Schools and districts should expect credible, independent evidence before investing in educational technology, not marketing claims or engagement metrics from the company selling the product.
• Watch What Actually Happens on Screen: Spend a minute observing the app in action. If much of the experience involves dressing a character, selecting accessories, collecting rewards, or responding to flashing images, it is reasonable to question the app's educational value.
• Ask About Durable Knowledge: Prioritize tools that build knowledge and skills students can carry into other contexts, instead of evaluating apps based on flashy features or time-on-task metrics.
• Verify Cost Transparency: Look for programs that disclose which costs were excluded and why, including device costs, connectivity, adult time, content production, and research and development amortized across all users.
• Teach AI Literacy, Don't Ban AI: Rather than preventing students from using AI tools altogether, teach them to actively evaluate AI-generated responses, question chatbots' assumptions, and identify gaps in reasoning.

The Accountability Gap That's Costing Schools Millions

The Gates Foundation and ADQ partnership announced in December 2025 that more than 93% of EdTech products in low- and middle-income countries are not tested for proof-of-learning impact. This gap is not abstract. The most recent global estimate puts 70% of 10-year-olds in low- and middle-income countries unable to read and understand a simple written text.

One reason the numbers remain so opaque is that EdTech companies rarely build their cost calculations the same way twice. Some count device costs; others do not. Some include teacher time; others treat it as invisible. Some disclose infrastructure costs; others bury them. This inconsistency makes it nearly impossible for a funder to stack one proposal against another on the same desk.

The American Psychological Association's report stresses that the primary responsibility for ensuring EdTech products are supported by evidence lies with the companies that develop them and the institutions that choose to adopt them. Developers should be prepared to demonstrate that their products improve learning. Schools and districts should demand that evidence before writing the check.

For parents, the advice is simpler: look and ask. Watch the app. Ask a teacher, principal, or school board whether there is credible, independent evidence showing that the app leads to learning. Engagement metrics alone are not enough.