A six-month investigation into how students actually use AI in classrooms reveals a starkly mixed picture: some AI tools outperform traditional teaching methods, while others are easily gamed or ignored by students. Instruction Partners, a Nashville-based nonprofit, observed over 100 teachers and students across 16 school systems in seven states to move past the hype and see what AI education tools genuinely accomplish in practice.

Which AI Tools Actually Help Students Learn?

The research examined 20 student-facing AI products and sorted them into three categories: general-purpose chatbots like ChatGPT and Claude; education-focused platforms such as Brisk Teaching, Google Classroom, and MagicSchool; and targeted instructional tools built for specific learning tasks, including Amira Learning, Khanmigo, and Quill.

The findings were revealing. General-purpose chatbots posed the biggest threat to student thinking, while targeted tools designed for specific instructional jobs produced the most consistent and strongest learning experiences. Education-focused platforms delivered mixed results depending entirely on how teachers used them.

"I saw tools that held a higher bar for learning than we see in paper pencil instruction and did more to tailor the questions and the feedback for kids to the specific sort of responses they were giving than a teacher could possibly do all at the same time," said Emily Freitag, co-founder and CEO of Instruction Partners.

Emily Freitag, Co-founder and CEO, Instruction Partners

Freitag, a former assistant commissioner of curriculum and instruction in Tennessee, emphasized that seeing how students actually use products matters far more than understanding what they are designed to do. In one telling example, she watched a student create multiple books without ever reading them, simply because the app allowed him to generate content quickly.

How Can Teachers Make AI Tools Actually Work in the Classroom?

• Build Clear Routines: The same product produced dramatically different results depending on whether teachers established clear routines and actively monitored student dashboards in real time, turning passive tool use into active learning.
• Use AI for Collaborative Learning: Apps that enabled students to work together and see each other's solutions side-by-side on a Smartboard shone in the research, creating significantly more learning opportunities than solo-use tools.
• Leverage AI's Unique Strengths: The most effective tools did something teachers could not do alone, such as offering instant feedback on writing or quickly analyzing an entire group's mathematical work to identify patterns.
• Avoid Permanent Skill Isolation: While AI can re-level reading passages with a single click, teachers must ensure this becomes a bridge back to grade-level material rather than a permanent lower setting that limits student growth.

In one striking example, a teacher used a math app to display four students' solutions to a problem side-by-side on a Smartboard, allowing the class to evaluate each approach. Freitag observed that students learned far more from examining each other's work this way than from solo practice.

What Problems Did Researchers Actually Observe?

The investigation uncovered serious usability issues that undermine learning. Students discovered they could hit the "Refresh" button to repeat the same question instead of completing a full lesson, or simply Google answers and paste them into assignments. In other cases, students received detailed, actionable feedback on their writing but ignored it entirely and rewrote without consulting it.

Apps designed for solo use often failed to maintain engagement. Students who spent perhaps 10 minutes engaged with a lesson were "tuned out" by minute 42, suggesting that many AI tools lack the interactive or social elements needed to sustain attention.

"My take is that the future is hybrid. Teachers have a key and lead role, and there's tech that gives them an assist," explained Freitag.

Emily Freitag, Co-founder and CEO, Instruction Partners

Freitag also raised concerns about one of AI's most popular features: the ability to automatically simplify reading passages. While this can help struggling readers access text, it risks creating a "permanent lower setting" rather than a temporary scaffold to help students reach grade-level material. Arman Jaffer, CEO of Brisk Teaching, countered that re-leveling "should be used to increase entry points to high-quality instructional material," not to limit student potential.

What Does This Mean for Schools Considering AI Tools?

The research suggests that AI in education is not a single category with uniform results. Instead, schools must evaluate tools based on how they are actually used in classrooms, not just on product marketing or design intent. One of the most promising AI innovations is its ability to identify where students struggle with foundational skills and need more practice. However, forcing isolated skill practice without connecting it to meaningful reading or knowledge can backfire.

The Instruction Partners investigation, called the AI in Action Learning Tour, is a year-long effort to observe how actual students and teachers use AI. The initial findings released this week represent only the beginning of this research, suggesting that more nuanced guidance on AI tool selection and implementation will emerge as the study continues.

For educators and school leaders, the takeaway is clear: AI tools are not inherently good or bad for learning. Their effectiveness depends on thoughtful implementation, teacher oversight, and alignment with how students actually engage with technology in real classrooms.