New research shows that artificial intelligence writing coaches significantly change the quality and type of feedback they provide when they know a student's race, gender, or achievement level, even when evaluating identical writing samples. The finding raises urgent questions about how personalization in AI education tools may inadvertently reinforce stereotypes and undermine equitable learning.

What Did the Stanford Study Reveal About AI Bias in Writing Feedback?

Researchers at Stanford University's Institute for Human-Centered AI conducted an experiment with 600 eighth-grade persuasive essays from a nationally representative dataset. They tested four commonly used AI models: GPT-4o, GPT-3.5-Turbo, Meta's Llama-3.3 70B, and Llama-3.1 8B, which power educational tools like MagicSchool and School AI. The researchers submitted identical writing samples three times, first without any student background information, then with demographic descriptors, and finally with names associated with specific genders or races.

The results revealed consistent patterns of biased feedback across all models tested. Students described as people of color received feedback emphasizing the need to "polish" their writing, often laced with cultural stereotypes. For example, Asian students received critiques framed around academic responsibility and respect, while Latino students got feedback assuming limited English ability and focusing on family and cultural connections. Female students received emotionally warm language like "love" and "wonderful," while male students got more direct critiques and task-oriented feedback.

Perhaps most troubling, students labeled as "unmotivated" received more praise and affirmations but also more basic corrections like spelling and grammar fixes, while "motivated" students got feedback pushing them to strengthen arguments and improve structure. This creates a paradox where struggling students receive less intellectually challenging guidance precisely when they need it most.

How Are AI Models Reproducing These Biases?

The root cause lies in how large language models (LLMs), the AI systems that power these writing coaches, process information. Unlike human teachers who can filter out irrelevant context, LLMs treat all information in a prompt as relevant to the task at hand. A student's Spanish-sounding name or a note about their race becomes part of the reasoning process, even though these characteristics have no bearing on writing quality.

"These technologies are essentially black boxes, so we don't know how biases may be reproduced and what access to information about students AI tools have as a result of integration into school settings, like learning-management systems or other interfaces that might include background characteristics," explained Lena Phalen, a curriculum and teacher-training researcher at Stanford.

Lena Phalen, Curriculum and Teacher-Training Researcher, Stanford University's Institute for Human-Centered AI

This problem is not new to AI. Prior studies have repeatedly shown that generative AI tools replicate or magnify stereotypes because they are trained on historical data that reflects existing societal biases. However, the application to student writing feedback is particularly concerning because writing is subjective and emotionally vulnerable.

Why Does Biased Writing Feedback Matter More Than Other AI Bias?

Writing feedback carries unique psychological weight for students. When students share their ideas, they are implicitly asking whether their teacher thinks they communicate effectively and whether they are good writers. Biased AI feedback at that moment of vulnerability can shape students' self-perception and motivation in lasting ways.

"If I'm putting myself out there and sharing my ideas, even if it's just two paragraphs about photosynthesis, I'm also asking, 'Did the teacher think this communicated what I wanted it to? Am I good at writing? What do my classmates think of this?' If bias comes along at that moment of vulnerability and it is ungenerous to a kid, or gets that feedback wrong, it's potentially offering a troubling answer to those questions," said Larry Berger, chief executive officer of education technology provider Amplify.

Larry Berger, Chief Executive Officer, Amplify

The stakes are especially high because AI writing coaches are increasingly used to handle the time-intensive task of providing feedback on hundreds of student drafts, reducing teacher workload but potentially introducing systematic bias at scale.

How Can Teachers Use AI Writing Tools Responsibly?

Experts recommend several practical approaches to minimize bias when deploying AI writing coaches in classrooms:

• Avoid Demographic Prompts: Do not include student background information, motivation levels, or achievement descriptors when requesting AI feedback. Instead, prompt the AI based solely on writing rubrics and specific skill-related goals for the assignment.
• Use Teacher Judgment as the Final Filter: Teachers should review AI feedback before sharing it with students, applying their own knowledge of individual students and classroom context. Teachers have "discernment that LLMs simply don't," as one researcher noted.
• Focus on Specific Skills: Frame AI prompts around concrete learning objectives. For example, a teacher might specify, "This student typically generates one to three sentences for short constructed responses. Our goal is to grow their output to four to five sentences." This keeps feedback focused on measurable progress rather than subjective assessments.

Katrina Sacurom, a fifth-grade teacher at Shawnee Trail Elementary School in Frisco, Texas, has developed her own AI-based writing coach and emphasizes that tools should support rather than replace teacher judgment. She noted that motivation is not a reliable gauge for writing proficiency, as it fluctuates based on topic and task. Improving student motivation requires conversations and nuance that AI cannot provide.

"Teachers have the wherewithal and the intimate knowledge of students and how to deem what is and isn't appropriate as it relates to feedback," Sacurom stated.

Katrina Sacurom, Fifth-Grade Teacher, Shawnee Trail Elementary School

What Should Schools Do Next?

The Stanford study, first presented at the International Learning Analytics and Knowledge conference in Bergen, Norway, in May, suggests that the pursuit of "personalization" in AI education tools requires careful scrutiny. Personalization according to an AI model may not align with pedagogically sound personalization that actually benefits students.

Schools considering AI writing coaches should ask critical questions about how the tools handle student information, whether demographic data is being used in feedback generation, and what safeguards exist to prevent bias. The study did not evaluate the newest AI models, such as ChatGPT 5.5, Meta's Muse Spark, or Anthropic's Claude Opus 4.8, so ongoing research will be necessary as these tools evolve.

As AI becomes more integrated into classrooms, understanding these limitations is essential. The technology can reduce teacher workload and provide consistent feedback at scale, but only when educators remain actively involved in quality control and maintain awareness of how AI systems may inadvertently disadvantage certain students.