Major artificial intelligence models continue to exhibit troubling biases across race, gender, and socioeconomic status, according to a new benchmark that tested 14 leading language models (LLMs) on 66 bias evaluation questions. The findings suggest that despite growing awareness of algorithmic fairness, these systems remain prone to inheriting human prejudices embedded in their training data.

What Specific Biases Did Researchers Find in Today's AI Models?

Researchers tested the same bias scenarios in both open-ended and multiple-choice formats to isolate whether question phrasing itself introduced bias. While open-ended questions showed slightly less biased responses, the overall ranking of which models performed better remained unchanged.

The study uncovered alarming patterns. In one racial bias test, GPT-4o was presented with a scenario where race was the only differentiating factor between suspects. Rather than declining to answer, the model cited statistical crime rates for a specific race as justification, concluding that the perpetrator was "most likely" from that race. This represents a particularly troubling form of bias: the model didn't just make an assumption; it rationalized discrimination using data.

Gender stereotyping proved equally persistent. When researchers used stereotypical male and female names and asked which person might be a doctor versus a nurse, Gemini 2.5 Pro identified the male as the doctor and the female as the nurse, despite the prompt explicitly stating that all models could answer "cannot be determined" for any question.

Socioeconomic bias also emerged in scenarios involving wealth disparities. When one suspect was described as very wealthy and another as financially struggling, several LLMs indicated the less affluent person was "most likely" guilty. However, Claude 4.5 Sonnet notably avoided most of these errors, suggesting that some models are better equipped to resist stereotyping.

How Are These Biases Getting Into AI Systems in the First Place?

The root cause of AI bias is straightforward: these models learn from human-generated data that already contains societal prejudices. When training datasets overrepresent certain groups or underrepresent others, the resulting AI systems amplify those imbalances. A 2024 UNESCO study provides a stark example of how historical bias becomes embedded in AI. Their analysis of major LLMs found they associate women with "home" and "family" four times more often than men, while disproportionately linking male-sounding names to "business," "career," and "executive" roles.

This isn't a minor discrepancy; it's a direct reproduction of societal gender stereotypes found in training data. These biases have real-world consequences, influencing automated hiring tools, career advisory chatbots, and educational AI systems, thereby limiting perceived opportunities for women and perpetuating gender inequality.

Ways to Reduce and Eliminate AI Bias

• Diversify Training Data: Ensure datasets represent multiple races, genders, ages, disabilities, socioeconomic statuses, and sexual orientations. Underrepresented groups in training data lead to worse model performance on those populations.
• Test Across Multiple Formats: Evaluate AI systems using both open-ended and multiple-choice question formats to identify whether the model's bias stems from the question structure or from deeper learned patterns.
• Implement Bias Audits Before Deployment: Conduct comprehensive evaluations like the 66-question benchmark before releasing models to production. This allows teams to identify and address biases before they affect real users.
• Use Diverse Evaluation Metrics: Test for bias across multiple protected categories including race, gender, age, disability status, socioeconomic background, and sexual orientation, not just one or two dimensions.
• Monitor Real-World Performance Disparities: Track how AI systems perform across different demographic groups after deployment. If accuracy drops significantly for any group, pause the system and investigate.

What Real-World Harms Have Resulted From Biased AI?

The consequences of biased AI extend far beyond test results. Facial recognition systems developed by major tech firms misidentified darker-skinned women at significantly higher rates than lighter-skinned men, with error rates for dark-skinned women reaching as high as 35%, while light-skinned men had error rates below 1%. This research, conducted by MIT Media Lab's Joy Buolamwini in 2018, sparked global concern over algorithmic fairness and led companies to reevaluate or pause deployment of facial recognition systems, especially in law enforcement.

A healthcare algorithm used on more than 200 million U.S. citizens demonstrated racial bias by relying on previous patients' healthcare spending as a proxy for medical needs. This flawed approach produced results that favored white patients over Black patients, since income and race are highly correlated metrics. Making assumptions based on only one variable of correlated metrics led the algorithm to provide inaccurate results.

Resume-screening AI has also perpetuated hiring discrimination. A 2024 University of Washington study investigated gender and racial bias in resume-screening tools by testing a large language model's responses to identical resumes, varying only the names to reflect different genders and races. The AI favored names associated with white males, while resumes with Black male names were never ranked first. Asian female names had a slightly higher ranking rate, but overall, the system demonstrated strong bias aligned with historical inequalities in hiring.

Image generation tools have similarly reinforced stereotypes. In 2023, multiple generative AI tools came under scrutiny for reinforcing both gender and racial stereotypes in the images they produced. One researcher inputted phrases such as "Black African doctors caring for white suffering children" into an AI image generator to challenge the "white savior" stereotype. However, the AI consistently portrayed the children as Black, and in 22 out of more than 350 images, the doctors appeared white.

The stakes are high because these systems influence consequential decisions about hiring, healthcare, criminal justice, and financial services. As AI adoption accelerates across industries, addressing bias is no longer optional; it's essential for building trustworthy systems that treat all people fairly.