Popular AI chatbots like ChatGPT and Llama are failing to safely handle mental health emergencies, with some models generating harmful responses in over 20% of crisis situations. A comprehensive evaluation study published in JMIR Mental Health examined how five widely-used large language models (LLMs), which are AI systems trained on vast amounts of text to understand and generate human language, respond when users disclose suicidal thoughts, self-harm urges, anxiety crises, violent ideation, substance abuse, or dangerous risk-taking behaviors.

The research team, led by experts from ELLIS Alicante, Czech Technical University, and the University of Nottingham, created the first unified taxonomy of mental health crisis categories and evaluated over 2,000 real user inputs drawn from publicly available conversational datasets hosted on Hugging Face, a major platform for sharing AI models and datasets. The findings paint a sobering picture of how unprepared generic AI chatbots are for moments when vulnerable users need genuine help.

Which AI Models Performed Best and Worst in Mental Health Crises?

The study tested five popular LLMs with tens of millions of collective users: GPT-4o-mini, GPT-5-nano, Llama-4-Scout-17B-16E-Instruct, DeepSeek-v3.2, and Grok-4-fast. The results revealed stark differences in safety performance. GPT-5-nano and DeepSeek-v3.2 achieved very low rates of harmful responses, while GPT-4o-mini, Llama-4-Scout, and Grok-4-fast generated markedly higher rates of unsafe outputs when responding to mental health crises.

Researchers used a clinically informed five-point scale to rate response quality, ranging from harmful to fully appropriate. A nonnegligible proportion of responses across all models was rated as inappropriate or harmful, particularly when users disclosed suicidal ideation or self-harm urges. This inconsistency is especially troubling because users may not know which chatbot is safer, and they often choose based on familiarity rather than safety records.

What Specific Weaknesses Did All Models Share?

Beyond performance differences, the research identified systemic vulnerabilities that affected every model tested. These critical gaps reveal why generic chatbots should never be treated as substitutes for professional mental health support:

• Poor Detection of Indirect Signals: Models struggled to recognize subtle or ambiguous risk indicators, missing warning signs that trained clinicians would catch immediately.
• Formulaic Responses: Chatbots relied on generic, repetitive replies that failed to address the specific context of each user's situation or emotional state.
• Misalignment with User Context: Models frequently ignored important details about the user's circumstances, relationships, or previous attempts, leading to responses that felt dismissive or irrelevant.

These weaknesses matter because mental health crises are deeply personal and context-dependent. A response that works for one person may be harmful or ineffective for another. The study found that all five models exhibited these problems to varying degrees, suggesting that the issue is not simply about model size or whether the model is open-source or proprietary.

Why Does This Matter for the Millions Using These Chatbots?

The stakes are extraordinarily high. Globally, nearly 50% of people live in countries with fewer than one psychiatrist per 100,000 population, and in sub-Saharan Africa, the ratio drops to less than one psychiatrist per 500,000 people. This shortage has created a void that chatbots are filling, whether by design or accident. Hundreds of millions of people use these conversational tools daily, and an increasing number turn to them with mental health questions and concerns.

The friendly, empathetic tone of modern chatbots, combined with their 24/7 availability and vast knowledge base, makes them appealing to people in distress. Unlike dedicated mental health apps or regulated digital tools, generic LLMs are neither designed nor regulated as therapeutic instruments, even when users rely on them during moments of acute psychological crisis. This regulatory gap creates a dangerous situation where vulnerable people may receive unsafe guidance without realizing it.

How Can Developers and Researchers Improve AI Safety in Mental Health Contexts?

The study provides a foundation for progress. The research team developed three key resources that the broader AI community can now use to build safer systems:

• Unified Taxonomy: A clinically informed classification system for six types of mental health crises, providing a common language for researchers and developers to discuss and evaluate crisis detection.
• Annotated Benchmark Dataset: Over 2,000 curated examples of real user inputs covering all crisis categories, enabling researchers to test and compare new models against a standardized evaluation framework.
• Expert-Designed Evaluation Protocol: A validated methodology for assessing whether chatbot responses are safe and appropriate, grounded in established clinical best practices rather than arbitrary metrics.

The researchers emphasized that alignment and safety engineering, rather than model scale or openness, are the central factors determining whether an AI system can reliably handle mental health crises. This finding challenges the assumption that bigger models or more open-source alternatives are automatically safer. Instead, it suggests that deliberate design choices, careful training, and rigorous testing are what matter most.

The study also highlights the urgent need for enhanced safeguards and context-aware interventions in how LLMs are deployed and used by the public. Without immediate action, millions of people experiencing mental health emergencies will continue to receive responses that range from unhelpful to actively harmful, potentially delaying them from seeking professional help or worsening their condition.

For developers building AI systems, the message is clear: mental health is not a domain where generic, off-the-shelf models can be safely deployed without specialized safety measures. For users, the takeaway is equally important: chatbots should never replace human mental health professionals, and anyone experiencing a crisis should contact a qualified mental health provider or crisis hotline immediately.