Sam Altman, CEO of OpenAI, has spoken enthusiastically about transforming healthcare through AI-driven personal chatbots, but a growing chorus of experts warns that the technology powering these systems fundamentally cannot understand right from wrong. While the potential benefits of AI in medicine sound genuinely promising, researchers and security officials are raising alarms about deploying increasingly powerful systems without built-in moral frameworks or adequate safeguards.

Why Can't AI Systems Make Ethical Decisions?

The core problem is deceptively simple: artificial intelligence, no matter how sophisticated, processes information without understanding morality. One of the world's leading voices in AI research has been direct about this limitation. Yoshua Bengio, a pioneering figure in deep learning, stated that current AI systems and those foreseeable in the near future lack something fundamental.

"People need to understand that current AI and the AI we can foresee in the reasonable future does not, and will not, have a moral sense or moral understanding of what is right and what is wrong," warned Bengio.

Yoshua Bengio, AI Researcher

This distinction matters enormously when considering healthcare applications. An AI chatbot can analyze medical data, recognize patterns in symptoms, and suggest treatments with impressive accuracy. But it cannot grapple with the ethical complexities that doctors face daily: weighing patient autonomy against medical necessity, considering quality of life versus survival, or navigating cultural and religious beliefs about treatment.

What Real-World Harms Are Already Happening With AI?

The dangers are not merely theoretical. Facial recognition technology, developed ostensibly for security purposes, is already being weaponized to monitor and suppress minority populations, particularly Uyghur Muslims in China. Deepfake technology threatens democratic processes by making it impossible to distinguish authentic communications from fabricated ones.

Ken McCallum, Director General of MI5, the British intelligence agency, has warned that AI-generated impersonation poses an existential threat to truth itself. He stated that the technology could undermine the very fabric of society by making it impossible to know what is real.

"The fabric of society could be undermined by AI's impersonating real people so that it would no longer be possible to distinguish truth from falsehood. Deep fake technology is a threat to democracy and could be harnessed by hostile states to sow confusion and disinformation at the next general election," warned McCallum.

Ken McCallum, Director General of MI5

These harms are not speculative future scenarios. They are happening now, in real time, affecting real people. The scientific journal Nature argued in 2023 that the focus on distant, catastrophic AI scenarios has distracted from the immediate damage already occurring.

How to Evaluate AI Healthcare Tools Responsibly

• Demand Transparency: Healthcare providers should require clear documentation of how AI systems make recommendations, what data they use, and what limitations they have before deploying them in clinical settings.
• Maintain Human Oversight: AI chatbots should augment, not replace, human medical judgment; doctors must retain final decision-making authority and be trained to recognize when AI recommendations may be inappropriate for individual patients.
• Assess Bias and Fairness: Healthcare organizations should conduct independent audits to ensure AI systems do not perpetuate existing disparities in medical care across different demographic groups.
• Establish Clear Accountability: Organizations deploying AI in healthcare must define who is responsible when the system makes an error or causes harm, ensuring patients have recourse.
• Protect Privacy Rigorously: Healthcare AI systems should minimize data collection, encrypt sensitive information, and give patients genuine control over how their medical data is used.

What Happens When We Prioritize Convenience Over Caution?

The broader concern extends beyond healthcare. MIT physicist Max Tegmark has explored scenarios in which AI-enabled systems gradually consolidate power under the guise of public safety. In one scenario, citizens are required to wear security devices that continuously monitor location, health data, and conversations. The system could even inject lethal toxins if someone attempts to remove the device.

What makes such scenarios plausible is that they do not begin with obvious tyranny. They begin with convenience, safety promises, and protection. People voluntarily adopt technologies that make life easier, often without fully understanding the long-term implications. This gradual erosion of human autonomy may pose a greater threat than any single catastrophic AI failure.

The challenge facing Altman and other AI leaders is not whether AI can be useful in healthcare. It clearly can be. The challenge is ensuring that the deployment of these powerful systems includes robust ethical frameworks, meaningful human oversight, and genuine accountability for harms. Without these safeguards, even well-intentioned applications risk undermining the very trust that healthcare depends on.