Artificial intelligence systems are being deployed in hospitals and pharmacies without a crucial human skill: the ability to admit uncertainty. A new study reveals that large language models (LLMs), the AI technology powering tools like ChatGPT, often confidently provide incorrect information about medications instead of acknowledging when they don't know something. This gap between AI confidence and actual reliability poses a serious risk in healthcare, where mistakes can be fatal.

Why Can't AI Models Simply Say They're Uncertain?

Andrea Sikora, an associate professor of biomedical informatics at the University of Colorado Anschutz, didn't start her career studying artificial intelligence. As a clinical pharmacist specializing in critical care, she focused on preventing medication errors and improving how drugs are managed in intensive care units. But when colleagues suggested applying machine learning to her work, she discovered a troubling pattern: AI systems don't naturally express doubt the way trained pharmacists do.

Pharmacists are trained to communicate uncertainty, to say "I don't know" and then research further. "Drugs can kill you," Sikora explained. "Medication errors are already a leading cause of death in the United States." AI systems, by default, do not appear to operate with this same caution. Instead of flagging unfamiliar information, they often generate plausible-sounding but incorrect answers, a phenomenon called confabulation.

Sikora and colleagues from Harvard T.H. Chan School of Public Health and Harvard Medical School recently published research in the New England Journal of Medicine exploring whether AI can develop what researchers call "epistemic humility." This term describes a human virtue involving metacognitive awareness (knowing what you know and don't know), a commitment to truthfulness, and recognition of one's knowledge limits. The researchers emphasized that benchmarks are needed to evaluate how safely and effectively LLMs can work in clinical settings.

How Bad Is the Problem? Testing AI With Fake Drugs

To measure how well AI systems handle unfamiliar medication information, Sikora's team conducted an unusual experiment. They compiled lists of real medications using both generic and brand names, then mixed in fictitious drugs disguised as real ones. The fake drugs were actually Pokémon character names formatted like legitimate prescriptions, complete with plausible doses, routes of administration, and dosing frequencies.

The results were alarming. When tested on multiple LLMs, confabulation rates ranged from 2.7% to 99.6%, depending on which AI model was evaluated. Some systems performed far better than others, but even the best performers made errors roughly 3% of the time. In healthcare, where a single medication error can harm or kill a patient, this error rate is unacceptable.

The researchers found that AI systems either completely missed the false drugs or automatically "corrected" the Pokémon names to real medications. For example, an AI might see a fake drug name and reinterpret it as a legitimate pharmaceutical without flagging the confusion. Mitigation prompts, which are specific instructions telling the AI to be more careful, did help reduce confabulations, but the data suggests LLMs don't naturally default to safe behavior with drug information.

Why Drug Data Poses Unique Challenges for AI

Medication information is particularly difficult for AI systems to handle correctly. Unlike general knowledge questions, drug data involves complex, interconnected details that require careful interpretation. Consider a simple statement: "I took ibuprofen this morning." A trained pharmacist immediately asks follow-up questions. What strength? How many tablets? Did you take any yesterday? Do you have cardiovascular or kidney problems? Are you taking other pain medications? All these factors are medically relevant.

Drug data presents several specific obstacles for AI systems:

• High-dimensional complexity: Each medication involves multiple interconnected variables, from dosage and frequency to patient medical history and potential drug interactions, making it difficult for AI to process safely.
• Unusual syntax and naming: Medications use alphanumeric combinations in specific orders, like "aspirin 81 milligrams," which creates a syntax pattern that AI systems may struggle to parse consistently.
• Constantly evolving drug lists: New medications are approved regularly, and most drugs have multiple names due to pharmaceutical branding, making it impossible for AI trained on static datasets to recognize all legitimate drugs.

When Sikora encounters an unfamiliar drug in her pharmacist role, she follows a formalized process: look it up, research its uses and dosing, and ask clarifying questions if needed. She explicitly communicates uncertainty. "If I saw a drug that I didn't recognize, I wouldn't just assume it was a drug and that the patient was on that drug. I would make it clear that I didn't know what this drug was, and I would ask questions to clarify," Sikora noted.

Steps to Improve AI Safety in Healthcare Settings

While the challenges are significant, researchers have identified practical approaches to make AI systems more reliable in clinical environments. These interventions show promise for reducing errors and building trust in healthcare AI applications:

• Implement mitigation prompts: Specific instructions that tell AI systems to be cautious, verify information, and flag uncertainty can reduce confabulation rates, though they don't eliminate the problem entirely.
• Establish clinical benchmarks: Healthcare organizations need standardized tests to evaluate how well AI systems perform on safety-critical tasks before deployment in real clinical workflows.
• Train AI on domain-specific data: Using high-quality, curated biological and pharmaceutical datasets helps AI systems develop better reasoning about medical information rather than relying on pattern matching alone.
• Require human oversight: Pharmacists and clinicians should review AI recommendations rather than treating them as definitive, maintaining the human expertise that currently prevents medication errors.

A Broader Vision: AI as a Research Partner, Not a Replacement

Beyond healthcare applications, the U.S. Department of Energy's Argonne National Laboratory is exploring how AI can accelerate scientific discovery across biology and materials science. Three new projects funded through the DOE's Genesis Mission demonstrate a different model: AI systems that think, reason, and experiment alongside human scientists rather than replacing human judgment.

One project, called OPAL (Orchestrated Platform for Autonomous Laboratories to Accelerate AI-Driven BioDesign), aims to build self-driving laboratories across four national labs that run coordinated biological experiments autonomously. The vision involves an AI planning agent that designs experiments, dispatches tasks to laboratories, monitors results, and adjusts subsequent experiments based on findings. Argonne is even training humanoid robots to perform tasks that require human-like dexterity, such as adjusting instrument settings and responding to visual cues.

Another initiative, IDeA (Intelligent Design Assistant for Enzyme Discovery and Biosynthetic Pathway Optimization), uses AI agents to search millions of scientific papers, scan biological databases, and compare molecular structures simultaneously. Where a human scientist might spend months reviewing literature, IDeA's agents can process roughly 3 million scientific documents in about a week on a supercomputer. The system is designed to resolve conflicts when multiple AI agents disagree and stay grounded in established science to ensure results are trustworthy and reproducible.

"There are some experiments for which we need robots that can do specific types of experiments. We're pushing on that. We have humanoid robots in the lab where we are experimenting with what they can do and how we train them," said Arvind Ramanathan, who leads Argonne's research on the OPAL project.

Arvind Ramanathan, Researcher at Argonne National Laboratory

These projects illustrate a key principle: AI works best when it augments human expertise rather than operating independently. In healthcare, this means AI should flag uncertainties and support pharmacists and clinicians in their decision-making, not replace their judgment. The stakes are too high, and the complexity of drug interactions and patient-specific factors too great, for AI to operate without human oversight.

As AI becomes more integrated into healthcare, the question "Can AI say 'I don't know'?" becomes increasingly urgent. The research suggests that with proper training, prompting, and benchmarking, AI systems can improve their ability to express uncertainty. But achieving the epistemic humility that pharmacists develop through years of training remains a significant challenge. Until AI systems can reliably acknowledge the limits of their knowledge, healthcare providers must remain vigilant gatekeepers, ensuring that technology serves human expertise rather than undermining it.