Artificial intelligence systems used in healthcare often fail to acknowledge when they don't know something, creating dangerous blind spots in pharmacy and drug management. A pharmacist-turned-AI researcher has uncovered a critical vulnerability: large language models (LLMs), the AI systems powering many healthcare applications, cannot reliably distinguish real medications from fabricated ones, and they rarely admit uncertainty about drugs they encounter.

What Happens When AI Encounters a Drug It Doesn't Recognize?

Andrea Sikora, an associate professor of biomedical informatics at the University of Colorado Anschutz, began her career as a clinical pharmacist specializing in critical care. Her transition into AI research started unexpectedly when a mentor asked a simple question: why wasn't she pursuing her passion project on medication management as her main focus? That question, combined with an encounter at a professional conference, led her to explore how machine learning could solve real problems in pharmacy.

"I basically said, 'What's that?'" Sikora recalled when first asked about machine learning. But as she learned more about AI systems, she realized they could help address the complexity of managing medications in intensive care units, where patients often take multiple drugs with intricate interactions and dosing requirements.

The challenge became clear when Sikora and her team tested how LLMs handle drug data. In a study titled "Drug or Pokémon? Large language model performance in identification of fabricated medications," researchers compiled lists of real medications alongside fictitious drugs disguised as Pokémon character names, complete with plausible doses and administration routes. The results were alarming: confabulations, or false statements where the AI either missed the fake drug entirely or automatically converted the Pokémon name into a real medication, ranged from 2.7% to 99.6% depending on the model tested.

Why Is Drug Data So Challenging for AI Systems?

Drug information presents unique obstacles that general-purpose AI systems struggle to navigate. The data is what researchers call "high dimensional," meaning a single medication involves multiple interconnected details that all matter for safety.

• Complex dosing information: A simple statement like "I took ibuprofen this morning" requires AI to understand strength, tablet count, frequency, and patient medical history including cardiovascular and kidney function.
• Unusual syntax patterns: Medications use alphanumeric combinations in specific orders, such as "aspirin 81 milligrams," which creates syntax patterns unfamiliar to AI trained on general language.
• Constantly evolving drug names: New medications launch regularly, and most drugs have multiple names due to brand variations, making it impossible for AI to have seen every drug in its training data.

When a trained pharmacist encounters an unfamiliar drug, they follow a formalized process: look it up, research its uses and dosing, and crucially, acknowledge the gap in their knowledge. "If I saw a drug that I didn't recognize, I wouldn't just assume it was a drug and that the patient was on that drug," Sikora explained. "I would make it clear that I didn't know what this drug was, and I would ask questions to clarify".

AI systems, by contrast, do not naturally default to this kind of epistemic humility. A recent perspective published in the New England Journal of Medicine, co-authored by Sikora and researchers from Harvard T.H. Chan School of Public Health and Harvard Medical School, explored this gap. The team defined epistemic humility as "a human virtue that involves metacognitive awareness, a moral commitment to truthfulness, and recognition of the limits of one's knowledge".

"Drugs can kill you. Medication errors are already a leading cause of death in the United States. Drugs are a deeply unforgiving thing to make a mistake about," said Sikora. "Maybe LLMs are going to make us better, and that certainly would be exciting. On the other side, you could say we have now introduced an entirely new way to mess things up. So that's worthy of intrigue or concern."

Andrea Sikora, Associate Professor of Biomedical Informatics at University of Colorado Anschutz

How Can Researchers Improve AI Safety in Pharmacy?

Sikora's research has identified several practical methods to help AI systems work more safely with drug data. These approaches range from better prompting techniques to specialized benchmarking tools designed specifically for pharmaceutical applications.

• Mitigation prompts: Carefully worded instructions given to AI systems can reduce confabulations, though the Pokémon study showed these are not foolproof across all models.
• Specialized benchmark testing: Tools like the Rx-LLM Benchmark Suite, developed by Sikora's team, create standardized tests to evaluate how well LLMs perform with real-world drug data before deployment in clinical settings.
• Transparency about uncertainty: Implementing systems that require AI to flag when it encounters unfamiliar medications or conflicting information, rather than generating confident-sounding but potentially false answers.

The stakes in healthcare are extraordinarily high. Medication errors are already among the leading causes of preventable death in the United States, making the integration of AI into pharmacy workflows a matter that demands rigorous safety evaluation before widespread adoption.

Sikora emphasizes that researchers and clinicians must implement benchmarks to evaluate the efficacy and safety of LLMs in healthcare settings. The Pokémon study serves as a proof of concept: by testing AI systems against fabricated medications alongside real ones, researchers can identify which models are prone to confabulation and which prompting strategies help reduce errors.

As AI becomes increasingly integrated into healthcare, from clinical summaries to translation services for non-English speakers, the question of whether these systems can admit uncertainty becomes not just academic, but a matter of patient safety. Sikora's work suggests that before AI takes on a larger role in pharmacy and drug management, the field must solve a fundamental problem: teaching machines to say "I don't know."