A new benchmark published this week reveals that leading voice AI models, including ElevenLabs and Google's Gemini, struggle significantly with newly approved drug names, mispronouncing up to one in three when read aloud in clinical contexts. The DOSE (Drug-name Oral Synthesis Evaluation) benchmark, released by Synthio Labs, tested nine commercial text-to-speech systems on 274 medication names, including 146 recently approved drugs, each embedded in realistic clinical sentences.

Why Does Voice AI Fail on New Drug Names?

The gap between how well voice AI handles familiar medications and newly approved ones reveals a fundamental limitation in how these models learn language. ElevenLabs' eleven_v3 model, which passed 93% of established drug names, dropped to 67.1% accuracy on newly approved ones, a decline of nearly 26 percentage points. Google's Gemini TTS performed worse, falling from 89.1% on established names to just 61.6% on new drugs, a gap of nearly 28 percentage points.

The reason is straightforward: older medications have had years or decades to appear in training data, pharmacy databases, closed captions, and everyday speech. New drug names, engineered by pharmaceutical branding teams to sound distinct from existing medications, have no such history. When a voice AI model encounters a name it has never seen before, it must guess the pronunciation based on spelling alone, a task that often fails for pharmaceutical nomenclature.

"Voice models are sold on how human they sound. A model can sound human and still mispronounce the drug name, and in pharma that is the failure that matters," said Rajashekar Vasantha, Co-founder and CTO of Synthio Labs.

Rajashekar Vasantha, Co-founder and CTO, Synthio Labs

Where Does Voice AI Mispronunciation Matter Most?

Voice AI has moved into healthcare settings where accuracy directly affects patient safety. Pharmacy chains deploy voice bots to confirm prescription refills. Telehealth platforms use text-to-speech to read back prescribing information. Medication-reminder apps read drug names aloud to patients, many of whom are elderly, visually impaired, or managing multiple prescriptions simultaneously. In these contexts, a mispronounced drug name is not a minor inconvenience; it is a potential patient safety issue that regulatory bodies like the FDA and the World Health Organization already track as a category of medication error.

The DOSE benchmark is the first public, comparative measurement of this specific failure mode. Before now, the problem existed mostly as anecdotal complaints from pharmacists or scattered app-store reviews, with no standardized way to measure how often it occurred across different voice AI systems.

How to Evaluate Voice AI for Healthcare Applications

• Test on newly approved drugs: Ask vendors to demonstrate performance on medications approved within the last 12 months, not just legacy drugs that have been in circulation for years.
• Require clinical-context testing: Evaluate how the model performs when drug names are embedded in realistic sentences and medical language, not read in isolation as single words.
• Check for generic name accuracy: Verify performance on International Nonproprietary Names (generic drug names), where Microsoft Azure's system passed fewer than half of tested names, according to the benchmark.
• Demand transparency on training data: Ask when the model's training data was last updated and whether it includes recent pharmaceutical nomenclature and clinical language.

The DOSE benchmark tested each model by scoring pronunciations on a scale of 0 to 5, with a score of 4 or higher counting as a pass. Across all nine systems tested, general-purpose voice AI models passed between 63.1% and 80.3% of drug names overall, but those numbers mask the sharp drop on newly approved medications.

Synthio Labs' own RxPronounce model, designed specifically for pharmaceutical use, performed better than general-purpose systems, passing 91.2% of all drug names and 87% of newly approved ones, roughly 11 percentage points ahead of the next-best system. The company released the full dataset with audio samples on Hugging Face, allowing other researchers and vendors to examine the results in detail.

"The pattern tracks where training data runs out. Every model handles metformin, a legacy diabetes drug, but they struggle with the molecule approved last quarter, exactly the name a launch team or a patient most needs spoken correctly," explained Supreet Deshpande, Co-founder and CEO of Synthio Labs.

Supreet Deshpande, Co-founder and CEO, Synthio Labs

The benchmark also surfaced a particularly striking failure: one system spelled out the drug name Xofluza letter by letter rather than attempting to pronounce it, a fallback behavior that suggests the model had no confidence in its phonetic guess. Microsoft Azure's text-to-speech system showed a different vulnerability, passing fewer than half of generic drug names, the formal International Nonproprietary Names that pharmacists and healthcare providers use routinely.

As of now, none of the tested vendors have issued public responses to the DOSE results, and Synthio Labs' release did not include rebuttals or comments from any of the companies whose models were evaluated. The absence of vendor pushback leaves the benchmark's findings largely uncontested in the public record, even as the AI industry continues to ship new voice and language models at a rapid pace.

For healthcare organizations considering voice AI for patient-facing applications, the DOSE benchmark provides the first standardized measurement of a real-world failure mode that previous benchmarks did not address. The results suggest that general-purpose voice AI models, no matter how natural they sound, may not be suitable for medication-related applications without additional fine-tuning or domain-specific alternatives.