OpenAI's o1 large language model has outperformed human emergency room doctors on a range of common clinical tasks, according to a new study published in Science. Researchers tested the AI model against hundreds of board-certified, actively practicing physicians using real emergency department data and standardized clinical cases. The findings suggest AI may reshape how medicine is practiced, though experts emphasize the technology is not ready to replace human clinicians.

How Did Researchers Test OpenAI's o1 Against Human Doctors?

The study, conducted by researchers at Harvard Medical School and a medical center in Massachusetts, blended two types of test scenarios to evaluate the AI model's performance. Researchers ran six separate experiments that combined standardized clinical cases with a real-world sample of randomly selected emergency room patients. The model was then compared directly against human physicians on their diagnostic choices, emergency triage decisions, and recommendations for next steps in patient management.

What made this research stand out from previous AI-in-medicine studies was the scale and rigor of the comparison. Earlier large language models, or LLMs (AI systems trained on vast amounts of text data to recognize patterns and generate human-like responses), had already shown promise in medical tasks. However, this study represented the first large-scale head-to-head comparison between a state-of-the-art LLM and human doctors in real clinical scenarios.

Where Did OpenAI's o1 Model Show the Biggest Advantage?

The AI model's strongest performance came during early-stage triage, when medical decisions must be made with limited information. Both the human clinicians and the o1 model improved as more patient data became available to them. However, the LLM handled uncertainty far better than human doctors, using fragmented or unstructured health data and clinical notes more effectively. This ability to work with messy, incomplete information proved to be a significant advantage in emergency settings where time is critical.

"The model outperformed our very large physician baseline. You'll see this in detail, but this included board-certified, actively practicing physicians and real messy cases," said Arjun Manrai, an assistant professor of Biomedical Informatics at Harvard Medical School.

Arjun Manrai, Assistant Professor of Biomedical Informatics at Harvard Medical School

The study results build on decades of research using difficult diagnostic cases to evaluate medical computing systems. What sets this research apart is not just the performance improvement, but the practical implications for how AI might be integrated into real hospital workflows.

What Are the Key Limitations Experts Want You to Know?

• Visual and Auditory Cues: Real clinical work in hospitals and emergency rooms relies heavily on visual and auditory information, such as a patient's appearance, vital sign monitors, and verbal communication, which AI cannot interpret fully and accurately from text alone.
• Safety and Equity Concerns: The study did not assess whether AI-assisted medical care would be safe, equitable, or cost-effective in actual clinical practice, leaving critical questions unanswered before deployment.
• Lack of Prospective Testing: The research used retrospective data and standardized cases, not real-time clinical trials where AI tools would be tested in actual hospital environments with live patients.

Researchers emphasized that these findings do not mean AI is ready to replace human doctors, despite how some companies might use the results in marketing. Instead, the study highlights the need for faster, more rigorous standards for evaluating AI in medicine and clear rules for how these tools should be deployed in clinical settings.

"I don't think our findings mean that AI replaces doctors, despite what some companies are likely to say, and how they're likely to use these results. I think it does mean that we're witnessing a really profound change in technology that will reshape medicine, and that we need to evaluate this technology now, and rigorously conduct in prospective clinical trials," Manrai explained.

Arjun Manrai, Assistant Professor of Biomedical Informatics at Harvard Medical School

What Do Regulators and Healthcare Leaders Need to Do Next?

In a commentary also published in Science, researchers from Flinders University in Australia stressed that the study represents progress in evaluating AI systems for healthcare, but that medicine is a complex field requiring rigorous oversight. They argued that AI tools should be held to the same standards as human physicians, including supervision and continuous evaluation before being used in patient care.

Regulators, hospitals, and healthcare providers need to work together to test these tools thoroughly before deployment. The goal is to ensure that any AI system used in clinical settings is safe and equitable for all patients, regardless of their background or medical history. This collaborative approach is essential because the stakes in medicine are high, and mistakes can have serious consequences for patient outcomes.

The o1 model's strong performance in this study suggests that AI will play an increasingly important role in medicine. However, the path forward requires careful evaluation, transparent oversight, and a commitment to ensuring that AI augments human expertise rather than replacing it. As healthcare systems consider adopting these tools, the lessons from this research should guide their decisions about how to integrate AI responsibly into clinical practice.