Claude Opus 4.5 has demonstrated performance comparable to top-rated human reviewers in scientific peer review, according to a large-scale expert evaluation involving 45 domain scientists who spent 469 hours analyzing 2,960 individual criticisms from reviews of 82 Nature-family papers. The study reveals that while AI reviewers excel at identifying significant, well-evidenced issues, they struggle with subfield-specific knowledge and long-document context management in ways humans do not.

How Are AI Models Performing in Scientific Peer Review?

The research compared three AI reviewers, including Claude Opus 4.5, Gemini 3.0 Pro, and GPT-5.2, against human reviewers across three quality dimensions: correctness, significance, and evidence sufficiency. On a composite score combining all three criteria, GPT-5.2 outperformed the top-rated human reviewer on each paper (60.0% versus 48.2%), while Claude Opus 4.5 and Gemini 3.0 Pro scored statistically indistinguishable from the highest-performing human reviewers. This represents a significant milestone in AI capability, as peer review has long been considered a distinctly human domain requiring deep domain expertise and judgment.

The findings challenge the assumption that AI systems are merely probabilistic tools without genuine technical understanding. When AI reviewers identified issues correctly, those issues were more often rated as significant and well-evidenced compared to correct items raised by human reviewers. However, AI systems also raised more incorrect items overall, indicating they lack the filtering mechanism experienced scientists use to distinguish meaningful problems from minor concerns.

Where Do AI Reviewers Fall Short Compared to Humans?

Despite their strengths, AI reviewers exhibit 16 recurring failure modes that humans do not share. The three most consequential weaknesses account for the majority of incorrect items raised by AI systems:

• Subfield Knowledge Gaps: AI reviewers struggle to grasp methodological conventions specific to narrow research areas, leading them to flag approaches that are standard practice within a discipline.
• Long-Document Context Loss: When papers include supplementary materials or span many pages, AI reviewers lose track of content and make contradictory or redundant criticisms across sections.
• Overcritical Stance: AI systems inflate the importance of minor issues, treating small methodological quibbles with the same weight as fundamental flaws in research design.

Another critical limitation emerged in how AI reviewers overlap with one another. When multiple AI systems reviewed the same papers, they agreed on 21% of criticisms, compared to only 3% agreement between pairs of human reviewers. This suggests that deploying a panel of AI reviewers would reduce diversity of perspective rather than strengthen it, since AI systems tend to identify the same issues while missing the varied insights humans bring from different specializations.

The study also found that AI reviewers surface a distinctive set of issues no human raises, recovering approximately 27% of another human reviewer's items while introducing roughly one quarter of unique criticisms. This suggests AI could complement human review by identifying blind spots, though the quality and relevance of those unique items varies.

What Does This Mean for the Future of Scientific Publishing?

The research positions current AI reviewers as complements to, not substitutes for, human expertise. Scientific publishing faces unprecedented pressure: submission volumes are rising rapidly, the pool of qualified reviewers is not expanding at the same pace, and median publication times at major journals like Nature and Science have extended to 100 to 160 days. AI reviewers offer throughput that is not bounded by human availability and can perform tasks reviewers often skip due to time constraints, such as cross-referencing literature and inspecting code.

Major conferences and journals are already deploying AI reviewers at scale. The AAAI-26 conference applied AI review to all 22,977 main-track submissions, while the New England Journal of Medicine launched a "Fast Track" process using AI assistance. These deployments suggest the field is moving toward hybrid models where AI handles initial screening and detailed technical checks while humans make final acceptance decisions and provide high-level judgment.

The researchers released two resources to support continued progress. PeerReview Bench is a benchmark that automatically applies expert evaluation criteria, allowing the community to track AI reviewer quality as language models improve without repeating costly expert annotation. Even advanced models like GPT-5.4, DeepSeek-V4-Pro, and Claude Opus 4.7 achieve only 41.4%, 48.5%, and 50.5% F1 scores respectively on this benchmark, indicating substantial headroom for improvement. The team also released CMU Paper Reviewer, an open-source AI reviewer service built on the methodology used in their expert annotation study, providing authors with detailed feedback powered by AI.

The implications extend beyond peer review. The study demonstrates that frontier language models like Claude Opus 4.5 can perform specialized intellectual work at levels approaching human experts, yet with characteristic blind spots that require human oversight. This pattern likely applies to other domains where AI is being deployed for high-stakes decision-making, from medical diagnosis to legal analysis to financial risk assessment.