A new benchmark developed by Columbia University and the University of Cambridge is forcing artificial intelligence models to prove they understand the physics behind material properties, not just predict the right numbers. Over 100 crystalline materials were tested, revealing that some AI models achieved similar accuracy scores while fundamentally misunderstanding how atoms actually behave. This distinction matters enormously for industries relying on AI to discover new materials.

Why Do AI Models Get the Right Answer for the Wrong Reasons?

Machine learning models trained to predict material properties often focus on a single metric: energy prediction. This narrow optimization can mask critical errors in how the model simulates atomic behavior. When researchers assessed models on their ability to predict thermal conductivity, a property that depends heavily on how atoms vibrate at different temperatures, they discovered a troubling pattern.

Some models predicted both atomic vibrations and thermal conductivity accurately, while others failed at the vibration prediction despite achieving similar scores on formation energy and material stability. The problem is that errors in force calculations, which determine how atoms respond to temperature or mechanical stress, can remain hidden when models focus solely on energy prediction.

"There are cases in which ML models give apparently sensible predictions, but for the wrong reasons," explained Balázs Póta, the graduate student who led the research.

Balázs Póta, Graduate Student, University of Cambridge

This distinction between correct answers and correct reasoning has real consequences. As robotic laboratories increasingly automate material discovery, a model that misrepresents a material's underlying physics risks discarding promising candidates and promoting unsuitable ones.

What Makes a Model "Physics-Aware"?

Michele Simoncelli, assistant professor of applied physics at Columbia, defines the gold standard for AI in materials science. A truly physics-aware model must predict macroscopic properties as a consequence of correctly describing the materials' atomistic physics, namely their atomic vibrations.

"We can call an atomistic ML model 'physics-aware' when it predicts the macroscopic properties of materials as a consequence of correctly describing the materials' atomistic physics," said Michele Simoncelli.

Michele Simoncelli, Assistant Professor of Applied Physics, Columbia University

The research team, which included collaborators from the University of Cambridge and other institutions, refined their models with material-specific training and achieved agreement within a few percent of reference quantum calculations. Notably, they also achieved agreement with experimental measurements for lithium bromide, demonstrating that physics-aware modeling can bridge the gap between theory and real-world behavior.

How to Evaluate AI Models for Materials Science

• Thermal Conductivity Testing: Assess whether models accurately predict how heat moves through a material by simulating atomic vibrations, not just energy levels.
• Force Calculation Accuracy: Verify that models correctly compute forces between atoms, which determine how materials respond to temperature and mechanical stress.
• Multi-Property Validation: Test models across multiple material properties to ensure consistent physics-aware reasoning, not just accuracy on a single metric.
• Comparison to Quantum Calculations: Benchmark AI predictions against traditional quantum-mechanical calculations to identify discrepancies in microscopic simulations.

The team's work has been incorporated into Matbench Discovery, an interactive leaderboard that ranks machine learning models not only on their ability to predict crystal stability and structure, but also on thermal conductivity predictions. Janosh Riebesell, creator and maintainer of Matbench Discovery, noted that the tool quickly exposed performance discrepancies among models that appeared identical on traditional metrics.

Who Is Already Using This Benchmark?

The rapid adoption of this physics-aware benchmark signals growing industry recognition that accuracy alone is insufficient. Meta, Microsoft, Radical AI, and Orbital Materials have already begun using the benchmark to evaluate their machine learning models. This widespread adoption suggests that companies developing AI for materials discovery understand the stakes: a model that arrives at correct answers through flawed physics could lead to wasted research and development resources.

"Our benchmark provides a 'physics-aware' signal to ML developers to optimize their models," stated Balázs Póta.

Balázs Póta, Graduate Student, University of Cambridge

Michele Simoncelli noted that while thermal conductivity is especially sensitive to microscopic vibrational physics, the general principle applies more broadly. "Thermal conductivity is especially sensitive to the microscopic vibrational physics of a material, but the general idea is relevant to other physical properties as well, a topic which we are currently exploring in the group," he explained.

Michele Simoncelli

The research underscores a fundamental shift in how the AI and materials science communities evaluate machine learning models. Rather than accepting models that simply produce correct predictions, the field is moving toward requiring models that demonstrate they understand the underlying physics. This distinction could accelerate the discovery of new materials for applications ranging from batteries and semiconductors to thermal management systems, ensuring that AI-driven materials science builds on a foundation of genuine physical understanding rather than statistical correlation.