OpenAI recently announced that its AI systems solved ten major unsolved problems in mathematics and theoretical computer science, including the first construction of a non-sofic group and a proof about Ramsey numbers. Yet despite these landmark achievements, large language models (LLMs), the AI systems that power ChatGPT and similar tools, are not uniformly better than humans at all mathematical tasks. Understanding where these models excel and where they still fall short offers insight into the current state of AI reasoning.

What Types of Math Problems Can LLMs Actually Solve?

The ten problems OpenAI's systems solved represent extraordinary progress, but the pattern of their successes reveals something interesting about how LLMs approach mathematics. Most of the headline-grabbing solutions involved finding counterexamples, mathematical objects that disprove a conjecture, rather than constructing formal proofs. The non-sofic group construction and the Ramsey number result both fall into this category. This raises a natural question: are LLMs particularly good at finding counterexamples, or is this pattern merely coincidental ?

The answer is more nuanced than it first appears. Defining what counts as "finding a counterexample" is trickier than it sounds. Consider a famous result by mathematician Vinogradov, which states that every sufficiently large positive integer can be expressed as a sum of three prime numbers. Technically, proving this theorem involved showing that certain integers do not have a particular property, yet we would never call this "finding a counterexample." The distinction matters because it helps clarify what LLMs are actually good at versus what we might assume they are good at.

Where Do Large Language Models Still Struggle?

Despite solving ten major problems, LLMs have not yet achieved human-level performance across all mathematical domains. If they had, the speed advantage of AI systems over human mathematicians would likely produce a flood of new results far exceeding what we currently see. The fact that we do not observe this suggests meaningful gaps remain in how LLMs approach certain types of mathematical reasoning.

One area where humans still maintain an edge involves problems with complex quantifier structures, where multiple layers of logical conditions must be carefully balanced. Many important mathematical results, when stated formally, begin with alternating sequences of quantifiers (statements like "for all X, there exists Y such that..."). Determining which quantified variable is the "interesting" one, and where to focus computational effort, remains a distinctly human strength. LLMs can find proofs and construct arguments, but they do not yet consistently outperform human mathematicians at navigating these layered logical structures.

How to Evaluate AI Math Capabilities Responsibly

• Examine the Evidence Base: When AI systems claim to solve problems, scrutinize whether the solution rests on a single case study or multiple validated trials. A single successful example, even when impressive, does not establish a general principle.
• Distinguish Between Problem Types: Recognize that LLMs may excel at certain mathematical tasks, such as finding counterexamples or exploring large solution spaces, while remaining weaker at others, such as constructing formal proofs or navigating complex logical hierarchies.
• Consider the Role of Human Expertise: Even when AI systems contribute to mathematical breakthroughs, human mathematicians typically provide essential guidance on problem selection, interpretation, and validation. The AI tool deserves credit, but not sole credit.

The broader lesson from OpenAI's recent announcements is that AI progress in mathematics is real and accelerating, but it is not uniform. LLMs are becoming powerful tools for mathematical exploration, capable of tackling problems that seemed intractable just years ago. Yet they remain tools, not replacements for human mathematical intuition and judgment. As these systems continue to improve, the most productive approach will likely involve collaboration between human mathematicians and AI systems, each contributing their respective strengths.

The gap between what LLMs can do and what they cannot yet do is narrowing, but it has not closed. Researchers and practitioners should remain cautious about extrapolating from recent successes to claims of general mathematical superiority. The evidence suggests a more measured conclusion: LLMs are exceptionally good at certain classes of problems, and the field is still learning which ones those are.