Anthropic's co-founder Christopher Olah has issued a stark warning: the company he helped build operates under incentives that can conflict with doing the right thing, and the world needs external moral voices to hold AI labs accountable. Speaking at the Vatican on May 25, 2026, during the presentation of Pope Leo XIV's first encyclical on artificial intelligence, Olah delivered a remarkably candid critique of his own industry, acknowledging that geopolitical pressure, pride, and the drive to stay commercially viable shape decisions at every frontier AI lab, including Anthropic.

What Did Olah Actually Say About AI Lab Incentives?

Olah's remarks were notably self-critical for someone leading interpretability research at one of the world's most prominent AI safety companies. He admitted that his words "may sound strange coming from the co-founder of an AI company," but emphasized that every frontier AI lab "operates inside a set of incentives and constraints that can sometimes conflict with doing the right thing". Rather than claiming Anthropic could solve these problems alone, he called for "informed critics who will tell the labs when we are failing" and "moral voices that the incentives cannot bend."

The timing and venue of this message carry weight. Olah was not speaking to a tech conference or investor audience, but to the Pope, the Roman Curia, academics, and diplomats at the Vatican's Synod Hall. His appeal was explicitly for religious communities, civil society, scholars, and governments to take AI governance seriously and "push events in a better direction." This positioning of external institutions as necessary checks on AI development represents a significant acknowledgment from inside the industry that self-regulation has limits.

How Does Olah Frame the Nature of AI Systems Themselves?

Beyond the incentives question, Olah offered a philosophical reframing of what AI models actually are. He rejected the common engineering analogies that dominate tech discourse, arguing that "AI systems are not engineered the way a bridge or an airplane is engineered." Instead, he described them as "grown" on structures modeled after the brain, drawing from "an enormous inheritance of human thought and speech". This distinction matters because it challenges the idea that creators fully understand or control what they've built.

Olah

Olah's interpretability research background informed his observation that AI models are "far more subtle, odd and beautiful than science fiction prepared us for." He then introduced a novel analogy: "It's a little bit like bringing a fictional character to life. And now we're entering an extraordinary world where those fictional characters speak to us, do work, have jobs. This clearly raises questions beyond computer science". The fictional character framing sidesteps the tool-versus-person binary that usually dominates AI ethics discussions, instead positioning AI systems in an ontologically strange space that most people have felt but rarely articulated.

Why Is This Message Significant for Anthropic's Leadership?

Dario Amodei and Daniela Amodei founded Anthropic in 2021 after departing OpenAI, specifically because they wanted to pursue a different approach to AI development centered on safety and alignment. The company has since raised billions in independent funding, including a major Series F round backed by investors like Google and Spark Capital, establishing it as one of the best-capitalized AI safety companies in the world. Yet Olah's Vatican speech suggests that even with this focus on safety, the structural incentives facing the company remain a concern worth flagging publicly.

His remarks also underscore a philosophical tension within Anthropic itself. The company's safety-focused approach, codified through methods like Constitutional AI, reflects a deliberate departure from the culture of OpenAI and other organizations that preceded it. But Olah's warning suggests that good intentions and safety-focused research methods are not sufficient shields against the pressures of competition, geopolitical dynamics, and commercial viability. The implication is that even companies founded explicitly to do AI differently still operate within a system of incentives that can bend their moral compass.

How to Understand the Broader Context of This Moment

• Anthropic's Independence: The company has built its own substantial funding base and operates entirely independently from Elon Musk or other early OpenAI backers, despite historical connections through personnel and ideas that flowed from OpenAI to Anthropic's founding.
• Competitive Pressure: Musk's xAI, which launched in 2023, now competes directly with Anthropic in the large language model space, creating a landscape where different AI labs pursue different philosophies and business models.
• External Accountability Mechanisms: Olah's call for informed critics, moral voices, and government oversight suggests that the AI industry recognizes the need for checks beyond internal ethics teams and safety research.

Olah's most pointed statement captured the core of his argument: "Some might believe that matters of AI are best handled by computer scientists like myself. They are mistaken. The questions raised by AI are bigger than the AI research community. Not just in their implications, but also in their nature". This is not a claim that computer scientists lack expertise, but rather that the questions AI raises are fundamentally philosophical, theological, and ethical in character. Domain expertise in building AI does not confer authority over its governance.

For Anthropic, the speech represents both a validation of its safety-focused mission and a humbling acknowledgment of its limitations. The company's Claude model family has expanded significantly, and it has secured major enterprise partnerships, cementing its position as a serious player in the AI industry. Yet one of its co-founders chose a global stage to argue that even well-intentioned AI labs cannot be trusted to govern themselves. That message, delivered with intellectual honesty and without defensiveness, may ultimately matter more to Anthropic's long-term credibility than any technical achievement or funding round.