Anthropic's Claude Fable 5, a safeguarded version of its Claude Mythos 5 model, has created a testing nightmare for independent evaluators who cannot reliably determine whether they are assessing the full capabilities of the model or a degraded fallback version. Before the model was pulled from circulation, multiple independent organizations discovered that Claude Fable 5 automatically routes certain prompts to Claude Opus 4.8, a less capable predecessor, without always making this switch transparent to users.

Why Can't Testers Evaluate Claude Fable 5 Accurately?

The core problem stems from how Anthropic designed Claude Fable 5's safety architecture. The company deployed classifiers that screen each prompt before it reaches the model, flagging questions about cybersecurity, biology, chemistry, and AI model engineering. When a prompt is flagged, one of two things happens depending on how the user accesses the model.

Through Anthropic's own applications, including the Claude Code harness used in some evaluations, flagged prompts are automatically routed to Claude Opus 4.8, which answers in Claude Fable 5's place. However, the system records this switch in a separate log event rather than in the answer text itself. This means evaluators had to manually search through logs and separate out tasks Claude Opus 4.8 had answered if they wanted to distinguish between responses from the two models.

Through the API, which is how most evaluators accessed the model, the same flag produced an outright refusal with no answer provided. Evaluators could then enable a fallback to retry the prompt on Claude Opus 4.8 or score the task as a failure.

What Did Independent Testers Actually Find?

The inconsistency created two different evaluation approaches, each producing different results. Some organizations chose a "pure" evaluation of Claude Fable 5 to measure its capabilities without influence from Claude Opus 4.8, while others conducted a "practical" evaluation that included refusals and fallbacks as part of the real-world experience.

Artificial Analysis, which evaluated Claude Fable 5 before its launch, recorded the model falling back to Claude Opus 4.8 on roughly 8 percent of tasks in its Intelligence Index, a composite of 10 tests of economically useful tasks. Most of these fallbacks were responses to science questions. Artificial Analysis included all fallback responses as part of its evaluation, producing blended scores.

Vals AI, which tests both public and proprietary benchmarks of economically useful AI tasks, published two separate sets of scores for Claude Fable 5, one including Claude Opus 4.8 fallback answers and one that counted every refusal as a failure. This dual-scoring approach highlights the fundamental ambiguity: the same model produces different performance profiles depending on how you measure it.

How Are Evaluators Responding to These Challenges?

The testing complications extend beyond the technical architecture. Anthropic introduced a mandatory 30-day data retention policy for Claude Fable 5 usage, requiring all users to accept that their prompts and outputs would be stored for a month. Some evaluators withheld proprietary prompts because of this policy, further limiting the comprehensiveness of independent assessments.

The situation reflects a broader tension between safety and transparency. While Anthropic's intention to restrict certain high-risk applications may have merit, the implementation has made it nearly impossible for independent researchers and organizations to conduct fair, reproducible benchmarks. Claude Mythos 5, the unrestricted version, was never publicly released and therefore could not be independently evaluated at all.

Steps for Understanding Claude Fable 5's Real Capabilities

• Check the evaluation methodology: When reviewing Claude Fable 5 benchmark scores, determine whether the evaluator used a "pure" assessment (measuring only unfiltered responses) or a "practical" one (including fallbacks and refusals). These produce significantly different results.
• Look for fallback rates: Ask whether the evaluation reports how often the model was routed to Claude Opus 4.8 or refused to answer. An 8 percent fallback rate means roughly one in twelve tasks may not reflect Claude Fable 5's actual performance.
• Review data retention concerns: Understand that some evaluators may have withheld proprietary test cases due to Anthropic's 30-day data retention requirement, potentially limiting the breadth of published benchmarks.
• Compare multiple sources: Because different organizations used different evaluation approaches, consulting multiple independent assessments provides a more complete picture than relying on a single benchmark score.

The Claude Fable 5 testing situation underscores a critical challenge in the AI industry: as models become more powerful and more tightly controlled, independent verification becomes harder. Evaluators face a choice between accepting incomplete data or withholding proprietary information. Neither option produces the kind of transparent, reproducible benchmarks that have historically driven progress in machine learning.

This testing opacity also raises questions about how developers and organizations can make informed decisions about which AI models to build on. If the public benchmarks for a model do not reflect its actual performance in real-world scenarios, the risk of unexpected failures or capability gaps increases significantly.