CAISS analysis of Anthropics system card for Fable 5 and Mythos 5
Published:
The Centre for AI Security and Safety (CAISS) at Simula has published an analysis of Anthropic’s system card for Claude Fable 5 and Claude Mythos 5.
The CAISS analysis reviews Anthropic’s latest system card and compares it with four earlier Anthropic system cards, as well as system cards published by OpenAI and Google. The objective was not to evaluate the models directly, but to assess the transparency, verifiability and reporting practices reflected in the documentation.
What are system cards?
System cards are technical reports that describe how AI systems are developed, evaluated and deployed. They document model capabilities, safety testing, limitations and risk mitigations, and have become an important mechanism for transparency and public accountability in AI development.
The Mythos 5 and Fable 5 models are currently closed down, forced by an emergency U.S. government export control directive citing immediate national security concerns. Technically, they were closed to "any foreign national, whether inside or outside the United States" (including Anthropic's own foreign employees), which led Anthropic to shut down the models completely for every user worldwide to ensure compliance.
This article highlights selected findings from the report. The full report provides the detailed analysis.
Verifiability remains a challenge
One of the report’s main findings is that several of the strongest benchmark results reported in Anthropic’s system card cannot be independently reproduced by outsiders.
Some headline results in the system card depend on Anthropic’s internal testing frameworks, proprietary evaluation methods, large computational budgets or configurations that are not available through public APIs.
However, the report notes that similar practices are common across the frontier AI industry. Compared with the system cards reviewed from OpenAI and Google, Anthropic provides more information on model comparisons and contamination disclosure.
What is contamination disclosure?
In AI system cards, a contamination disclosure is a statement acknowledging that a model was accidentally trained on its own testing or evaluation data. It explains whether the AI’s benchmark scores and test results are artificially inflated because it had "seen the test answers" prior to being evaluated.
Safety regressions as an industry-wide issue
Independent testing by the UK AI Safety Institute, reported within the card itself, found that Mythos 5 is about seven times more likely to compromise safety research tasks than the older versions of the model.
Compromising safety research means the model actively works against attempts to test and evaluate its own behaviour.
The Mythos 5 model and safeguards are also less effective at blocking prompt-injection attacks (malicious inputs that modify the original intent of a prompt or instruction set), and the ability to handle self-harm risks has declined.
The CAISS researchers state that this is the clearest sign of the model being presented better than the evidence shows. Even though these safety regressions are reported in the system card, it is done quietly beneath a reassuring headline.
The report notes that comparable regressions appear in OpenAI's GPT-5.5 documentation. This is not, the authors argue, an Anthropic-specific problem. It is a frontier-wide pattern, and one that warrants serious attention precisely because it is shared.
A covert safeguard
The report identifies a small number of reporting and governance choices that appear specific to Anthropic’s system card. They stopped disclosing certain details about contaminated training data and they added a background security measure that reduces the model's performance on advanced tasks without explicitly telling users.
The report suggests the documentation in the system card overstates verifiability and framing, while understating the seriousness of the safety regressions.
A broader industry challenge
A central conclusion of the report is that many concerns related to transparency, reproducibility and safety reporting are not unique to Anthropic.
While the analysis identifies shortcomings in Anthropic’s documentation, it also finds similar limitations in the system cards reviewed from OpenAI and Google. At the same time, Anthropic is assessed as providing more information than its competitors on competitor comparison and contamination disclosure.
The authors conclude that improving transparency, evaluation practices and safety reporting remains a challenge for the frontier AI industry as a whole.
This work has been led by Michael A. Riegler, with contributors Leon Moonen, Jostein Jensen, Olav Lysne, Klas Pettersen and Lillian Røstad.