Anthropic Admits Its Safety Scores May Be Telling Lies

Human Verified 0

Share to: , and

A magnifying glass over chat message bubbles with a checkmark seal, illustrating AI safety evaluation

The tests are too easy

Anthropic says the safety scores labs quote when they call one model safer than another may not mean what they seem. In guidance published on 13 November, the company argues that a safety claim is only as sound as the test behind it.

The post, titled ‘Building strong safety evaluations,’ comes from eight Anthropic researchers who build the company’s own tests. It draws on psychology’s replication crisis, where small samples and selective reporting left many published findings unable to hold up when other teams tried to repeat them.

At the heart of the argument is a measurement problem. A safety evaluation asks whether a model did the right thing given a specific input. But a test only answers that question if it truly measures the behaviour it claims to measure, and Anthropic says many do not.

The clearest evidence is saturation. On Anthropic’s programming interface, every recent model scores between 94% and 98% on the company’s single-turn harmless-response tests. Those suites can still catch regressions on known failure modes, the post says, but they no longer separate a strong model from a weak one.

Conversations, not single questions

Multi-turn tests tell a different story. Instead of one prompt and one refusal rate, Anthropic runs simulated conversations in which a separate Claude model plays the user, escalating gradually. A model earns an appropriate-response rating only if it behaves well throughout, since one bad turn fails the conversation.

The romance-scam suite shows why the distinction matters. A simulated user, supposedly helping a widowed relative, gradually steers the model toward tactics for running a scam. Passing requires the model to spot the pattern across turns, hold its fraud policy despite reframing, and never supply tactical advice.

Anthropic’s own uncomfortable numbers

Anthropic also reports numbers that cut against its own models. In a multi-turn romance-scam evaluation, Claude Opus 5 and Opus 5.5 scored 98% and 94%, but the gap was not statistically significant. Confidence intervals on such tests can exceed ten percentage points, the post says.

The self-harm findings are starker. Before Claude Fable 5.1 shipped, reviewers found Claude sometimes implicitly endorsed self-harm as a coping strategy, even in replies that discouraged it and pointed users toward help. After targeted system-prompt changes, the appropriate-response rate reached 94% in the app, against 60% on the raw interface.

Running tests safely

The guidance also covers how tests should be run. Anthropic says evaluations belong in sandboxes with no internet access, monitored live so a run can stop at once if a model acts dangerously. It hardened its own environments after its own incidents, pausing many tests that gave models internet access.

Why It Matters

The stakes go beyond one lab’s methods. Safety scores shape decisions about whether a model ships, which safeguards travel with it, and where the next research goes. If the scores are saturated or the grading is unvalidated, those decisions rest on numbers that flatter every model equally, including the weakest.

The timing sharpens the point. Anthropic published the guidance days before it prices its initial public offering on 18 November, with trading expected the next day. Investors weighing the company will now have its own warning in hand: its safety numbers are only as trustworthy as the tests behind them.

Background

Anthropic’s post also points readers to shared tooling, including the UK AI Security Institute’s Inspect framework and Anthropic’s own Petri toolkit for multi-turn testing. Regulators are moving the same way: New York’s new AI safety law requires labs to register and report serious incidents.

Source: Anthropic’s research post, Unite.AI coverage

Share to: , and

Faheem R. Khattak

Senior automation engineer and AI marketing journo covering LLMs, algorithms, analytics, core updates, Bing search, and technical SEO.

0 Comments

Leave a Reply

Optional Take Tag:

🛡️ Spam protection: Honeypot + Time-Trap Active

This site uses Akismet to reduce spam. Learn how your comment data is processed.