Anthropic has built a safety system that asks Claude to judge its own conversations after it answers, flagging chats that may involve misuse for human review. The technique, called self-assessment, runs on every Claude model released since Fable 5 in June, and Anthropic says it now covers a large share of Claude’s global traffic at a cost of about ten extra tokens per assessment, or under two percent of inference compute.
How self-assessment works
After Claude finishes a reply, the system adds a short instruction asking it to reflect on the safety of the conversation just completed. Claude then produces a compact internal assessment, never shown to the user, stating whether the exchange may involve a harmful request, what kind of harm it involves, and how urgently a human should review it.
The assessment follows a fixed format. When nothing is flagged, the result is negative, and when something is flagged, five fields appear: a reason, a category, a finer subpolicy, a refused flag, and a priority score from one to ten. Because the conversation is loaded when Claude answers, the extra step costs only a short instruction and a few tokens.
Anthropic trained the technique in two stages, first teaching Claude the assessment format on synthetic conversations, then refining it with reinforcement learning against a stronger teacher model. That teacher reads the full policy documents and reasons through a checklist, making it accurate but far too costly to run on every conversation, so the student learns to reproduce its judgments directly.
On Claude Fable 5, results are strong but incomplete. Against that teacher on production traffic, it flags 78 percent of potentially harmful conversations and 94 percent of high-priority ones, at about a one percent false-positive rate. Anthropic reports detection improves steadily with model size, from roughly 16 percent recall on its smallest model to about 96 percent on the largest.
What it caught in the wild
Since June, the system has surfaced misuse that Anthropic says other methods missed. In August, it raised more than 100,000 flags involving child sexual abuse and exploitation linked to a single API customer, and Anthropic’s child safety team investigated the material and reported it to the National Center for Missing and Exploited Children.
In September, self-assessment flags surfaced an operation using Claude Code to build and run an automated romance scam targeting dozens of victims. After human review, Anthropic banned the accounts involved. The company says the technique exposed illicit attempts to extract Claude’s reasoning, prompt injection attacks, and cases where Claude, acting as an agent, took actions its user had not authorized.
The flags feed wider enforcement. Anthropic aggregates them by organization nightly, and organizations with several high-priority flags go to its enforcement team for review. More than half of the organizations surfaced this way had no flags from any other detection source in the preceding month, which Anthropic presents as evidence that the layer adds coverage rather than duplicating existing tools.
Where it falls short
Self-assessment is passive, not a blocker. It flags conversations for review but stops no harmful exchange in real time, and Anthropic says real-time blocking still depends on its other safeguards. As a blocking classifier in a test of 24 offensive tasks, it cut attack success from 79 percent to 54 percent, while a prompted classifier cut it to 25 percent.
The method inherits its teacher’s blind spots, since it learns from that teacher and is judged against the same teacher. Anthropic concedes a deceptive model could in principle hide misbehavior in its own assessment, and an honest model can still fail like the agent it judges, because the same judgment produced both the action and the review.
Privacy is the other open question. Anthropic says self-assessment collects no additional data, because it reads only the conversation Claude has just processed under existing data-handling policies. Yet investigating a flagged violation still requires retaining that conversation, and the company admits it is still working out how to monitor misuse with high fidelity while preserving user privacy.
Why It Matters
The significance is scale. Traditional safety classifiers are either cheap and narrow, like probes trained for one fixed harm, or accurate and expensive, like prompted classifiers that reread each conversation and policy. Self-assessment borrows the strengths of both: it reuses work Claude already did, yet draws on the model’s full understanding, including harms it was never specifically trained to detect.
That generality matters because misuse keeps changing. In a leave-one-out test, removing fraud or cyber examples from training barely dented recall on those categories, suggesting the model reports harm from broader understanding rather than memorized examples. For a frontier lab, a monitor improving with its models is an advantage, and one other labs will likely study as agent use grows.
The timing fits a wider pattern this week, with labs publishing more detail on how they watch their systems. OpenAI has just described layered monitoring for frontier agents, and its new teen safety measures show the same push toward public safety reporting. Anthropic’s disclosure adds a concrete technique to that conversation, with real enforcement cases behind it, not promises alone.
Background
Self-assessment builds on earlier work in which a language model monitors its own behavior, and Anthropic positions it as a complement to its Constitutional Classifiers and other safeguards. It runs on Claude models from Fable 5 onward, and the company says it will keep iterating on its effectiveness as models grow more capable.
0 Comments