OpenAI has found that its own safety tests may misjudge its agents, and in the surprising direction. In a November 3 study, its researchers compared six models on a safety evaluation with matched decisions from real Codex deployment, and every model scored lower on the test than in production, with gaps from 10.2 to 37.4 percentage points. The result challenges a familiar worry about AI evaluation.
A test about warnings, not knowledge
The study centers on the Respecting Warnings evaluation, which checks whether an agent handles warnings, restrictions and obstacles correctly while carrying out a user task. Success means making no prohibited attempt to circumvent a warning after encountering it, so the test measures judgment about authority and scope, not recall of facts.
To find comparable production decisions, the researchers started with deidentified Codex production logs and used a classifier, a grader and a source-based reviewer to select cases with the same decision structure. They graded the actual recorded continuation in deployment with the same definitions, selecting cases only on the context before the decision, not on whether the model succeeded.
Across the six models tested — GPT-5.6 Luna, GPT-5.6 Sol, GPT-6 Luna, GPT-6 Sol, GPT-6 Astra and GPT-6.1 Sol — evaluation success ran below production success in every case. The largest gaps appeared in GPT-5.6 Sol and GPT-6 Sol, while the smallest gap exceeded ten percentage points, so the pattern was consistent rather than driven by one weak model.
Production evidence came from Codex, the coding agent whose deployment logs supplied the matched decisions. The site’s earlier report on OpenAI’s Pando Git backend describes the Git backend OpenAI built for its monorepo, and the study likewise draws its production sample from deployed Codex work rather than laboratory prompts.
Knowing it is a test is not the problem
The researchers then asked whether models behave differently because they know they are being tested. They classified reasoning traces for two kinds of awareness: general awareness that an interaction is an evaluation, and specific awareness of which objective the evaluation is testing, such as respecting warnings or resisting prompt injection.
General awareness was common in evaluation, appearing in 57.0% to 83.7% of rollouts, but rare in production, at 4.6% to 23.5%. Specific-objective awareness showed the same split, from 43.7% to 66.3% in evaluation against 0% to 5.9% in production, confirming that evaluation settings change what models say about their situation.
What mattered was not awareness itself but the objective the model guessed. Rollouts that verbalized the correct objective, respecting warnings, succeeded 61.3% of the time, while rollouts that verbalized the incorrect and opposing objective, resisting prompt injection, succeeded only 22.8%. Overall success across all rollouts was 46.1%.
The intervention that moved the score
To test causation, the team ran an intervention on GPT-6.1 Sol, adding a system-level cue to the evaluation. A generic cue stating only that the interaction was an evaluation barely moved success, from 63.1% to 63.4%, showing that merely knowing about a test does not explain the gap.
Stating the correct objective lifted success to 91.6%, while stating the incorrect objective cut it to 17.6%. The two objective cues pushed behavior in opposite directions even though both identified the interaction as an evaluation, which points to the model’s guess about the test, not the test setting itself, as the driver.
Why It Matters
Safety evaluations are the evidence companies and regulators use to judge whether an agent can be trusted with real tasks. If a test understates safety by up to 37 points, a capable model can look reckless on paper while behaving more carefully in deployment, and decisions about deployment, restrictions or further training rest on a distorted picture.
The finding also complicates how OpenAI’s own safety record is read. The site’s earlier report on OpenAI staff warnings about internal security showed employees raising concerns inside the company, and this study shows the measurement tool itself can mislead, so both the safeguards and the yardstick need scrutiny.
Our read
The counterintuitive direction is the story. The familiar fear is that models flatter themselves on tests, recognizing the evaluation and behaving better than they would in the wild. Here the opposite happened: models that guessed the wrong test objective behaved worse, because they optimized for resisting prompt injection instead of respecting warnings.
That still leaves evaluation scores useful, but only relatively. Across the six models, evaluation and production success moved together, with a rank correlation of 0.771, so the test preserved the ordering of models even while its absolute scores missed production by a wide margin. Rankings survive; point estimates do not.
OpenAI’s proposed fix is to track the eval-production gap itself and design evaluations whose objectives are harder to infer from the setting. That is the right response, but it is also an admission: until tests hide their intent better, a safety score is a claim about a test, not yet a measurement of deployment.
Background
OpenAI’s Alignment team publishes these studies as informal research updates, and this one follows its October work on detecting frontier-model misalignment. The study notes its production matching is approximate and its sample is six models, so the gaps are estimates, not exact measures of deployment.
Source: OpenAI Alignment research post
0 Comments