OpenAI has built an internal system that lets Codex agents investigate ChatGPT slowdowns on their own. The Performance Factory turns each regression into a task that moves from detection to a proposed fix, with engineers approving every change. It ran more than 1,700 investigations in just over three months.
The system grew out of OpenAI’s performance team, which spent years chasing ChatGPT latency by hand. Engineers examined metrics and logs, compared traces and read code, work that needed deep knowledge of each service. As the same slowdowns recurred, the team built more than 30 profiling tools.
How the Factory Works
At the centre sits a task service where engineers define workflows in configuration files and prompts. A dispatch loop checks which tasks are ready, then starts a Codex turn with the instructions for that state. The agent records progress and moves the task on when a step is done.
Agents work in the cloud through an internal API, inside OpenAI’s network with access to code, metrics, logs, traces and CPU profiles. Each agent runs under a service identity, so its actions are scoped and visible. Subscriptions wake agents when a deployment finishes or a review comment lands.
The Regression Workflow
The first workflow handles regressions end to end. When an alert fires, a task opens and an agent investigates, using a shared skill called perf-forensics that encodes the team’s playbook. It finds when the slowdown began, which requests suffered, and whether a deployment or flag change explains it.
A second Codex session then tries to disprove the diagnosis, rechecking assumptions and testing counterfactuals. In one case it caught an agent blaming image downloads while the file service had already recovered. OpenAI graded 100 root-cause suggestions by hand and found 70% accurate.
Fixing the problem is the hard part. Once a diagnosis passes review, Codex writes a fix and opens a pull request, with a generated summary, charts and traces for the engineer. If comments land on the pull request, the workflow wakes an agent to address them before merging.
The funnel is narrow, and the figures come from OpenAI’s team. Of more than 600 cases that reached a root cause, around 400 advanced to a proposed fix. About 50 shipped fixes were influenced by or matched agent suggestions, while around 15 merged as the exact pull requests Codex wrote.
Our Read
This is the honest version of agents doing engineering work: superb at triage, modest at shipping. Seventeen investigations a day is a real gain, and the adversarial review is the design choice that makes it trustworthy. OpenAI’s agents finished half of research tasks in a separate test.
The pattern matters more than the numbers. Workflows with explicit states, a second agent that attacks the first one’s conclusions, and a human at the merge gate is a template other teams can copy. OpenAI says it is exploring letting agents ship low-risk fixes with no human in the loop.
Why It Matters
Every large service team faces the same problem: regressions outpace the engineers who chase them. A factory that triages 17 alerts a day and hands engineers checked diagnoses with evidence attached changes what a small performance team can cover. The bottleneck is now the merge review, not the investigation.
Background
OpenAI has been building Codex into more of its own engineering, and the factory is the latest step. GPT-6.2 Astra and Sol launched in November with new agent capabilities, and the company says the factory pattern will spread to other workflows, including proactive monitoring that finds slowdowns before alerts fire.
Source: OpenAI’s engineering post and VentureBeat


0 Comments