A Routine Database Job Knocked ChatGPT Offline for Hours

Human Verified

Share to: , and

A dim data center corridor at night with rows of glowing server racks, one rack dark and unlit, and a lone technician walking in the distance

OpenAI says a routine database maintenance job knocked out ChatGPT and its API for three and a half hours on 21 November. Requests failed at a peak rate of 75 percent on paid and free ChatGPT between 1:48 and 5:21 Pacific time, the company says in its post-incident write-up.

What failed, and why it spread

The trigger was maintenance on a Postgres database service used by OpenAI’s Primary API service. The work caused both read replicas of that database to fail at the same time, and the replicas were load-bearing, so every request path depending on them failed with them.

The Primary API service is the quiet centre of OpenAI’s stack. It provides user authentication and service-to-service authentication across the entire platform, so when its replicas went down, the login and handshake underneath ChatGPT and the API went with them.

Recovery was not quick. OpenAI says repeated attempts to revive either failed replica did not work, and neither did bringing up replacements in other regions. The fix came from working with its unnamed service provider to put two new replicas into service in the same region.

What OpenAI has already changed

OpenAI has already made four changes. It placed a moratorium on similar maintenance until further notice, adjusted configuration settings to reduce connection churn, and shipped a feature flag that shifts traffic away from replicas gradually, on a percentage basis, when needed.

It also tested directing all queries to the primary database instance, and says the test verified its hypothesis about why the primary failed. The company has not named the service provider behind the Postgres service, and the write-up does not say when the moratorium will lift.

What comes next

Longer-term fixes are promised rather than shipped. OpenAI says it will establish a pre-maintenance review process, hold adequate buffer capacity as hot standbys during maintenance, and build strategies to recover rapidly from replica lag, with architectural changes to eliminate the risk entirely.

The company also says it will examine its entire infrastructure and services stack to add more redundancy and fail-safe mechanisms. In its own words, outages of this magnitude are particularly damaging, and it came up short here, though it is committed to preventing a repeat.

Our Read

The candid part of this postmortem is the architecture it exposes. A read replica is supposed to be the safe copy, yet here two of them sat on the critical path for every login, and losing both took the product down. That is a design lesson, not a one-off outage.

OpenAI’s transparency is the better half of the story. Most providers describe an outage in a sentence and move on; this write-up names the mechanism, the failed recovery attempts, and the fixes already shipped. Microsoft’s own November outage drew a similar full account, and the comparison favours candour.

For customers, the warning is about single-provider stacks. When the default AI layer fails at 75 percent, every product built on it fails too, and no retry logic survives an authentication service that cannot authenticate. Redundancy has to sit below the model, not above it.

Why It Matters

Reliability is now part of the model race, not separate from it. Labs compete on benchmarks and prices, but a three-and-a-half-hour outage at 75 percent failure is the kind of event that pushes enterprise buyers toward second providers and forces the whole industry to publish better postmortems.

Background

The outage landed in a crowded week for OpenAI. GPT-6.2 Astra and Sol launched days earlier with new safety ratings, and the company has since published the system card behind them. The incident is a reminder that the infrastructure under the models matters as much as the models themselves.

Source: OpenAI status page postmortem

Share to: , and

Faheem R. Khattak

Senior automation engineer and AI marketing journo covering LLMs, algorithms, analytics, core updates, Bing search, and technical SEO.

0 Comments

Leave a Reply

Optional Take Tag:

🛡️ Spam protection: Honeypot + Time-Trap Active

This site uses Akismet to reduce spam. Learn how your comment data is processed.