The July Bug That Hid for Months — Then Killed Microsoft 365

Human Verified 0

Share to: , and

A glowing world map of network nodes with one dark failed hub at the centre and amber warning signals cascading outward, with small envelopes drifting disconnected

Microsoft has traced last week’s global Microsoft 365 outage to a software defect that sat hidden in its network gateway code since July, its preliminary post-incident report says. Three failures combined to knock out more than 80% of the gateway servers at the worst point.

Three failures, one cascade

The defect came from a fix prepared for the gateway servers that sit in front of Microsoft 365 services, handling connections before they reach Exchange Online. It caused lock contention inside an internal process, leaving servers overwhelmed and unresponsive under certain loads.

Microsoft says the faulty change passed its normal testing and rolled out worldwide over several weeks from July, running for more than three months without any impact. Its labs used older, less powerful hardware, while the defect struck hardest on newer machines with more processor cores.

About fifteen hours before the outage, Teams traffic moved from a backup gateway onto the primary gateway that also carries Exchange Online, raising its load and changing traffic patterns. At the same time, nearly 14% of gateway capacity was offline because of an unrelated problem from the day before.

Neither the extra load nor the missing capacity would have caused an outage on its own, Microsoft says. Together, during the peak traffic window in the United States, they triggered the hidden defect, and requests began piling up until servers stopped responding altogether.

When servers lost connectivity, traffic management automatically sent users to healthy endpoints, as it does every day. But the redirected load triggered the same defect on those servers too, and within two hours the failure had spread across more than 80% of the gateway fleet.

Partitions that normally divide the gateway into separate capacity units and regions should have contained the damage. A separate issue stopped those bulkheads working, Microsoft says, which is how a local failure spread across its entire network gateway infrastructure worldwide.

Teams, OneDrive, SharePoint and the Office apps could move away from the broken gateway, so most of them kept working through the outage. Exchange Online could not make the same move for scale reasons, which is why email stayed down longest.

Recovery was slow because the usual automated processes were never designed for a failure this broad. Each time engineers brought servers back, waiting traffic overwhelmed them again, and the months-old defect meant rollbacks and feature flags could not quickly undo the change.

Once engineers confirmed the defect, they built an update that restored the affected code to its previously working state. They took large groups of servers offline, applied the fix and restarted them, working through more than half a region at a time before returning it to service.

Why It Matters

The outage cut off email for businesses, schools and governments in more than 70 countries, with hundreds of thousands of complaints logged by Downdetector. Microsoft says more than 430 million people use its 365 services, and affected customers can claim service credits under its agreement by 31 December.

Our Read

The report’s most uncomfortable detail is that the safety nets made things worse: failover shifted load onto the faulty gateway, and redirection spread the defect server by server. Microsoft’s candour here is notable, but the final report, due later this week, must explain how its testing and partitions get fixed.

Background

The outage began on Thursday 12 November and services were restored over the following two days, with Microsoft apologising and ruling out any cyberattack or AI cause. Its final post-incident report, with more technical detail and repair steps, is due later this week.

Source: Neowin’s report and MyBroadband’s account of Microsoft’s preliminary post-incident report.

Share to: , and

Faheem R. Khattak

Senior automation engineer and AI marketing journo covering LLMs, algorithms, analytics, core updates, Bing search, and technical SEO.

0 Comments

Leave a Reply

Optional Take Tag:

🛡️ Spam protection: Honeypot + Time-Trap Active

This site uses Akismet to reduce spam. Learn how your comment data is processed.