Microsoft Final Report Blames Testing Gaps for 365 Outage

Microsoft has published its final report into the global Microsoft 365 outage that cut off email for people in more than 70 countries last week, blaming a software defect that its own testing never caught because the tests ran on the wrong kind of server. The report, released on Friday evening, 20 November, lists 13 fixes the company says it will make to stop the same failure happening again.
The outage began on Thursday, 12 November, and ran for almost a full day. Microsoft records the main impact from 16:20 UTC on 12 November to 16:10 UTC on 13 November, and declared the incident resolved on 14 November. Exchange Online, the cloud email service behind Outlook, was hit hardest, with most users unable to reach their mailboxes. Copilot, Teams, SharePoint, OneDrive and other Microsoft 365 services were also affected.
A bug asleep since July
At the centre of the outage was a defect that had been sitting dormant since July, when Microsoft started rolling out the software update that introduced it. The flaw lived in the network gateway servers that act as a front door to Outlook and other Microsoft 365 services, and it came from a Windows update used only on those servers. Standard versions of Windows were not affected.
The defect was a form of lock contention: under heavy load, tasks inside the Windows process that sets up incoming connections ended up waiting on each other until the servers ran out of computing power and memory and stopped responding. The report says the problem struck hardest on newer, more powerful servers, because they try to do more at once, which is why the company's testing missed it.
Microsoft acknowledged that its testing did not adequately represent the combination of hardware and traffic mix encountered during the incident. Its load tests tended to run on older, weaker machines, while the contention grows worse on newer hardware with more cores. Testing that mirrors the production environment is one of the 13 next steps in the report, and Microsoft says it should be in place by the end of March.
Three conditions, then a cascade
The report lays out three conditions that finally set the dormant bug off. First, nearly 14 percent of the gateway's server capacity was already offline that day for routine software updates. Second, all of Microsoft Teams' traffic landed on those same servers. Teams normally runs through a separate gateway, but about 15 hours before the outage a routing glitch on that gateway in Microsoft's US East region set off an automatic switch that moved all Teams traffic worldwide onto the servers that handle email. Third, the US workday got going, and widespread failures began around 8 a.m. Pacific as traffic climbed.
What turned a local failure into a global one was the way Microsoft's own safeguards behaved. As servers stopped responding, systems automatically moved their traffic to healthy servers nearby, which carried the same bug and buckled under the extra load. At the low point, more than 80 percent of the gateway's servers in North America were unresponsive. Traffic was then shifted to other regions, and within about two hours servers in every region were affected.
Getting the service back was its own problem. As servers came back online they were swamped by pent-up demand and froze again, a pattern engineers call a thundering herd. Microsoft says it will ease traffic back onto recovering servers gradually, although that fix does not yet have a date.
What Microsoft will change
To keep a failure from spreading again, Microsoft says it will change how its systems shift traffic between regions, so a problem in one part of the world stays there. It also plans to split up the email servers, so a surge from another service like Teams cannot take them down, and Teams will switch over only in the affected region. Some of the fixes are due by the end of the year and others by the end of March, while several still have no date.
Microsoft says its first notice to customers went out about 20 minutes after the outage took hold, and it plans to shorten that by the end of the year. No business email was lost, although some delayed messages to consumer mailboxes inactive for more than 30 days bounced back.
The report also confirms what Microsoft said the day after the outage began: it was not hackers, and it was not AI. The incident was confirmed not to be caused by a cyberattack, with no evidence of malicious activity, and the change was not caused by AI tooling. Satya Nadella, Microsoft's chief executive, acknowledged at the company's Ignite conference this week that it has work to do on resilience. The outage was at least the fourth significant one this year.