Newsletter Subscribe
Enter your email address below and subscribe to our newsletter
[forminator_form id="25163"]

x.com.economictimes.com.theregister.com.Microsoft has traced last week's global Microsoft 365 outage, which knocked Outlook and Exchange Online offline for many users for most of a day, to a hidden software defect introduced in July. The bug surfaced only when a Teams traffic shift and a capacity shortfall hit the company's networking gateway at the same time. Charles Lamanna, Microsoft's executive vice president of Copilot, Agents and Platform, set out the cause in a post on X on Monday, November 16. He wrote that the November 12 incident "was caused by three main factors that together caused a global outage for the networking gateway that fronts many Microsoft 365 services."x.com
According to Lamanna's post, the first factor was a latent defect that caused lock contention inside an internal process, so servers became overwhelmed and stopped responding under certain loads. The defect came from a fix built specifically for Microsoft 365 gateway servers, and he said it does not apply to broader Windows deployments. The change began rolling out in July and ran worldwide for more than three months without problems. Load testing missed it because the tests focused on weaker hardware with fewer cores, while the problem gets worse on newer machines with more cores.x.com
The second factor came about 15 hours before the outage, when Teams failed over from an alternative gateway to the primary one that also serves Exchange Online, adding load and changing traffic patterns. The third was that nearly 14% of the gateway's server capacity was out of service because of an unrelated issue from the day before. Microsoft said neither of these would have caused harm on its own.economictimes.com
The failure started during peak U.S. traffic. As unresponsive servers dropped out, traffic was sent to healthy ones, which then hit the same defect. "In less than two hours, the previously unknown defect was encountered across our fleet," Lamanna wrote, adding that "at its lowest, more than 80% of servers were unresponsive". Teams, OneDrive, SharePoint and Office apps moved off the faulty gateway and recovered sooner, but Exchange Online could not.abcnews.com+1
Recovery was slow because restored servers were immediately overwhelmed by waiting traffic. The defect was also months old, so a simple rollback wasn't possible. Engineers instead wrote an update that put the code back to its earlier state.theregister.com
Microsoft's preliminary report shows that its monitoring flagged connectivity problems at 16:03 UTC on November 12. The mitigation was applied at 07:20 UTC on November 13, and Microsoft confirmed services were healthy at 16:10 UTC that day. Lamanna said a final post-incident review with more details will come later this week, and The Register reported that it is due on November 20. "We are deeply sorry for the impact to all our customers," Lamanna wrote.theregister.com+2