Skip to content

Tech Industry

Microsoft blames July bug for global Microsoft 365 outage

Microsoft cited three causes and said more than 80% of networking gateway servers went unresponsive at peak impact.

https://breach-signal.com/tech-industry/microsoft-blames-july-bug-365-outage/

Share

Key facts

  • The cascading failure left more than 80% of Microsoft's networking gateway servers unresponsive at peak impact.
  • The outage disrupted Outlook, Exchange Online, Teams, Copilot, OneDrive, and SharePoint worldwide.
  • Around 15 hours before the outage, Microsoft Teams moved automatically to Exchange Online networking infrastructure, adding traffic to those servers.
  • An unrelated problem from the previous day left nearly 14% of the server capacity in the networking gateway offline.

Microsoft said a software bug introduced in July caused the worldwide Microsoft 365 outage that ran from 11:20 AM ET on November 12 to 11:10 AM ET on November 13, when services were fully restored.

The company named three causes for the failure: an undetected software bug, higher traffic after a Teams failover — an automatic shift of users to back up infrastructure — and reduced server capacity. Around 15 hours before the outage, Microsoft Teams moved automatically to Exchange Online networking infrastructure, adding traffic to those servers. An unrelated problem from the previous day left nearly 14% of the server capacity in the networking gateway offline.

The networking gateway — the system that routes traffic between users and cloud services — fronts many Microsoft 365 services. The cascading failure left more than 80% of Microsoft’s networking gateway servers unresponsive at peak impact. The outage disrupted Outlook, Exchange Online, Teams, Copilot, OneDrive and SharePoint worldwide.

Exchange Online could not switch to alternative infrastructure and recover, leaving users unable to access mailboxes. Safeguards meant to stop problems from spreading between regions did not contain the cascading outages, so the disruption spread worldwide.

Microsoft said its labs used weaker hardware with fewer cores than production servers, so normal load testing missed the latent defect — a hidden flaw that does not appear under ordinary conditions. Machines with more cores and newer hardware made the defect worse, while Microsoft’s load testing focused on weaker hardware SKUs, or server models, with fewer cores.

The hidden defect was in software rolled out globally for weeks with no impact, Microsoft said. Lower server capacity combined with increased load at peak U.S. traffic, triggering the latent July bug in the network gateway.

When servers stopped responding, Microsoft’s systems sent users to other available servers, but those servers became overwhelmed by extra traffic and failed. Restarting affected servers led to them being quickly overwhelmed again by backed-up traffic — requests that had been waiting — which made recovery difficult.

Woman holding her head in frustration and gesturing stop, representing system outage stress.
A visual metaphor for the overwhelming stress and frustration of managing cascading server failures during a major outage. Photo by RobinHiggins on Pixabay

Microsoft said customer impact was over by 11:10 AM ET on November 13. Engineers produced a fix for the bug and pushed it to affected servers, and services stabilized throughout Friday. Because the bug dated back to July, Microsoft could not roll back the code, so engineers created a patch to return it to its previous good state. Engineers took every server offline to apply the fix and restored each one carefully, causing long waits and loss of service.

Since then, the company has added capacity, improved monitoring, and put safeguards in place to stop traffic redirection — sending users to different servers — from causing similar cascading failures. The company said its investigation continues and expects a final post-incident report — a formal review of what went wrong — later this week.

Charles Lamanna, who leads Copilot, Agents and Platform at Microsoft, apologized for the disruption on X. Lamanna apologized for the impact to customers and said the company recognizes the effect on their normal business and operations. Lamanna said the recovery processes were not designed for restoring service from a broad unresponsive state.

Microsoft 365 relies on two independent network gateway systems to handle outages and disruptions, according to Lamanna. The failover from the alternative networking gateway to the primary gateway supporting Exchange Online was part of normal operations and standard resiliency efforts. Teams experienced a separate outage earlier on November 12, but Microsoft said it is unclear whether the incidents are related.

Affected businesses can file a claim with Microsoft to receive credits for the downtime, Mashable reported. Mashable called it the biggest outage of the year by a wide margin and said the fix required about 24 hours over two business days.

Was this helpful?

Join the discussion

Please read our Comment Policy before commenting. First-time comments are held for moderation. By posting a comment you agree to our Terms of Use and Privacy Policy, which explain how your name, email address and comment are used.

Your email address will not be published. Required fields are marked *