Microsoft Exchange Online Outage Explained: Causes, Impact and Lessons

•

 6 min video

•

 4 min read

YouTube video ID: ytmawkPTP24

Source: YouTube video by Fireship — Watch original video

PDF

Last Thursday, Microsoft's Exchange Online, the backend for Outlook email, experienced a significant outage that lasted approximately 36 hours. The incident, which began around 11:17 AM Eastern time, affected not only Outlook but also Copilot, SharePoint, and about a dozen other Microsoft 365 services. Reports of the outage came from 74 countries within an hour, leading to widespread disruption, including traders on Wall Street resorting to older communication methods.

The Microsoft Outage: A Cascade of Failures

Initially, Microsoft theorized an "elevated load" was causing the issue, but quickly clarified that it was due to "fewer servers" becoming "unexpectedly unavailable." The root cause was a failure in how load balancing works. Load balancers continuously ping servers for health checks, removing unhealthy ones from rotation. In this incident, a large number of servers failed simultaneously, overwhelming the remaining healthy servers, which then also failed their health checks, creating a domino effect.

Microsoft's status updates confirmed that servers were failing automated health checks, and the process responsible for establishing secure connections became so overwhelmed that servers stopped responding. The classic solution of restarting the servers proved ineffective because, as Microsoft stated, "the servers can become unresponsive again once they receive traffic" due to the immense backlog of clients attempting to reconnect.

To mitigate the issue, Microsoft attempted an "alternative health check method," essentially bypassing standard health checks to see if servers could function despite being unhealthy. Compounding the problem, the outage also affected the 365 admin center, preventing administrators from using their tools to diagnose the issue. Even Microsoft's PR firm's email bounced when contacted by NBC News.

The real-world impact was significant: government agencies reverted to paper records, a Colorado jail processed bond payments in person, and Australia effectively got a long weekend due to the time difference. Notably, Microsoft Teams remained operational throughout the outage.

Unrelated but Similar: The FAA Grounding

On Friday afternoon, a separate, major incident occurred when the Federal Aviation Administration (FAA) grounded all departing flights in the United States for approximately five hours. This was only the second such nationwide grounding since 9/11, resulting in over 9,000 flight cancellations. Despite initial speculation and claims by some tech influencers that the FAA grounding and the Microsoft outage were connected, officials quickly debunked these theories.

The Homeland Security Secretary and Microsoft both stated there was no connection between the two events. The Transportation Secretary initially blamed overloaded switches and routers, later attributing the issue to an L3 Harris defense contractor who swapped out a router at a remote FAA site. This change reportedly flooded other routers with thousands of login attempts every few seconds. Other sources pointed to old Cisco routers as the culprit, though the FAA has yet to release an official root cause.

Shared Underlying Issues

While not directly related, both the Microsoft outage and the FAA problems shared a common underlying theme: a network configuration change overwhelming a critical system.

Regarding the Microsoft outage, a full post-mortem is pending, but it's known that they began rolling back a recent network configuration change. Microsoft later apologized, stating the incident was "introduced by an issue during an earlier network capacity upgrade." Routine network maintenance apparently exposed a latent bug in the system responsible for secure connections, causing connection requests to pile up and servers to run out of processing power. Ironically, the upgrade intended to add capacity ended up removing it.

Both Microsoft and the FAA were quick to emphasize that these issues were not caused by AI or "shitification of software," but rather by human error. This assertion is particularly important for Microsoft, which recently offered voluntary exit packages to approximately 60,000 employees (about a quarter of its workforce) to fund the construction of more data centers for AI agents.

The Microsoft outage alone is estimated to have caused roughly $7 billion in lost productivity. Affected users may be entitled to service credits by contacting Microsoft support, though email is not recommended.

Open Router: A Solution for API Management

The video's sponsor, Open Router, offers a single API to access over 600 different AI models. It automatically selects the best model for each request based on intelligence, speed, and cost. Open Router recently rebuilt its routing algorithm to score providers based on the true cost of a successful response, including caching, token return speed, and request reliability. These scores refresh frequently, allowing the system to automatically reroute traffic if a provider slows down or implements rate limiting. The new router has reportedly led to 20% faster time to first token and 60% fewer rate limit errors without requiring code changes.

  Takeaways

  • A 36‑hour outage of Microsoft Exchange Online began at 11:17 AM ET and disrupted Outlook, Copilot, SharePoint and dozens of other Microsoft 365 services across 74 countries.
  • The failure was traced to load‑balancer health checks; a sudden loss of many servers overwhelmed the remaining ones, creating a cascading domino effect.
  • Microsoft’s attempts to restart servers and use an alternative health‑check method failed because the backlog of reconnection traffic quickly made servers unresponsive again.
  • The outage forced government agencies, a Colorado jail and Wall Street traders to revert to manual processes, while Teams stayed online and the admin center was unusable.
  • Both the Microsoft incident and the FAA grounding were linked to network‑configuration changes that overloaded critical systems, highlighting human error over AI as the root cause.

Frequently Asked Questions

Why did the load balancers cause a domino effect during the Microsoft outage?

Load balancers continuously ping servers and remove any that fail health checks. When many servers went down simultaneously, the remaining healthy servers were flooded with traffic, causing them to fail their own health checks and creating a cascading domino effect.

How did the FAA grounding relate to the Microsoft outage?

Although some speculated a connection, officials confirmed the FAA grounding and the Microsoft outage were unrelated; both incidents simply involved network‑configuration changes that overwhelmed critical systems, not a shared cause.

Who is Fireship on YouTube?

Fireship is a YouTube channel that publishes videos on a range of topics. Browse more summaries from this channel below.

Does this page include the full transcript of the video?

Yes, the full transcript for this video is available on this page. Click 'Show transcript' in the sidebar to read it.

Helpful resources related to this video

If you want to practice or explore the concepts discussed in the video, these commonly used tools may help.

Links may be affiliate links. We only include resources that are genuinely relevant to the topic.

Full transcript is not shown on this page

This page focuses on the summary and original notes. For full verification, refer to the original YouTube video.

PDF