What Caused Amazon’s Outage?

Robotic vacuum cleaners halted in their tracks. Doorbell cameras stopped watching for package thieves, though some of those deliveries were canceled anyway. Netflix and Disney movies were interrupted, and The Associated Press had trouble publishing the news.

A major outage in Amazon’s cloud computing network Tuesday severely disrupted services at a wide range of U.S. companies for hours, raising questions about the vulnerability of the internet and its concentration in the hands of a few firms. 

How did it happen? 

Amazon has said nothing about exactly what went wrong. The company limited its communications Tuesday to terse technical explanations on an Amazon Web Services dashboard and a brief statement delivered via spokesperson Richard Rocha that acknowledged the outage had affected Amazon’s own warehouse and delivery operations but said the company was “working to resolve the issue as quickly as possible.” It didn’t immediately respond to further questions Wednesday. 

The incident at Amazon Web Services mostly affected the eastern U.S., but still impacted everything from airline reservations and auto dealerships to payment apps and video streaming services to Amazon’s own massive e-commerce operation. 

What is AWS? 

Amazon Web Services is a cloud-service operation — it stores its customers’ data, runs their online activities and more — and a huge profit center for Amazon. It holds roughly a third of the $152 billion market for cloud services, according to a report by Synergy Research Group — a larger share than its closest rivals, Microsoft and Google, combined. 

It was formerly run by Amazon CEO Andy Jassy, who succeeded founder Jeff Bezos in July. 

Too many eggs in one basket? 

Some cybersecurity experts have warned for years about the potentially ugly consequences of allowing a handful of big tech companies to dominate key internet operations. 

“The latest AWS outage is a prime example of the danger of centralized network infrastructure,” said Sean O’Brien, a visiting lecturer in cybersecurity at Yale Law School. “Though most people browsing the internet or using an app don’t know it, Amazon is baked into most of the apps and websites they use each day.” O’Brien said it’s important to build a new network model that resembles the peer-to-peer roots of the early internet. Big outages have already knocked huge swaths of the world offline, as happened during an October Facebook incident.

Even under the current model, companies do have some options to split their services between different cloud providers, although it can be complicated, or to at least make sure they can move their services to a different region run by the same provider. Tuesday’s outage mostly affected Amazon’s “US East 1” region. 

“Which means if you had critical systems only available in that region, you were in trouble,” said Servaas Verbiest, lead cloud evangelist at Sungard Availability Services. “If you heavily embraced the AWS ecosystem and are locked into using solely their services and functions, you must ensure you balance your workloads between regions.” 

Hasn’t this happened before? 

Yes. The last major AWS outage was in November 2020. There have been numerous other disruptive and lengthy internet outages involving other providers. In June, the behind-the-scenes content distributor Fastly suffered a failure that briefly took down dozens of major internet sites including those of CNN and The New York Times, plus the British government home page. Another that month affected provider Akamai during peak business hours in Asia in June.

In the October outage, Facebook — now known as Meta Platforms — blamed a “faulty configuration change” for an hourslong worldwide outage that took down Instagram and WhatsApp in addition to its titular platform. 

What about the government? 

It was unclear how, or whether, Tuesday’s outage affected governments, but many of them also rely on Amazon and its rivals. 

Among the most influential organizations to rethink its approach of depending on a single cloud provider was the Pentagon, which in July canceled a disputed cloud-computing contract with Microsoft that could eventually have been worth $10 billion. It will instead pursue a deal with both Microsoft and Amazon and possibly other cloud service providers such as Google, Oracle and IBM. 

The National Security Agency earlier this year awarded Amazon a contract with a potential estimated value of $10 billion to be the sole manager of the NSA’s own migration to cloud computing. The contract is known by its agency code name “Wild and Stormy.” The General Accountability Office in October sustained a bid protest by Microsoft, finding that certain parts of the NSA’s decision were “unreasonable,” although the full decision is classified. 
…