How a small bug escalated into a massive outage that destroyed the internet

A massive AWS outage On Monday, which brought down some of the world’s most popular apps and services, it all started with a problem.

The bug – occurred when two automated systems were trying to update the same data simultaneously — which snowballed into something much more serious that Amazon engineers rushed to fix, the company said in a statement Thursday. postmortem evaluation.

THE massive cloud service outage meant people couldn’t order food, communicate with hospital networks, access mobile banking, or connect to their security systems and smart home devices. Major global companies, including Netflix, Starbucks and United Airlines, were temporarily unable to allow customers to access their online services.

“We apologize for the impact this event has caused on our customers,” Amazon said in a statement posted on the AWS website. “We know this event has had a significant impact on many customers. We will do everything we can to learn from this event and use it to further improve our availability.”

At a high level, the problem stemmed from two programs competing to write the same DNS entry – essentially a record in the Internet phone book – at the same time, resulting in an empty entry. This threw several AWS services into disarray.

“The phone book analogy is pretty apt in that the people on the other line are there, but if you don’t know how to reach them, then you have a problem,” Angelique Medina, manager of Cisco’s ThousandEyes Internet Intelligence network monitoring service, told CNN. “And that phone book actually went poof.”

Indranil Gupta, a professor of electrical and computer engineering at the University of Illinois, used a classroom analogy to explain Amazon’s technical analysis in an email to CNN. Suppose two students, one who works quickly, the other who works more slowly, are asked to collaborate on a shared notebook.

The slowest student “pays attention in short bursts, but his or her work may conflict or contradict that of the faster student,” he wrote. At the same time, the faster student may “constantly try to ‘fix’ things quickly” and wipe out the slower students. the student’s work because it is obsolete.

“The result… a blank page (or a crossed out page) in the laboratory notebook, when the professor comes to inspect it,” he writes.

This “empty page” brought down AWS’s DynamoDB database, creating a cascading effect that impacted other AWS services like EC2, which offers virtual servers for developing and deploying applications, and Network Load Balancer, which manages requests on the network. When DynamoDB came back online, EC2 tried to bring all of its servers back online at once and couldn’t keep up.

Amazon is making a number of changes to its systems following the outage, including fixing the “race condition scenario,” which caused both systems to overwrite each other’s work in the first place, and adding an additional test suite for its EC2 service.

Outages like Monday’s, while rare, are just a reality, Gupta said. But what matters is how these issues are addressed.

“Large-scale outages like this just happen. There’s nothing you can do to prevent it, just like how people get sick,” Gupta told CNN by phone. “But I think the way the company responds to outages and keeps customers informed is really, really key.”