3 ms·
Didn’t this happen already a few years ago? massive S3 outage happened in 2017. Some connected devices were very confused. Mayhem..of a not-minor variety.. but
by jelliclesfarm 6y ago
Didn’t this happen already a few years ago? massive S3 outage happened in 2017. Some connected devices were very confused. Mayhem..of a not-minor variety.. but also a tad hilarious:
https://www.theverge.com/2017/3/2/14792442/amazon-s3-outage-cause-typo-internet-server https://www.theverge.com/2017/3/2/14792442/amazon-s3-outage-...
[..] On Tuesday morning, members of the S3 team were debugging the billing system. As part of that, the team needed to take a small number of servers offline. “Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended,” Amazon said. “The servers that were inadvertently removed supported two other S3 subsystems.”
The subsystems were important. One of them “manages the metadata and location information of all S3 objects in the region,” Amazon said. Without it, services that depend on it couldn’t perform basic data retrieval and storage tasks.[..]
[..] While S3 was down, a variety of other Amazon web services stopped functioning, including Amazon’s Elastic Compute Cloud (EC2), which is also popular with internet companies that need to rapidly expand their storage.
Amazon said S3 was designed to be able to handle losing a few servers. What it had more trouble handling was the massive restart. “S3 has experienced massive growth over the last several years and the process of restarting these services and running the necessary safety checks to validate the integrity of the metadata took longer than expected,” the company said.[..]
And this!!
[..] It’s also making a change to the AWS Service Health Dashboard. During the outage, the dashboard embarrassingly showed all services running green, because the dashboard itself was dependent on S3. The next time S3 goes down, the dashboard should function properly, the company said.[..]