4 ms·

Unlikely. S3 was publicly rebuilt in the wake of the 2017 S3pocalypse.

by QuinnyPig 7y ago

bagels 7y ago
What does this refer to?
- x0x0 7y ago
  Some S3 eng accidentally dropped a big chunk of the servers that were the s3 equivalent of an hdfs nameserver, ie mapping blob name to location info, as part of an unrelated config change. While attempting to recover, the s3 team discovered and/or decided the nameserver needed a full restart. That's when they discovered the info in the nameserver had grown so large since the last full restart years previous that it took far longer than expected to restart the nameserver. Right around that point in time my guess is they realized just how shit their morning was going to be. And their afternoon. Somewhere in there, they realized that their health dashboard depended on s3 working. Though to be fair, as an aws customer, we -- along with the rest of internet -- were well aware that stuff was badly broken. I feel terribly for whoever did this, because IIRC, he or she just fat fingered part of a command in a standard playbook, and the config script had no safeguards. I personally took down a company you've heard of in the exact same way; I knocked all pops off the internet because the config script had a hard requirement around certain values that was neither communicated to me nor checked. And I was trying to figure out wtf I did to a system that I was not particularly familiar with while receiving forwarded texts from the CEO about cascading datacenter down alerts.
  - mywittyname 7y ago
    If you don't mind me asking, what was the punishment for what you did?
    - x0x0 7y ago
      Just taking the company dark and being personally embarrassed. There was no punishment, though there was a lot of teasing. Also spending 4-ish weeks cleaning up the mess that was made.
- ms512 7y ago
  Likely referring to the February 28th, 2017 S3 Service Disruption in the Northern Virginia (US-EAST-1) Region, for which Amazon published a postmortem at https://aws.amazon.com/message/41926/ https://aws.amazon.com/message/41926/

Unlikely. S3 was publicly rebuilt in the wake of the 2017 S3pocalypse.

bagels 7y ago

What does this refer to?

x0x0 7y ago
Some S3 eng accidentally dropped a big chunk of the servers that were the s3 equivalent of an hdfs nameserver, ie mapping blob name to location info, as part of an unrelated config change. While attempting to recover, the s3 team discovered and/or decided the nameserver needed a full restart. That's when they discovered the info in the nameserver had grown so large since the last full restart years previous that it took far longer than expected to restart the nameserver. Right around that point in time my guess is they realized just how shit their morning was going to be. And their afternoon. Somewhere in there, they realized that their health dashboard depended on s3 working. Though to be fair, as an aws customer, we -- along with the rest of internet -- were well aware that stuff was badly broken. I feel terribly for whoever did this, because IIRC, he or she just fat fingered part of a command in a standard playbook, and the config script had no safeguards. I personally took down a company you've heard of in the exact same way; I knocked all pops off the internet because the config script had a hard requirement around certain values that was neither communicated to me nor checked. And I was trying to figure out wtf I did to a system that I was not particularly familiar with while receiving forwarded texts from the CEO about cascading datacenter down alerts.
- mywittyname 7y ago
  If you don't mind me asking, what was the punishment for what you did?
  - x0x0 7y ago
    Just taking the company dark and being personally embarrassed. There was no punishment, though there was a lot of teasing. Also spending 4-ish weeks cleaning up the mess that was made.
ms512 7y ago
Likely referring to the February 28th, 2017 S3 Service Disruption in the Northern Virginia (US-EAST-1) Region, for which Amazon published a postmortem at https://aws.amazon.com/message/41926/ https://aws.amazon.com/message/41926/