3 ms·

Reminds me of when s3 went down in 2017, and took a chunk of the internet with it (to include their own status dashboard) due to someone entering a command wron

Reminds me of when s3 went down in 2017, and took a chunk of the internet with it (to include their own status dashboard) due to someone entering a command wrong. https://aws.amazon.com/message/41926/ https://aws.amazon.com/message/41926/

mattl 6y ago
Other than websites and maybe some web-ish apps not too much else would be affected by S3 outage, a DNS outage would affect more of the Internet vs the web.
- megous 6y ago
  All of Google was gone from my country due to bungled router configuration for half a day some time ago. Now that was fun to watch. Everything from 8.8.8.8 to all their backend APIs for android, to all their web properties, went poof.
  - mattl 6y ago
    Yikes. Time to use a few different providers?
- CreepGin 6y ago
  Websites and web-ish apps are not a chunk of the internet? Also there are tons of non-web-ish apps that depend on S3.
  - dylan604 6y ago
    i work on multiple projects that are 100% data crunching with content stored on S3 and accessed solely by ec2 instances. there is no UI or public access. yet if S3 went down, we'd be up a certain creek without a certain self-propelling device and a big whole in the boat without a bucket (physically and in the cloud)
- Bedon292 6y ago
  Maybe "the internet" was a bit too generic. But it destroyed a lot of tech worker productivity for the day at least. Slack, GitHub, GitLab, BitBucket, Docker, etc all had issues. https://venturebeat.com/2017/02/28/aws-is-investigating-s3-issues-affecting-quora-slack-trello/ https://venturebeat.com/2017/02/28/aws-is-investigating-s3-i...
- eru 6y ago
  There's actually quite a few enterprise client of S3. Lots in finance, too.
ckozlowski 6y ago
I was just coming to see if someone mentioned this! I work for AWS, and it's both surprising and heartening to see how often I see this anecdote get repeated, and many times by us to our own customers. We had teams looking for weeks at all of our other services for other instances of this. It was a good operational lesson and one I'm happy to see shared still. If a command will let you perform a self-inflicted wound like that without checks, then it's time to review that command. Err on the side of caution and bias towards minimizing blast radius even if it means sacrificing some speed. "Move fast and break things" may be true at times, but not when the "*" character is ever involved. =)
- Bedon292 6y ago
  Yeah its an interesting one. I tend to think of it in the context of, well you screwed up and deleted the wrong instance, but at least it wasn't that bad. I also think about the response to it. Sure it sucked that it happened, and someone was probably feeling really bad about it. But it was a learning opportunity as well. Not just for them, or AWS, but for everyone in tech. Put those safety nets in place.
- josmala 6y ago
  It is my personal opinion that you don't need "*" character ever there are other alternatives available always. And if avoiding that character makes it far safer I'm all for it. rm -rf /+(?) If anyone not understanding the joke, + character matches 1 or more instances, and (?) matches any one character. And / is start of entire filesystem. rm is remove. -rf (recursive, never ask verification). So it removes every file that you have rights to remove. You can also do same thing accidentally by proxy. When my first computing teacher asked entire class to write anything to terminal to show that you had to write specific commands and computers don't understand normal language. Then she told to press enter. I asked her if she was sure. She asked why. I said that I wrote "format c:" . I have never seen teacher walk so fast in a classroom. I think that was last time she used that line in the classroom.

mattl 6y ago

Other than websites and maybe some web-ish apps not too much else would be affected by S3 outage, a DNS outage would affect more of the Internet vs the web.

megous 6y ago
All of Google was gone from my country due to bungled router configuration for half a day some time ago. Now that was fun to watch. Everything from 8.8.8.8 to all their backend APIs for android, to all their web properties, went poof.
- mattl 6y ago
  Yikes. Time to use a few different providers?
CreepGin 6y ago
Websites and web-ish apps are not a chunk of the internet? Also there are tons of non-web-ish apps that depend on S3.
- dylan604 6y ago
  i work on multiple projects that are 100% data crunching with content stored on S3 and accessed solely by ec2 instances. there is no UI or public access. yet if S3 went down, we'd be up a certain creek without a certain self-propelling device and a big whole in the boat without a bucket (physically and in the cloud)
Bedon292 6y ago
Maybe "the internet" was a bit too generic. But it destroyed a lot of tech worker productivity for the day at least. Slack, GitHub, GitLab, BitBucket, Docker, etc all had issues. https://venturebeat.com/2017/02/28/aws-is-investigating-s3-issues-affecting-quora-slack-trello/ https://venturebeat.com/2017/02/28/aws-is-investigating-s3-i...
eru 6y ago
There's actually quite a few enterprise client of S3. Lots in finance, too.

ckozlowski 6y ago

I was just coming to see if someone mentioned this! I work for AWS, and it's both surprising and heartening to see how often I see this anecdote get repeated, and many times by us to our own customers. We had teams looking for weeks at all of our other services for other instances of this. It was a good operational lesson and one I'm happy to see shared still. If a command will let you perform a self-inflicted wound like that without checks, then it's time to review that command. Err on the side of caution and bias towards minimizing blast radius even if it means sacrificing some speed. "Move fast and break things" may be true at times, but not when the "*" character is ever involved. =)

Bedon292 6y ago
Yeah its an interesting one. I tend to think of it in the context of, well you screwed up and deleted the wrong instance, but at least it wasn't that bad. I also think about the response to it. Sure it sucked that it happened, and someone was probably feeling really bad about it. But it was a learning opportunity as well. Not just for them, or AWS, but for everyone in tech. Put those safety nets in place.
josmala 6y ago
It is my personal opinion that you don't need "*" character ever there are other alternatives available always. And if avoiding that character makes it far safer I'm all for it. rm -rf /+(?) If anyone not understanding the joke, + character matches 1 or more instances, and (?) matches any one character. And / is start of entire filesystem. rm is remove. -rf (recursive, never ask verification). So it removes every file that you have rights to remove. You can also do same thing accidentally by proxy. When my first computing teacher asked entire class to write anything to terminal to show that you had to write specific commands and computers don't understand normal language. Then she told to press enter. I asked her if she was sure. She asked why. I said that I wrote "format c:" . I have never seen teacher walk so fast in a classroom. I think that was last time she used that line in the classroom.