24 ms·
Summary of the Amazon S3 Service Disruption
- ct0 10y agoI wouldn't want to be the person who wrote the wrong command! Sheesh.
- tyingq 10y agoMeh, that's a process problem, not a people problem. Playbooks that have you retype commands with complex options, with no confirmation, etc, are inviting that sort of thing.
- makira 10y agoThat person is the least likely to make that mistake again!
- oblio 10y agoNo! Off with this head!!! On a more serious note, if you've never done something like this, you haven't had enough interesting projects. I've had a decent career and I still managed to: * re-deploy the current application version in all our data centers, instead of the new version, in a period when our deployment wasn't a 0-downtime one * rename all the Jenkins jobs on the server to the same name, thus deleting hundreds of Jenkins jobs in one fell swoop "Let him who is without sin cast the first stone" and all that :)
- ckozlowski 10y agoThere's two things I've come to believe in my IT career regarding operations: 1. No organization anywhere is a paragon of excellence, and everyone can benefit from improvement. 2. Every organization is made up of humans just like you. With all that entails. Some things which seem blatantly obvious after the fact are easily overlooked when the pressure to deliver is high and other issues are taking precedence. I completely agree with your statement. In fact, when I do interviews, one of my favorite and most insightful questions to ask is, basically, "tell me about a time you screwed the pooch." If they don't have a story and they worked in ops, then it can suggest they didn't really do much. The really sharp ones I've interviewed have a good story or two (and can tell it in excruciating detail. =) * At a prior company I once tried appending to the list of NFS exports, but dropped the "no-root-squash" option, and instantly denied write permissions to our entire VMware farm. You can imagine what then happened to all of the VMs for this mission critical customer. =P
- tetha 10y agoAs I keep saying, there are people who screw up big time and people who are too scared to touch the system. I managed to gun down three productive clusters by deploying NTP by accident along with a tiny change. Kerpow, 12 minutes of downtime, full network outage due to DHCP and such. Great fun.
- Alupis 10y agoReminds me of the story about Bob Hoover.[1] He took off in his piston engine plane, only to lose power during the climb and was forced to make a crash landing. It turned out the airplane was fueled with jet fuel instead of regular gasoline (the ground crewman mistakenly thought the plane was a turbo prop). Instead of yelling at or firing the ground crewman, Hoover had this to say[2]: "There isn't a man alive who hasn't made a mistake. But I'm positive you'll never make this mistake again. That's why I want to make sure that you're the only one to refuel my plane tomorrow. I won't let anyone else on the field touch it." [1] https://www.aopa.org/news-and-media/all-news/2014/july/pilot/pilots https://www.aopa.org/news-and-media/all-news/2014/july/pilot... [2] http://www.squawkpoint.com/2014/01/criticism/ http://www.squawkpoint.com/2014/01/criticism/
- dahdum 10y agoAt these scales it's the fault of the system, not the individual, so hopefully they don't come down hard on them.
- notheguyouthink 10y agoAgreed. They also seemed to acknowledge that in the post, as they mentioned improving the tool to not allow such destructive options.
- joezydeco 10y agoYou mean like range checking the input parameters to the command? =) SHUT DOWN S3? ARE YOU SURE? (y/N) :
- unit91 10y agoMore like: SHUT DOWN S3? ARE YOU SURE? ("I'm absolutely positive."/n) :
- fixermark 10y agoWere it so easy. Likely more like "Shut down 73 servers? Are you sure? (y/N)": "Seventy-three? Wow, I hadn't realized our system grew that much. Probably a new backend dependency got added that I'm not familiar with yet; I'll look into it later." (Y)
- TheDauthi 10y agoEvery sysadmin at my previous job (a Fortune 500) would stop the moment that number is off the expected by a few machines, just long enough to verify it's correct. That may be due to having made a mistake like this once. I know that's true in several other large shops, as well. Source: I'm was the one the one they would call for our team... usually at 4:00 AM because one of our team members (which was also frequently me) didn't document something correctly.
- flamedoge 10y ago
- Johnny555 10y agoI brought down our production system after a typo in a command once... the dev team took the blame for allowing an illegal parameter to bring down the system.
- alexvy86 10y agoKudos to the dev team for that, I think most people wouldn't own that kind of issue
- Johnny555 10y agoIt was a very well-run engineering department where taking blame was not a career ending decision. I took full blame for the typo (at 3am trying to resolve a customer issue), but the dev team accepted full responsibility for letting it take down the system. Every mistake was used as a learning opportunity to ensure that the same and similar mistakes can't be repeated.
- unit91 10y agoI did once, too. Unfortunately, it was company of 4 people so I was also the dev team. :-) Never did it again though (yet).
- vhost- 10y agoMe too. I accidentally SSH'd into the wrong data center and ran a for loop restarting every database server once. Fixed that by putting the DC domain in red on the prompt.
- omni 10y agoThey just spent millions of dollars training that person to never make this mistake again, should definitely keep them around.
- DaiPlusPlus 10y agoThe wording of the article implies Amazon is shifting the blame entirely on the individual who typo'd: they indemnify themselves with "an authorized S3 team member using an established playbook..." ("don't blame us, our process is perfect!") There are process-fixes for this, such as requiring a two-person rule when at a production shell and modifying tooling to detect potentially unintentional commands (e.g. a SQL UPDATE without a WHERE) - but given what I know about Amazon's internal practices (i.e. the brutality) it wouldn't surprise me if they did terminate the unfortunate operator - not because they want to, but because AWS simply has too many large-scale customers who would demand immediate action like that.
- unit91 10y agoMaybe so. I didn't take it that way. I took it as, "nobody panic, we weren't hacked."
- fixermark 10y agoIt's both. This isn't "our system was compromised by attack;" it's "SNAFU." Everyone who's had operations experience knows that there will be, as time approaches infinity, more than zero SNAFU. That's why companies offer five nines of uptime, not 100% uptime.
- ckozlowski 10y agoNot at all. The "an authorized S3 team member using an established playbook..." is to establish that: 1. They were authorized to be doing it. 2. They were following an established process, and not winging it. The reason for that wording is to illustrate that operational changes are made in accordance with established CM. Mistakes happen. The system didn't catch the error. That's why the mitigation is "fix the system", and not "fire the team member". =)
- asciimo 10y agoI would love to hear their recollection of the day. Would make me feel better about my mistakes.
- orthecreedence 10y agoTLDR; Someone on the team ran a command by mistake that took everything down. Good, detailed description. It happens. Out of all of Amazon's offerings, I still love S3 the most.
- spamlord 10y agoCleversafe is much cheaper, higher quality, more flexible, etc.
- cflewis 10y ago"It happens" is the only reasonable takeaway you can get from a postmortem like this. My worry is that people read it and go "I am aghast that such a command can be run!" without knowing that little commands like that are run numerous times a day without incident. The only thing I read in there and go "hmmm" is that it took quite that long for the S3 service to recover, and that the status page wasn't hosted on someone that doesn't have an S3 dependency. That's just a plain "doh" moment :)
- ProAm 10y agoPeople need to realize when they go to the cloud it's not that 'it happens', it's that it will happen, and you have no ability to do anything about it. Fact of life and risk management.
- fixermark 10y ago... and it's a different risk from self-hosting, but self-hosting provides all sorts of similar issues (such as when you do this to yourself, the cost is now coming out of your pocket, not Amazon's, to employ software engineers to harden your scripts against making the same mistake twice).
- thr0waway1239 10y agoNot to mention, Amazon is catching the long tail of cloud failure like Google is catching the long tail of search keywords. They can now say with a somewhat straight face - "You know all those scripts you run to keep everything up? We have figured out many, many more possible ways for them to fail than you probably ever will, and we have added more layers of safeguards than you can even imagine."
- cnorthwood 10y agoI'm surprised how transparent this is, I can find Amazon often a bit opaque when dealing with issues.
- snewman 10y agoI've found them to be very opaque in most contexts, but for major outages (which have been rare), they do have a history of solid postmortems.
- DigitalBison 10y agoThe public postmortem from the big DynamoDB outage in 2015(?) is a great example I think: https://aws.amazon.com/message/5467D2/ https://aws.amazon.com/message/5467D2/
- deleted 10y ago[deleted]
- edblarney 10y agoThey have no choice in this situation. CEO's all over the world just realized that they can't only depend on S3, and they might have to double up on their infrastructure and have a parallel env. on Azure or Google as well.
- eric_h 10y agoNot necessarily - our infra's hosted on us-west-2 and we had no downtime. So while it is perhaps necessary to be multi-regional in Amazon, you wouldn't necessarily have to go multi-provider.
- ajross 10y ago> At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. It remains amazing to me that even with all the layers of automation, the root cause of most serious deployment problems remain some variant of a fat fingered user.
- danielodio 10y agoSounds like AWS needs Spinnaker for easy rollbacks! https://news.ycombinator.com/item?id=13776456 https://news.ycombinator.com/item?id=13776456
- NikolaeVarius 10y agoAutomation doesn't just allow you to create/fix things faster. It also allows you to break things faster.
- kevin2r 10y agoWe may think that an automated system requires less understanding of it in order to operate it. But from the other point of view, you have to know what you are doing, consequences of even an small change are big. This is one of the things that happens with windows, getting up a server is so easy, that people believe that they don't have to understand what's under the hood, and then, we get a lot of miss-configuration and operational issues.
- pm90 10y agoWonder what happened to the poor slob who did that. He/She was unauthorized AND caused a pretty serious outage... EDIT: derp, my bad, I read as "unauthorized" which was "authorized".
- apetresc 10y agoThe article specifically said "authorized".
- jph00 10y ago> From the beginning of this event until 11:37AM PST, we were unable to update the individual services’ status on the AWS Service Health Dashboard (SHD) because of a dependency the SHD administration console has on Amazon S3. Ensuring that your status dashboard doesn't depend on the thing it's monitoring is probably the first thing you should think about when designing your status system. This doesn't fill me with confidence about how the rest of the system is designed, frankly...
- tlack 10y agoAnd apparently they had never tried rebooting some of the most important parts of that system. Just when you start to think that someone's really gotten it right you come to learn they're just fumbling around in the dark like everyone else.
- jonhohle 10y agoMy interpretation of this is that the indexing system was resilient to lost of a certain amount of capacity (probably around ⅓ + 1 host). As a guess, the indexing system probably used some form of consensus (e.g. paxos) which has had an active leader for years. Deployments stay within that capacity constraint, so while hosts have been restarted and replaced (data center migrations, hardware lease expiration, failures, upgrades, etc.), they may have not recently run into a situation where quorum wasn't available for a partition, especially at the scale of restarting the entire fleet. Since restarting the entire fleet would incur downtime of all relevant S3 operations, it's unlikely that it was something ever intentionally done in production (and they may or may not have run that scenario in other environments). Source: I used to run several large scale services at Amazon.
- pyre 10y agoTo put what @jonhohle said another way, Amazon had probably never brought up the entirety of S3 from Zero to Production-Ready on in a production environment before. I wouldn't necessarily classify this as "fumbling around in the dark." Perhaps they should have tested this in a simulated environment, but (to be fair) on a distributed fault-tolerant system, it probably wasn't a top-priority situation to test.
- aorloff 10y agoWhat's missing is addressing the problems with their status page system, and how we all had to use Hacker News and other sources to confirm that US East was borked.
- Scramblejams 10y agoNo, this is addressed: >We understand that the SHD provides important visibility to our customers during operational events and we have changed the SHD administration console to run across multiple AWS regions.
- yellow_postit 10y agoWhich is fine until those regions go down. A status page, in my mind, should have a fallback on a completely different service provider.
- danesparza 10y agoThis was addressed in the post: "From the beginning of this event until 11:37AM PST, we were unable to update the individual services’ status on the AWS Service Health Dashboard (SHD) because of a dependency the SHD administration console has on Amazon S3. Instead, we used the AWS Twitter feed (@AWSCloud) and SHD banner text to communicate status until we were able to update the individual services’ status on the SHD. We understand that the SHD provides important visibility to our customers during operational events and we have changed the SHD administration console to run across multiple AWS regions."
- certifiedloud 10y agoThis is a bit off topic. The use of the word "playbook" suggests to me that they use Ansible to help manage S3. I wonder if that is the case, or if it's just internal lingo that means "a script". Unless there is some other configuration management system that uses the word playbook that I'm not aware of.
- JoshTriplett 10y ago"playbook" is a relatively common term for "documented step-by-step procedure for specific tasks". Effectively, a script with #!/bin/human at the top.
- p4lindromica 10y agoAlso known as a runbook
- hehheh 10y agoOr if you want to get real old school a checklist.
- dgemm 10y agoIt's a document with step-by-step instructions for the operator to follow.
- hueving 10y agoThe term "playbook" just originates from sports as a documentation of what you are going to do for for a specific task (i.e. a "play" in sports).
- jordigh 10y agoI'm pretty sure the sports meaning comes from the theathre meaning. Ansible has other bits of theatre metaphor in it, such as "roles" and "scripts".
- dorianm 10y agoSame as that db1 / db2 for GitLab, naming things is pretty important (E.g production / staging, production-us-east-1-db-560 etc.)
- conorh 10y agoThis part is also interesting: > While this is an operation that we have relied on to maintain our systems since the launch of S3, we have not completely restarted the index subsystem or the placement subsystem in our larger regions for many years. These sorts of things make me understand why the Netflix "Chaos Gorilla" style of operating is so important. As they say in this post: > We build our systems with the assumption that things will occasionally fail Failure at every level has to be simulated pretty often to understand how to handle it, and it is a really difficult problem to solve well.
- urda 10y agoI think you meant "Chaos Monkey" [1]. [1] https://github.com/Netflix/chaosmonkey https://github.com/Netflix/chaosmonkey
- deleted 10y ago[deleted]
- tantalor 10y agoNo, Chaos Gorilla is similar to Chaos Monkey, but simulates an outage of an entire Amazon availability zone http://techblog.netflix.com/2011/07/netflix-simian-army.html http://techblog.netflix.com/2011/07/netflix-simian-army.html
- ceejayoz 10y agohttp://techblog.netflix.com/2011/07/netflix-simian-army.html http://techblog.netflix.com/2011/07/netflix-simian-army.html > Chaos Gorilla is similar to Chaos Monkey, but simulates an outage of an entire Amazon availability zone. We want to verify that our services automatically re-balance to the functional availability zones without user-visible impact or manual intervention.
- deleted 10y ago[deleted]
- deleted 10y ago[deleted]
- Dangeranger 10y agoSo this is the second high profile outage in the last month caused by a simple command line mistake. > Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. If I would have guessed anyone could prevent mistakes like this from propagating it would be AWS. It points to just how easy it is to make these errors. I am sure that the SRE who made this mistake is amazing and competent and just had one bad moment. While I hope that AWS would be as understanding as Gitlab, I doubt the outcome is the same.
- imsofuture 10y agoAmazon has the wherewithal to not freaking publicly name their actually human employee, so I'd imagine their culture around outages is probably a lot more healthy.
- Dangeranger 10y agoWell to be fair, he named himself within his notes and did not object to the public nature of the disclosure. I agree with your sentiment though that names should not be included within postmortems in the general case.
- draw_down 10y agoI couldn't believe that when I saw it in that incident report. Just crazy.
- rolandog 10y ago> While I hope that AWS would be as understanding as Gitlab, I doubt the outcome is the same. I hadn't thought about how 'prone to forgive' an organization is. Should there be policies in place as to how big a mistake you are allowed to commit? Something akin to: ```if(log10(cost_of_mistake) >= 5): worker.fire() ```
- andrewguenther 10y agoNothing is going to happen to the engineer that did this, other than embarrassment and probably a couple jokes once sufficient time has passed. Amazon has a strong culture around blaming the process, not the person. The failure here wasn't that the engineer ran the command, it was that the engineer was able to run the command. Source: AWS employee
- dsr_ 10y ago" we have not completely restarted the index subsystem or the placement subsystem in our larger regions for many years. S3 has experienced massive growth over the last several years and the process of restarting these services and running the necessary safety checks to validate the integrity of the metadata took longer than expected" This is analogous to "we needed to fsck, and nobody realized how long that would take".
- edutechnion 10y agoFor the many of us who have built businesses dependent on S3, is anyone else surprised at a few assumptions embedded here? * "authorized S3 team member" -- how did this team member acquire these elevated privs? * Running playbooks is done by one member without a second set of eyes or approval? * "we have not completely restarted the index subsystem or the placement subsystem in our larger regions for many years" The good news: * "The S3 team had planned further partitioning of the index subsystem later this year. We are reprioritizing that work to begin immediately." The truly embarrassing that everyone has known about for years is the status page: * "we were unable to update the individual services’ status on the AWS Service Health Dashboard " When there is a wildly-popular Chrome plugin to fix your page ("Real AWS Status") you would think a company as responsive as AWS would have fixed this years ago.
- icelancer 10y ago>Unauthorized S3 team member This is not in the post.
- dmoy 10y ago) An authorized, not unauthorized. They sound almost identical, English language is... Yea :/ ) If it's a playbook for something with minimal intended impact sure. The issue is that the tooling had larger capabilities than should be. ) Yes that seems like a major, major problem.
- deleted 10y ago[deleted]
- dmd 10y agoIt says authorized, not unauthorized.
- jhull 10y agothe post mortem says an authorized team member
- deleted 10y ago[deleted]
- magd 10y agoOh, that interview question. “Tell me about something you broke in your last job"
- ckozlowski 10y agoIt's my favorite. =D
- BookmarkSaver 10y ago"The Internet."
- swalsh 10y agoI once changed a piece of code that was referenced by every page on the customer facing site (10's millions of visits a day) to use a new function that someone had previously written (and was called on 1 page in the site). I mistakenly didn't really look to closely at the implementation of the function, and didn't realize how badly it's caching strategy was designed. When this code was deployed it instantly caused a thundering herd on our cache servers bringing the site down for about ~40 seconds. I felt TERRIBLE about it.
- plandis 10y agoMy worst was that one time I accidentally took down some services that were essential for order processing via an unrelated, large stress test. My test accidentally consumed a large amount of bandwidth saturating the links to a shared service. I did all this while sitting 2 feet from a print out of "The 8 fallacies of distributed systems". Bandwidth is indeed not infinite, can confirm.
- rconti 10y agoPaged for "$HOSTNAME is down!" Usual checks to access $HOSTNAME failed Rushed to office at 6am before important process was about to run that needed that host. Plugged in keyboard+monitor, dead screen, nothing. Physically power-cycled server. Stood in front of monitor+keyboard. It occurred to me it was taking longer than expected to show POST screen. About that time, I got a page saying $ACTUALHOSTNAME is down. Walk around to the back of the racks. The monitor cable had come detached from the cable extender that I plugged into the server. I had never plugged the monitor in at all, just the extension. The server wasn't down in the first place, it just lost a virtual interface, which I was paged for, and stupidly tested that virtual interface instead of the REAL name/IP. And then I raced to the office just so that I could cause an outage.
- savanaly 10y agoNot as interesting an explanation as I was hoping for. Someone accidentally typed "delete 100 nodes" instead of "delete 10 nodes" or something. It sounds like the weakness in the process is that the tool they were using permitted destructive operations like that. The passage that stuck out to me: "in this instance, the tool used allowed too much capacity to be removed too quickly. We have modified this tool to remove capacity more slowly and added safeguards to prevent capacity from being removed when it will take any subsystem below its minimum required capacity level." At the organizational level, I guess it wasn't rated as all that likely that someone would try to remove capacity that would take a subsystem below its minimum. Building in a safeguard now makes sense as this new data point probably indicates that the likelihood of accidental deletion is higher than they had estimated.
- pinko 10y agoI always wonder about unintended consequences of this sort of thing. Like someday there will be a worm about to rampage through their servers and someone says, "take them all offline now!" and the answer is, "we can't because of the throttle safeguard we put in place after incident XYZ, it will be about 17 hours..."
- josephorjoe 10y agothen you need to "sudo take them all offline now!"
- rsynnott 10y ago> While this is an operation that we have relied on to maintain our systems since the launch of S3, we have not completely restarted the index subsystem or the placement subsystem in our larger regions for many years. This is the bit that'd worry me most; you'd think they'd be testing this.
- illumin8 10y agoA complete restart of the index subsystem would require downtime. Note: they are not saying those servers have never been restarted - it's highly likely they get restarted regularly. But, a complete restart of the index subsystem implies that you shut everything down first and restart it all at once, which is what was forced to happen two days ago.
- kwisatzh 10y agoWhy can't the index subsystem itself have a backup then? When the primary subsystem is being restarted/rebuilt, the secondary takes over.
- grogenaut 10y agothats kind of like asking why git doesn't have a single backup... it's a distributed system, there's not just one backup, there are lots of little partial backups.
- CodeWriter23 10y agoI'm just going to call this "PEBKAC at scale"
- nsgoetz 10y agoThis makes me want to write a program that would ask users to confirm commands if it thinks they are running a known playbook and deviating from it. Does anyone know if a tool like that exists?
- mrep 10y agoNot sure, but my company's fleet wide root scripts confirm first the exact command you want to run, then run on 1 host first and output the the full logs for you to inspect/confirm, and then finally start the full fleet wide run after you have confirmed the expected result of your output. They also output the full logs of across the entire fleet once your fleet wide script is run.
- matt_wulfeck 10y ago>Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. We have geo-distributed systems. Load balancing and automatic failover. We agonize over edge cases that might cause issues. We build robust systems. At the end of he day reliability -- a lot like security -- is most affected by the human factor.
- OhHeyItsE 10y agoI find this refreshingly candid; human, even, for AWS.
- matt_wulfeck 10y agoRemember folks, automate your systems but never forget to add sanity checks.
- DanBlake 10y agoSeems like something like Chaos monkey should have been able to predict and mitigate a issue like this. Im actually curious if anyone uses it at all- Curious if anyone in here at a large company (over 500 employees) has it deployed or not.
- atrudeau 10y agoAre customers going to receive any kind of future credit because of this? Would be a nice band-aid after such a hard smack on the head.
- deleted 10y ago[deleted]
- fixermark 10y agoWhatever the market will bear. ;)
- prh8 10y agoFor as much as people jumped all over Gitlab last month, this seems remarkably similar in terms of preparedness for accidental and unanticipated failure.
- erikbye 10y agoIn the case of GitLab weren't they actually close to losing a lot of data? Was any data lost (or chance of loss) in this incident?
- user15672 10y agoBeyond a typing mistake, it's not really very similar. The Gitlab incident was one avoidable problem after another, ending with a giant WTF when they found out that no-one had even tested the backups were working. This is a case of someone slipping on the keyboard, removing more capacity than intended and the recovery process taking longer than expected. The process actually seems to be working (to a given value of working), but the amount of downtime was way above acceptable. They've already put more safeguards into the tooling to prevent the situation from happening again. S3 is also orders of magnitude more complex than Gitlabs infrastructure, so while the amount of time the outage lasted for is not acceptable, it does show that they at least have working processes for critical situations that allow them to get back in service within a day, which is pretty impressive.
- hyperanthony 10y agoSomething that wasn't addressed -- there seems to be an architectural issue with ELB where ELBs with S3 access logs enabled had instances fail ELB health checks, presumably while the S3 API was returning 5XX. My load balancers in us-east-1 without access logs enabled were fine throughout this event. Has there been any word on this?
- Johnny555 10y agoI think it comes down to how important your ELB logs are -- if they are important enough that you don't want to allow traffic without logs (i.e. if you're using them for some sort of auditing/compliance), then failing when it can't write the logs seems like the right choice.
- hyperanthony 10y agoThanks, that is a fair perspective. In our case we're using ELB logs as a redundant trace and it isn't critical that our traffic stops if the access logs fail. It would be nice if this behavior became a toggle in ELB settings, but think we can set something up to disable access logs programatically if we start seeing S3 issues.
- ctrlrsf 10y agoGood luck with this. We tried to make changes yesterday to mitigate impact but AWS console was also affected. Was hesitant to make API calls for the changes since we werent sure they would complete successfully given all the services we found actually depended on S3 internally.
- cagataygurturk 10y agoSounds so chernobyl.
- Piskvorrr 10y agoBecause it was resolved in a few hours without any further fallout? (Pun definitely intended)
- djhworld 10y agoReally pleased to see this, it's good to see an organisation that's being transparent (and maybe given us a little peek under the hood of how S3 is architected) and most importantly they seem quite humbled. It would be easy for an arrogant organisation to fire or negatively impact the person that made the mistake, I hope Amazon don't fall into that trap and focus instead on learning from what happened, closing the book and move on.
- nissimk 10y agoI keep being reminded of something I read recently that made me feel uneasy about google's cloud spanner [1]: the most important one is that Spanner runs on Google’s private network. Unlike most wide-area networks, and especially the public internet, Google controls the entire network and thus can ensure redundancy of hardware and paths, and can also control upgrades and operations in general. Fibers will still be cut, and equipment will fail, but the overall system remains quite robust. It also took years of operational improvements to get to this point. For much of the last decade, Google has improved its redundancy, its fault containment and, above all, its processes for evolution. We found that the network contributed less than 10% of Spanner’s already rare outages. But when it fails it's going to be epic! [1] https://cloudplatform.googleblog.com/2017/02/inside-Cloud-Spanner-and-the-CAP-Theorem.html https://cloudplatform.googleblog.com/2017/02/inside-Cloud-Sp...
- fixermark 10y agoGoogle takes active steps to confirm that their fallback systems for mitigating failures work. http://queue.acm.org/detail.cfm?id=2371516 http://queue.acm.org/detail.cfm?id=2371516
- dap 10y ago> Removing a significant portion of the capacity caused each of these systems to require a full restart. I'd be interested to understand why a cold restart was needed in the first place. That seems like kind of a big deal. I can understand many reasons why it might be necessary, but that seems like one of the issues that's important to address.
- perlgeek 10y agoPossibly a consensus algorithm that refuses writes when it detects itself in a minority, because it think it's in the smaller part of a split-brain scenario. In this case, throwing away and then re-provisioning the split-off nodes is a viable approach.
- losteverything 10y ago"I did." That was CEO Robert Allen's response when the AT&T network collapsed [1] on January 15, 1990 He was asked who made the mistake. I can't imagine any CEO now a days making a similar statement. [1] http://users.csc.calpoly.edu/~jdalbey/SWE/Papers/att_collapse.html http://users.csc.calpoly.edu/~jdalbey/SWE/Papers/att_collaps...
- mcherm 10y agoFascinating! Do you know of any source that documents the claim that Robert Allen made this statement?
- losteverything 10y agoI'll try. It was a dark day. We all watched the news and I recall him saying that. The specific quote I don't remember but it was something like " you can consider that I did." I think he was asked what will happen to the person that caused it and who is that person. Everyone knew right away this had to be human error. Right away. Switches simply had too much redundancy. It was big then and not sure if I can locate a video.
- pcthrowaway 10y agoNot sure if there's a video, but I found an article with the exact quote: > As far as our customers are concerned, I did it. http://www.upi.com/Archives/1990/01/16/ATT-pinpoints-cause-of-long-distance-line-crash/2432632466000/ http://www.upi.com/Archives/1990/01/16/ATT-pinpoints-cause-o...
- losteverything 10y agoThanks. I got the gist at least. I admired him and his answer at the time. The culture was quite professional and blame really never existed.
- praneshp 10y agoHe doesn't seem to have been a great CEO though[0]. Very very sorry for the shitty link, his Wiki page was bare. [0]: http://www.cnbc.com/2009/04/30/Portfolios-Worst-American-CEOs-of-All-Time.html?slide=10 http://www.cnbc.com/2009/04/30/Portfolios-Worst-American-CEO...
- rkuykendall-com 10y agotl;dr: Engineer fat-fingered a command and shut everything down. Booting it back up took a long time. Then the backlog was huge, so getting back to normal took even longer. We made the command safer, and are gonna make stuff boot faster. Finally, we couldn’t report any of this on the service status dashboard, because we’re idiots, and the dashboard runs on AWS.
- fixermark 10y agoEverything except the "We're idiots" part I'd agree with. Self-hosting your diagnostic tools is an easy mistake to make, and I've seen both startups and large, multi-decade-experienced companies make it.
- rkuykendall-com 10y agoOf course. Meant it as more of an "I just spent 10 minutes searching for my car keys while holding them, because I'm an idiot." No disrespect to the engineers.
- jasonhoyt 10y ago"People make mistakes all the time...the problem was that our systems that were designed to recognize and correct human error failed us." [1] [1] http://articles.latimes.com/1999/oct/01/news/mn-17288 http://articles.latimes.com/1999/oct/01/news/mn-17288
- sp332 10y agoThis reminds me of Asimov's characteristically tiny story "Fault-Intolerant" https://unotices.com/book.php?id=38686&page=15 https://unotices.com/book.php?id=38686&page=15 (You can ignore the story at the top about Feghoot, the real story is below.)
- St-Clock 10y agoI am unpleasantly surprised that they do not mention why services that should be unrelated to S3 such as SES were impacted as well and what they are doing to reduce such dependencies. From a software development perspective, it makes sense to reuse S3 and rely on it internally if you need object storage, but from an ops perspective, it means that S3 is now a single point of failure and that SES's reliability will always be capped by S3's reliability. From a customer perspective, the hard dependency between SES and S3 is not obvious and is disappointing. The whole internet was talking about S3 when the AWS status dashboard did not show any outage, but very few people mentioned other services such as SES. Next time we encounter errors with SES, should we check for hints of S3 outage before everything else? Should we also check for EC2 outage?
- CWuestefeld 10y agoservices that should be unrelated to S3 such as SES were impacted I don't think this is particularly surprising. I'd already pretty much assumed that, e.g., a package of code for a Lambda function would be housed in an S3 bucket somewhere. What's really surprising to me is how many of those buckets appear to live in US-EAST-1, and aren't able to keep functioning in a catastrophe by failing over to a different region.
- kondro 10y agoYou specify the run region for each of those services and all the other components get restricted to those regions too. We're in ap-southeast-2 (Sydney) and none of our services were impacted yesterday.
- erikbye 10y agoWhat does everyone use S3 for? I'm genuinely curious. As my experiments with it have left me disappointed with its performance, I'm just not sure what I could use it for. Store massive amounts of data that is infrequently accessed? Well, unfortunately the upload speed I got to the standard rating one was so abysmal it would take too much time to move the data there; and then I suspect the inverse would be pretty bad as well.
- harryh 10y agoIt's a very easy way to cheaply store static data to be served over HTTP. There are many many many many many many people who want to do that.
- castis 10y ago1 scenario: if you run a website that has a lot of static content (multiple GB of images, css, js, etc) and you dont want your http server to be responsible for serving that content then you give it all to s3 and let them serve it for you.
- erikbye 10y agoWhat about the performance of serving it? Sounds like I would need to cache it myself, anyway.
- ckozlowski 10y agoPerformance out of S3 is generally really good. However, if you're looking to say, serve up a global website and your content is in a single S3 region, then you can leverage CloudFront CDN to serve up those objects. CloudFront integrates seamlessly with S3, and you don't pay transfer charges between CloudFront and S3.
- deleted 10y ago[deleted]
- peu4000 10y ago
- seanwilson 10y ago> At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. I find making errors on production when you think you're on staging are a big one for similar errors. One of the best things I ever did on one job was to change the deployment script so that when you deployed you would get a prompt saying "Are you sure you want to deploy to production? Type 'production' to confirm". This helped stop several "oh my god, no!" situations when you repeated previous commands without thinking. For cases where you need to use SSH as well (best avoided but not always practical), it helps to use different colours, login banners and prompts for the terminals.
- fixermark 10y ago"All teams have a test server. Some teams are fortunate enough to also have a separate production server."
- seanwilson 10y ago> "All teams have a test server. Some teams are fortunate enough to also have a separate production server." That's what I meant about the SSH comment. Not every team has the automation or infrastructure that allows you to avoid SSH.
- eric_h 10y agoWe have a deploy script that does exactly this, unfortunately we've all gotten our muscle memory so trained that most of us type the deploy command, press enter, type yes and press enter before we're ever even prompted. Fortunately in most cases a quick ctrl-c can prevent any actual damage.
- seanwilson 10y agoType 'yes'? Make it ask you to type out the deploy location. I think that helps a lot.
- westernmostcoy 10y agoTake a moment to look at the construction of this report. There is no easily readable timeline. It is not discoverable from anywhere outside of social media or directly searching for it. As far as I know, customers were not emailed about this - I certainly wasn't. You're an important business, AWS. Burying outage retrospectives and live service health data is what I expect from a much smaller shop, not the leader in cloud computing. We should all demand better.
- vorpalhex 10y agoEmails seem to be going out. I got one a while ago. I suspect this was an initial response geared towards the general audience and a more specific technical response will be forthcoming.
- Eiriksmal 10y agoInteresting observation. Maybe the answer is it that a behemoth like AWS does this because they _can_ get away with it. In contrast to AWS's cascading failures, the GitLab outage was a mere blip. Because they are several orders of magnitude smaller than Amazon, however, they had to be painfully transparent during their actual restore operations and in the post-moterm. AWS has more implicit trust that this won't happen again, since they've never (I think?) had something like this happen, so just a few lines about fixing the tool that let all the nodes shutdown is enough to restore confidence.
- perlgeek 10y agoAlso notably missing is the "we will automatically refund all affected customers" line that we'd expect from somebody who wants to provide excellent service. A graphical illustration of the service dependencies they were talking about would have been nice as well.
- CephalopodMD 10y agoI mean, it's in the SLA that they have to refund 10% for the billing period IIRC.
- fulafel 10y agoI guess this means it's much better for your app to fail over between regions than between availability zones.
- romanovcode 10y agoThey could've just not post anything. People already forgot about this disruption.
- tifa2up 10y agoDo you know if Amazon is giving any refunds/credits for the service outbreak?
- officelineback 10y agoI don't think it's automatic. I just helped my former boss with his decision to go for a refund (he asked me for help drafting a request, but I reminded him that like 99.99% of their S3 storage is backups that are IA-Standard, so it may not be worth it).
- all_usernames 10y agoOverall, it's pretty amazing that the recovery was as fast as it was. Given the throughput of S3 API calls you can imagine the kind of capacity that's needed to do a full stop followed by a full start. Cold-starting a service when it has heavy traffic immediately pouring into it can be a nightmare. It'd be very interesting to know what kind of tech they use at AWS to throttle or do circuit breaking to allow back-end services like the indexer to come up in a manageable way.
- njharman 10y agoDid you read the fucking article? That is EXACTLY what they are doing (among other things).
- dang 10y agoPlease stop posting like this so we don't have to ban you. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html We detached this comment from https://news.ycombinator.com/item?id=13776335 https://news.ycombinator.com/item?id=13776335 and marked it off-topic.
- Exuma 10y agoImagine being THAT guy.......... in that exact moment...... after hitting enter and realizing what he did. RIP
- sndean 10y agoI've never been in that sort of situation, but I really do wonder how I'd react. Hopefully in healthy way. How do surgeons react when they accidentally cut the wrong thing?
- hunterjrj 10y agoOh man, finally a chance to plug the book "Do No Harm: Stories of Life, Death, and Brain Surgery" by Henry Marsh. Marsh chronicles his career, and includes (at least) one story about the slip of the scalpel and the... result. Highly recommended.
- sndean 10y agoCool. This is why I love HN :) I'll make sure my soon-to-be MD girlfriend reads it too
- usernametbd 10y agoI can imagine being that guy in that exact moment. But I can't imagine being that guy after the event. There will be a constant fear and doubt in my mind. And a constant fear whether others trust me anymore. I couldn't quit because that might make me look bad and I couldn't continue because that might make me look bad.
- hanspeter 10y agoThis guy may actually need therapy to not suffer from some light degree of trauma.
- Vaanir 10y agoHe would make good friends with the Gitlab guy very easily I'd imagine.
- eplanit 10y agoSo this week the poor soul at Amazon, along with the Price-Waterhouse guy, are the poster children of Human Error.
- pwthornton 10y agoThis is the risk you run into by doing everything through the command line. This would be really hard to do through a good GUI.
- djhworld 10y agoI highly doubt this claim, humans make mistakes regardless of the control method. In this particular case the scripts didn't have adequate protections in place, but that's the benefit of hindsight
- pwthornton 10y agoOr the benefit of testing.
- andrewguenther 10y agoIt would be equally as hard through a good command line. Why is good design only possible in GUIs?
- pwthornton 10y agoIt's easier for humans to conceptualize what is happening if you visualize the design for them. A good command line would better protection would have helped, but this fails some of the core of good interface design.
- Piskvorrr 10y ago"Next/Next/Yes/Ok/Kill all humans/Yes/Ok/Accept/Ok/Ok/Yes/Next/Confirm" Does that sequence look familiar? Well, that's the GUI equivalent.
- idlewords 10y ago"we have changed the SHD administration console to run across multiple AWS regions." Dear Amazon: please lease a $25/month dedicated server to host your status page on.
- machbio 10y agoNo on in HN is questioning this - "The Amazon Simple Storage Service (S3) team was debugging an issue causing the S3 billing system to progress more slowly than expected." - they are debugging on Production System..
- skrowl 10y agoTLDR: Never type 'EXEC DeleteStuff ALL' When you actually mean 'EXEC DeleteStuff SOME'
- aestetix 10y agoIt sounds like this can be mitigated by making sure everything is run in dry run mode first, and for something mission critical, getting it double-checked by someone before removing the dry run constraint. It's good practice in general, and I'm kind of astonished it's not part of the operational procedures in AWS, as this would have quickly been caught and fixed before ever going out to production.
- user15672 10y agoI'm not sure you've understood what the problem is. They were removing some servers from a group. This isn't something that get's a dry run, not without spinning up the entire AWS infrastructure. It also wouldn't have helped a jot, since the issue came about after an employee executing a playbook made a typo. There's no way this could have been mitigated with a dry run. They're mitigating it in future by putting more aggressive safeguards in their tooling, which is the correct way to mitigate this sort of issue.
- jwildeboer 10y agoTL;DR typo. Oops.
- throwtotheway 10y agoI hope I never have to write a post-mortem that includes the phrase "blast radius"
- EdSharkey 10y agoIt's curious they needed to "remove capacity" to cure a slow billing problem. Is that code for a "did you try to reboot the system?" kind of troubleshooting? It sounds to me like the authorized engineer sent a command to reboot/reimage a large swath of the S3 infrastructure.
- sebringj 10y agoIf Amazon were a guy, he'd be a standup guy. This is a very detailed and responsible explanation. S3 has revolutionized my businesses and I love that service to no end. These problems happen very rarely but I may have backups just in case using nginx proxy approach at some point and because S3 is so good, everyone seems to adopt their API so its just a matter of a switch statement. Werner can sweat less. Props. I would add, it would be awesome if there was a simulation environment, beyond just a test environment that simulated servers outside requesting in, before a command was allowed to run onto production, like a robot deciding this, then could mitigate this, kind of like TDD on steriods if they don't have that already.
- tuxninja 10y agoI think they should have led with insensitivity about it and maybe a white lie. Such as... We took our main region us-east-1 down for X hours because we wanted to remind people they need to design for failure of a region :-) Shameless plugs (authored months ago): http://tuxlabs.com/?p=380 http://tuxlabs.com/?p=380 - How To: Maximize Availability Effeciently Using AWS Availability Zones ( note read it, its not just about AZ's it is very clear to state multi-regions and better yet multi-cloud segway...second article) http://tuxlabs.com/?p=430 http://tuxlabs.com/?p=430 - AWS, Google Cloud, Azure and the singularity of the future Internet
- tuxninja 10y agoI think they should have led with insensitivity about it and maybe a white lie. Such as... We took our main region us-east-1 down for X hours because we wanted to remind people they need to design for failure of a region :-) Shameless plugs (authored months ago): http://tuxlabs.com/?p=380 http://tuxlabs.com/?p=380 - How To: Maximize Availability Effeciently Using AWS Availability Zones ( note read it, its not just about AZ's it is very clear to state multi-regions and better yet multi-cloud segway...second article) http://tuxlabs.com/?p=430 http://tuxlabs.com/?p=430 - AWS, Google Cloud, Azure and the singularity of the future Internet
- asow92 10y agoMan, I'd really hate to be that guy.
- tn13 10y agoCan Amazon take responsibility and offer say 10% discount to all the customers who are spending >$X ?
- kinkrtyavimoodh 10y agoIt's already part of their SLA. https://aws.amazon.com/s3/sla/ https://aws.amazon.com/s3/sla/
- lasermike026 10y agoOps and Engineering here. My guts hurt just reading this. With big failures is never just one thing. There are a series of mistakes, bad choices, and ignorance that lead to a big system wide failures.
- mmanfrin 10y agoThere are quite a few comments here ignoring the clarity that hindsight is giving them. Apparently the devops engineers commenting here have never fucked up.
- scott_karana 10y agoOn the contrary: I feel like Amazon is taking some flak because everyone here has messed up before, and are surprised that engineers (seemingly lacking failure experience) were able to do what they did. I wouldn't task a junior sysadmin a server deletion, would you? Nor could I ever consider someone without a fuckup a senior ;)
- bandrami 10y ago"we have not completely restarted the index subsystem or the placement subsystem in our larger regions for many years." Yeah... nothing says "resilience" quite like that...
- mleonhard 10y agoAWS partitions its services into isolated regions. This is great for reducing blast radius. Unfortunately, us-east-1 has many times more load than any other region. This means that scaling problems hit us-east-1 before any other region, and affect the largest slice of customers. The lesson is that partitioning your service into isolated regions is not enough. You need to partition your load evenly, too. I can think of several ways to accomplish this: 1. Adjust pricing to incentivize customers to move load away from overloaded regions. Amazon has historically done the opposite of this by offering cheaper prices in us-east-1. 2. Calculate a good default region for each customer and show that in all documentation, in the AWS console, and in code examples. 3. Provide tools to help customers choose the right region for their service. Example: http://www.cloudping.info/ http://www.cloudping.info/ (shameless plug). 4. Split the large regions into isolated partitions and allocate customers evenly across them. For example, split us-east-1 into 10 different isolated partitions. Each customer is assigned to a particular partition when they create their account. When they use services, they will use the instances of the services from their assigned partition.
- spullara 10y agoTwitter once had 2 hours of downtime because an operations engineer accidentally asked a tool to restart all memcached servers instead of a certain server. The tool was then changed to make sure that you couldn't restart more than a few servers without additional confirmation. Sounds very similar to this situation. Something to think about when you are building your tools to be more error proof.
- idlewords 10y agoThere are very few things that Twitter hasn't had two hours of downtime because of.
- WillyOnWheels 10y agoOutage was almost a day long.
- thraway2016 10y agoSomething doesn't pass the smell test. Over two hours to reboot the index hosts?
- canadaduane 10y agoI assume "reboot" in this instance means more than turning it off and on again--it must return to a working state, with many volumes of data requiring log processing to find the last (and best) "good state".
- Piskvorrr 10y agoI don't think those would only have a mere 1 MB of data ;) Even considering RAM speeds, stuff scales in surprising ways when a third of the Internet relies on you. (Right, right, exaggerating - this was one AZ only; but you catch the drift, I assume)
- chirau 10y agoDeletions, shutdowns and replications should always either contain a SELECT to show affected entities or a confirmation (Y/n) of the step.
- davidf18 10y agoThere should be some sort of GUI interface that does appropriate checks instead of allowing someone to mistakenly type the correct information.
- EternalData 10y agoThis caused panic and chaos for a bit among my team, which I imagine was replicated across the web. Moments like these always remind me that a particularly clever or nefarious set of individuals could shut down essential parts of the Internet with a few surgical incisions.
- i336_ 10y ago> (...) [W]e have not completely restarted the index subsystem or the placement subsystem in our larger regions for many years. S3 has experienced massive growth over the last several years and the process of restarting these services and running the necessary safety checks to validate the integrity of the metadata took longer than expected. All those tweets saying "turn it off and back on again"? "We accidentally turned it off, but it hasn't been turned it off for so long it took us hours to figure out how to turn it back on." Poorly-presented jokes aside, this is rather concerning. The indexer and placement systems are SPOFs!! I mean, I'd presume these subsystems had ultra-low-latency hot failover, but this says they never restarted, and I wonder if AWS didn't simply invest a ton of magic pixie dust in making Absolutely Totally Sure™ the subsystems physically, literally never crashed in years. Impressive engineering but also very scary. At least they've restarted it now. And I'm guessing the current hires now know a lot about the indexer and placer, which won't do any harm to the sharding effort (I presume this'll be being sharded quicksmart). I wonder if all the approval guys just photocopied their signatures onto a run of blank forms, heheh.
- richardwhiuk 10y agoI don't think you don't understand the architecture of the system, if you are describing the indexer as a SPOF. The system is a collection of shards. If you replicate it to create a second shard, then you'll just have a large a system, which is still a single point of failure. The index, by necessity, has to be able to answer the question 'this object exists' or 'this object doesn't exit' - so it needs to have consensus.
- i336_ 10y agoHrm. My speculative presumption was going off the sole datapoint of "we have not completely restarted the index subsystem or the placement subsystem in our larger regions for many years". I'm not quite sure how to interpret "restart" in this context, mostly due to lack of exposure or experience. The report also says "Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. The servers that were inadvertently removed supported two other S3 subsystems." So you're right, it looks like multiple servers were supporting these systems, which does make sense (especially considering the load they would have seen). Okay. I guess I didn't quite think through the load requirements and thought these were single machines - which is certainly ludicrous thinking :) - and that's where I got the SPOF reasoning from. You're very right though, these consensus systems must be built as bottlenecks in order to see everything. And there aren't really any alternatives: "build extra indexers and placement systems!" just gives you "but what if _all_ of them get taken offline?" and "it can't leave the datacenter, it sees 100GB/s of throughput" (number taken out of thin air). Good points.
- fr4egy8e9 10y agoWhat most AWS customers don't realize is that AWS is poorly automated. Their reliability relies on exploiting the employees to manually operate the systems. The technical bar at Amazon is incredibly low and they can't retain any good engineers.
- mulmen 10y agoIf this boils down to an engineer incorrectly entering a command can we please refer to this outage as "Fat Finger Tuesday"?
- FormerAMZN 10y agoI'm a little surprised everyone is giving them a pass on this: S3 has experienced massive growth over the last several years and the process of restarting these services and running the necessary safety checks to validate the integrity of the metadata took longer than expected. I'm biased - so take this with a grain of salt. But, I don't think your fast growth is an excuse for not doing some fairly basic stuff. What other processes have you not restarted in a long time?
- dootdootskeltal 10y agoI don't know if it's a C thing, but those code comments are art!
- reitanqild 10y agoposting in wrong thread?
- dootdootskeltal 10y agooops yeah, I had so many HN tabs.
- vdchuyen 10y agoHuman is complicated. Let them relax and let the androids do the boring jobs :D
- feisky 10y agoGreat shock to the quick recovery of such a big system.
- deleted 10y ago[deleted]
- pfortuny 10y agoI guess it is time to define commands whose inputs have great distance in, say the Damerau-Levenshtein metric. For numerical inputs, one might use both the digits and the textual expression. This would make them quite cumbersone but much less prone to errors. Or devise some shorthand for them... 156 (on fi six). 35. (zer th fi). 170 (on se zer). 28 (two eig) evens have three letters odds have two. This is just my 2 cents.
- bsaul 10y agoWonder if every numbers for critical command lines shouldn't be spelled out as well. If you think about how checks works, you're supposed to write the number as well as the words for the number. -nbs two_hundreds instead of twenty is much less likely to happen.. just like rm -rf / should really be rm -rf `root`
- carlsborg 10y ago"As a result, (personal experience and anecdotal evidence suggest that) for complex continuously available systems, Operations Error tends to be the weakest link in the uptime chain." https://zvzzt.wordpress.com/2012/08/16/a-note-on-uptime/ https://zvzzt.wordpress.com/2012/08/16/a-note-on-uptime/
- hemant19cse 10y agoAmazon's AWS Outage Was More of a Design(UX) Failure and Less of Human Error. https://www.linkedin.com/pulse/how-small-typo-caused-massive-downtime-s3aws-hemant-kumar-singh https://www.linkedin.com/pulse/how-small-typo-caused-massive...
- sumobob 10y agoWow, I'm sure that person who mis entered the command will never, ever, ever, do it again.