33 ms·
> At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of
by ajross 10y ago
> At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended.
It remains amazing to me that even with all the layers of automation, the root cause of most serious deployment problems remain some variant of a fat fingered user.
- danielodio 10y agoSounds like AWS needs Spinnaker for easy rollbacks! https://news.ycombinator.com/item?id=13776456 https://news.ycombinator.com/item?id=13776456
- NikolaeVarius 10y agoAutomation doesn't just allow you to create/fix things faster. It also allows you to break things faster.
- kevin2r 10y agoWe may think that an automated system requires less understanding of it in order to operate it. But from the other point of view, you have to know what you are doing, consequences of even an small change are big. This is one of the things that happens with windows, getting up a server is so easy, that people believe that they don't have to understand what's under the hood, and then, we get a lot of miss-configuration and operational issues.
- pm90 10y agoWonder what happened to the poor slob who did that. He/She was unauthorized AND caused a pretty serious outage... EDIT: derp, my bad, I read as "unauthorized" which was "authorized".
- apetresc 10y agoThe article specifically said "authorized".
- danesparza 10y agoHow do you figure? From the blog post: " At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended."
- umanwizard 10y agoOne of the positive things about Amazon's culture is that they heavily emphasize blaming broken processes, not blaming people. I doubt the person involved will have any negative consequences beyond embarrassment.
- Raphmedia 10y agoCan you imagine the feeling? I once sent a debug email to our mailing list and felt terrible for days. Imagine bringing down the internet...
- erikbye 10y agoI would be horrified if I learned that Amazon or any other company of such size in any way castigates employees for such very human errs. The guilt (don't beat yourself up) he or she likely feels is bad enough. Anyway, to me this firstly sounds like a "tool" or command that was too powerful with not enough safeguards. Who knows, the command might even be ambiguous.
- rocgf 10y agoI think the article says he was authorized? Either way, at worst, he got fired.
- jmelloy 10y ago
- mcrider 10y agoSounds like an opportunity for machine learning. Anyone want to write an AI BOFH?
- mgkimsal 10y agoMost of the BOFH I've known have seemed artificially intelligent already.
- robotmlg 10y agoBasically what Netflix's Chaos Monkey is: https://github.com/netflix/chaosmonkey https://github.com/netflix/chaosmonkey
- beambot 10y agoHeaven forbid an AI whose primary object is Chaos Monkey gain sentience. That might just be worse than paperclips.
- agency 10y agoTo make error is human. To propagate error to all server in automatic way is #devops - DevOps Borat
- jerf 10y agoI've long said something like "To err is human. To fuck up a million times in a second you need a computer." I may have to upgrade that to take the mighty power of Cloud (TM) into account, though. Billions and trillions of fuck ups per second are now well within reach!
- M_Grey 10y agoI can't wait until quantum computing lets us add a degree of simultaneity to fucking up. Fuck up in many ways... AT ONCE!
- duskwuff 10y agoQuantum computing: giving humans the unprecedented ability to make every possible error at once
- beamatronic 10y agoIt will not be certain if you have fucked up or not until you actually go to check.
- BurningFrog 10y agoOr the root cause is a UI that allows mistakes like these.
- chrisseaton 10y agoIs it possible to build a UI that will not allow you to make a mistake? If the computer knows exactly what actions would be a mistake, why can't it just do the correct actions (those that aren't a mistake) automatically?
- avar 10y agoYou don't have to know what would be a mistake. E.g. if the tool is used most of the time to operate on a small set of servers, you have some extra confirmation or command-line option for removing a large set. That's good UI design in tools with powerful destructive capabilities. You make the UI to do lots of things v.s. the few things you do routinely different enough that there's no mistaking them.
- rrmm 10y agoYou can also have the program tell the user what's going to happen (if it can be computed beforehand), e.g. "This will affect 138 server(s)."
- avar 10y agoYes, but be careful. UIs like that tend to accumulate "--yes" options, because you don't feel like being asked every time for 1 server. Then one day you screw up the wildcard and it's 1000 servers, but you used the --yes template. Which is why I'm pointing out that to design UIs like these you should fall back on slightly different UIs depending on the severity of the operation.
- Atheros 10y agoThis doesn't work. Users learn to ignore the message.
- emeraldd 10y agoAutomation tends to make those kinds of errors worse rather than better. Perhaps more infrequent and of a different nature than before, but screwing up an automated action cascades much, much faster than a human initiated one. As a result, you have to watch things a good deal closer and build in more and tighter safe guards. For instance: https://thenextweb.com/shareables/2014/05/16/emory-university-server-accidentally-sends-reformat-request-windows-pcs-including/#.tnw_yrJSSrvP https://thenextweb.com/shareables/2014/05/16/emory-universit... Note: Automation is great, you just can't be sloppy with it. EVER. edit:fix minor typo
- bluejekyll 10y ago> established playbook A playbook actually represents a lack of automation for a particular task. The playbook itself should be automated, with automated tests that validate its correctness.
- beberlei 10y agoIts probably their name for an automated admin task. The post does bot imply that this was merely a checklist of things to do. Ansible calls their automatiin receipts playbook as well.
- bluejekyll 10y agoHm, based on the description, I would be surprised if they could fat-finger that.
- emeraldd 10y agoplaybooks can take arguments: http://stackoverflow.com/questions/30662069/how-can-i-pass-variable-to-ansible-playbook-in-the-command-line http://stackoverflow.com/questions/30662069/how-can-i-pass-v... So, fat-fingering something is imminently possible.
- seansmccullough 10y agoIt's probably a page on the internal Wiki that the S3 team follows for that particular task. Most the actual steps are probably automated, but it sounds more like a checklist. I used to follow runbooks/playbooks written on the internal wiki when I worked at Amazon.
- gigatexal 10y agoYeah as soon as I read this I felt bad for the employee. I remember writing an update statement without a where clause and having to restore the table from backup. But that was at a company not as advanced as Amazon. Fat fingering a key like that is just crazy (but comforting that even at Amazon it happens) and I'm sure they've fixed that from happening again.
- duskwuff 10y agoFWIW: Setting "safe-updates=1" in ~/.my.cnf will require UPDATE and DELETE statements in the client to have a WHERE clause which references a key. It's not perfect protection, but it will save you from a lot of mistakes.
- gigatexal 10y agohmm, that's kinda cool. I'm in a MS shop and I don't know if SMSS has the same feature. My manager just looked at me and said "welp, go restore the table and be more careful next time." I was a new DBA at the time, still, kinda new.
- amichal 10y agoThat's awesome, My worst DELETE fail however was: DELETE * FROM table WHERE [long condition that resolves to true for all records] Now i write SELECT or SELECT COUNT(*) over and over again until i see the data i expect and then change it to a DELETE/UPDATE. It's not my personal habit but some folks I know turn off auto commit and BEGIN a transaction every time they enter an interactive SQL sessions. They then default to ROLLBACK at least once before COMMITing them. That and having a user with read-only permissions or a read replica
- EvanAnderson 10y agoComputers are devices built to amplify human error.
- guelo 10y agoBicycles for the bumbling mind.
- CobrastanJorji 10y agoIt's one of the reasons that silly guarantees like "twelve 9s of reliability" are meaningless. There are humans here. "Accidental human mishap" is gonna happen sometimes, and when it happens it's probably gonna affect a lot of data. Heck, at around 7 or 8 nines you have to account for the possibility that your operations team will decide that all your data is a vicious pack of timberwolves and needs to be defeated.
- pavel_lishin 10y agoAs I understand it, those guarantees don't mean that the service will actually stay up for the given number of 9s; it's that you'll be reimbursed monetarily if and when they go down.
- HeyLaughingBoy 10y agoKinda the same thing, though. I mean, from my perspective there's no substantive difference between me saying "this service will stay up 99.xx% of the time and me buying insurance to pay you for the 0.xx% of the time I might fail. The alternative is that I use the insurance to pay my legal fees when you sue me for not meeting my uptime guarantees.
- jstanley 10y agoIt's not the same thing. The Amazon service might only be costing you $100/mo, but if it goes down the cost to your business might be millions. They'll reimburse you the $100, not the millions.
- SilasX 10y agoI don't think it even means that; their policy says that the reimbursement only happens when your reliability dips all the way down to three 9s: https://aws.amazon.com/s3/sla/ https://aws.amazon.com/s3/sla/
- dkersten 10y ago
- rmason 10y agoSo they're going to build a complex system to correct possible user command line errors. That new system itself will introduce possible errors. Wouldn't an administrative GUI have been much simpler to implement overall?
- Twirrim 10y agoNo, they're introducing safeguards to an already established system. It's not even that complex, for that matter.
- greglindahl 10y agoDo you have an assumption that GUIs are safer than command lines? Or have fewer programming bugs in them? I don't think either is true.
- mabbo 10y agoLook at the language used though. This is saying very loudly "Look, this isn't the engineer's fault here". It's one thing I miss about Amazon's culture- not blaming people when system's fail. The follow-up doesn't bullshit with "extra training to make sure no one does this again", it says (effectively) "we're going to make this impossible to happen again, even if someone makes a mistake".
- pmoriarty 10y agoPutting the capability to take down S3 in to the hands of a single engineer seems a bit much. Is mere extra training the right solution here? Maybe they need something like the procedure that's used in missile silos: Not allowing the shutdown system to function at all without the explicit authorization of least two people.
- timdorr 10y agoI think you have it backwards. The post does not say they will simply be training the problem away. They are putting safeguards into their tooling to prevent the case of a fat finger.
- baq 10y agoThe article leaves little doubt that they didn't know such an event would be so hard to recover from. They knew it wouldn't be easy, but they were surprised by how bad it was.
- notatoad 10y agoThe linked article also says the tools they use were changed to limit the amount of resources that could be taken down at a single time, the speed they could be taken down at, and a hard floor was put on the number of instances that could be stopped. That's a lot more than just extra training, and a lot better than a two-key system.
- ceejayoz 10y ago> Maybe they need something like the procedure that's used in missile silos... Probably a bad example. The system was a pain in the ass, so they went and circumvented some of its restrictions. http://gizmodo.com/for-20-years-the-nuclear-launch-code-at-us-minuteman-si-1473483587 http://gizmodo.com/for-20-years-the-nuclear-launch-code-at-u... > Those in the U.S. that had been fitted with the devices, such as ones in the Minuteman Silos, were installed under the close scrutiny of Robert McNamara, JFK's Secretary of Defence. However, The Strategic Air Command greatly resented McNamara's presence and almost as soon as he left, the code to launch the missile's, all 50 of them, was set to 00000000. > Oh, and in case you actually did forget the code, it was handily written down on a checklist handed out to the soldiers.
- tdicola 10y agoTo be fair, the real problem isn't that someone screwed up a playbook or command. The real problem is that a tiny mistake in a command can cause an entire service to be disrupted for hours. That's the problem that needs to be fixed.
- gregtaleck 10y ago"While removal of capacity is a key operational practice, in this instance, the tool used allowed too much capacity to be removed too quickly. We have modified this tool to remove capacity more slowly and added safeguards to prevent capacity from being removed when it will take any subsystem below its minimum required capacity level. This will prevent an incorrect input from triggering a similar event in the future."
- thewhitetulip 10y agoI wonder as to why this wasn't thought of while creating the system? Of course, I don't have experience at that scale, so just wondering.
- traviscj 10y agoIt seems like sometimes this is just how iteratively automating things works, especially on an internal-facing tool. You have some process that starts out being "deploy this app with this java code". You deploy once and while, so it's not a big deal. But then those changes get a bit more frequent and so you pull out the common bits and the process becomes "make this YAML change in git and redeploy the app". That works until you find yourself deploying 5 times a day, so you turn it into a MySQL table, and the process becomes "write a ROLL plan that executes this UPDATE x=y WHERE t=u; command" After a while you get super annoyed at some quirk of the commands and figure, "Ok, fine, I'll just add an endpoint and some logic that just does this for the command case." Then you wanna go on vacation and the new guy messed up the API request last week, so you figure, "I'll just add a little JS interface with a little red warning if the request is messed up in this way or that before I go". You get back from vacation and some original interested party (whoever has wanted all these changes deployed) watched the intern make the change and thinks they could just do it themselves if they had access to the interface. You're wary, but you make the changes together a few times and maybe even add a little "wait-for-approval" node in the state machine. Life is good. You've basically de-looped yourself, aside from a quick sanity check and button press, instead of what was a ~2 hour code + build + PR + PR approved + deploy process. Then that interested party goes to work for Uber and the rest of your team adds a few functionalities on top of the interface you built and it all goes pretty well, until you realize that now that this thing that used to be 20 YAML objects is now 50k database records, and a bunch of them don't even apply anymore. So you build a button to disable some group of them, but after getting it deployed you realize it's actually possible to issue a "disable all" request accidentally if you click a button in your janky JS front-end before the 50k records download and get parsed and displayed. Oops! This mistake that you and the original interested party would have never made (because you spent the last 2 years thinking about all this crap) is probably a single impatient anxious mouse-click away from happening. So you make a patch and deploy that. Congrats! You found that particular failure mode and added some protections for it, and maybe added some other protections like rate-limiting the deletions or updates or whatever. That's cool, but is that every failure mode? I bet it isn't. What happens when someone else thinks you have too many endpoints and just drops to SQL for the update? Basically, yeah, of course you think of this stuff while iterating on it. But you figure "only power users are on the ACL" or "my teammates will understand the data model before making changes, or ask me first" or "that's what ROLL plans are for" or "I'll show a warning in the UI" or whatever. Fundamentally, you're thinking about a way to do a thing, if you're even thinking about it at all. So yeah, that's what I've spent the last year or two doing. :-)
- tn13 10y agoNumber of lawyers must have drafted those lines and 5 people including Bezos must have approved those lines. Those lines are not reflective of what Amazon is but what picture Amazon wants to paint now. They have clarified it is their error and not some hacking attempt. Secondly they have not vilified the engineer in question because already Amazon's culture is a bit of a ??? in public mind. But they have got it right. Shit happens and this is not the first time it has happened or the last time it will happen. Also it will happen with Microsoft, Google and everyone else. May be we will build even better technologies that will rely on two different cloud providers instead of 1.
- hoodoof 10y agoWhy? People make mistakes.
- ASalazarMX 10y agoSomehow, it's reassuring to see how 45 years later, the rm -rf class of problem still persists.
- zamalek 10y agoEvery communication channel has its flaws. CLI is fast and that's why it is a favorite. It is also noisy. If you have to worry about a fat finger, you are using the wrong communication channel or could afford to be a bit more verbose within that channel. That's why rm has safety nets.
- Fiahil 10y agoSomeone forgot a --limit on their ansible playbook ? I did that too.
- logicallee 10y agoserious question - why did no one ever accidentally launch and nuke a city, with thousands of nuclear warheads able to do so on short notice? like, AWS presumably puts a lot more redundancy in, and yet with all that effort comes up this far short. Why? It has a huge amount of brainpower all set up so that this never ever happens. Whatever works for the military, can't they adopt those actual best practices?
- jessaustin 10y agoWhen I think about questions like these, I recall the Anthropic Principle. Perhaps on lots of planets, intelligent life ceased at the beginning of the Atomic Age. Here we are seven decades (several generations!) in, and we're still alive! The numerator on the odds almost doesn't matter, when you never get to see the denominator. Now that we're finding all these planets, perhaps we ought to start looking for nuclear extinction events? They probably wouldn't leave lasting evidence, but if they're common enough they wouldn't need to... Actually the accounts I've read seem to indicate that most missile operators simply decided they would never launch no matter what. God bless them, for that.
- marvinalone 10y agoIn 2016, the UK accidentally fired a Trident missile at the US mainland: http://www.dw.com/en/uk-government-covered-up-disastrous-failure-in-nuclear-missile-test-report/a-37230406 http://www.dw.com/en/uk-government-covered-up-disastrous-fai... This one did not carry a warhead. Others do...
- sorenbs 10y agoI highly recommend reading https://www.amazon.com/Command-Control-Damascus-Accident-Illusion/dp/0143125788 https://www.amazon.com/Command-Control-Damascus-Accident-Ill... Turns out the answer to your question is simply: luck.
- 72deluxe 10y agoGUIs are really great. They're a recent development in the computing industry that help mitigate this sort of problem. You can even put prompts in that get you to confirm Yes/No to continue. I think Borland do some RAD systems, and Microsoft have an IDE of sorts on the way too. EDIT: Please note that this is humour.
- autokad 10y agofeel sorry for that user, I'd want to hide in a corner
- keldaris 10y agoI don't understand how this is even possible in a company operating on that scale. Granted, I'm a lowly scientific programmer with no clue about running a cloud infrastructure, but I would have imagined that there would be at least a pretense of oversight for destructive commands run in such an environment. A scheme as simple as "any destructive command run on S3 subsystems is automatically run in a dry run form, and requires independent confirmation by 2-3 other engineers to actually come into effect" would have prevented this altogether. Given the overall prominence of S3, this incident seems to demonstrate a rather callous attitude on the part of the organization.
- umanwizard 10y agoI thought the same thing before I went into the industry but now that I've been in it for a few years (including two at Amazon), it doesn't surprise me. I suspect locking everyone down in the way you suggest would cost more in lost productivity (and costs for the infrastructure that would be required for greater auditing, etc.) than is lost in outages like this.
- StreamBright 10y agoIt is always going to be like that. If you write software that has a rule "do not remove more than 5% of capacity at once" it will always work, yet if you tell a systems engineer please do not remove 5% of the capacity at once it fails with 0.0x% chance. The solution is that you move the execution of change into a system that spits out steps that is automatically executed by the system itself, entirely removing the human factor.
- linsomniac 10y ago20 years ago I read a postmortem of Tandem and their Non-Stop Unix. A core take-away for me was: "Computer hardware has gotten way more reliable than it was." combined with "The leading cause of outages has become operators making mistakes."