10 ms·
Look at the language used though. This is saying very loudly "Look, this isn't the engineer's fault here". It's one thing I miss about Amazon's culture- not bla
by mabbo 10y ago
Look at the language used though. This is saying very loudly "Look, this isn't the engineer's fault here". It's one thing I miss about Amazon's culture- not blaming people when system's fail.
The follow-up doesn't bullshit with "extra training to make sure no one does this again", it says (effectively) "we're going to make this impossible to happen again, even if someone makes a mistake".
- pmoriarty 10y agoPutting the capability to take down S3 in to the hands of a single engineer seems a bit much. Is mere extra training the right solution here? Maybe they need something like the procedure that's used in missile silos: Not allowing the shutdown system to function at all without the explicit authorization of least two people.
- timdorr 10y agoI think you have it backwards. The post does not say they will simply be training the problem away. They are putting safeguards into their tooling to prevent the case of a fat finger.
- baq 10y agoThe article leaves little doubt that they didn't know such an event would be so hard to recover from. They knew it wouldn't be easy, but they were surprised by how bad it was.
- notatoad 10y agoThe linked article also says the tools they use were changed to limit the amount of resources that could be taken down at a single time, the speed they could be taken down at, and a hard floor was put on the number of instances that could be stopped. That's a lot more than just extra training, and a lot better than a two-key system.
- ceejayoz 10y ago> Maybe they need something like the procedure that's used in missile silos... Probably a bad example. The system was a pain in the ass, so they went and circumvented some of its restrictions. http://gizmodo.com/for-20-years-the-nuclear-launch-code-at-us-minuteman-si-1473483587 http://gizmodo.com/for-20-years-the-nuclear-launch-code-at-u... > Those in the U.S. that had been fitted with the devices, such as ones in the Minuteman Silos, were installed under the close scrutiny of Robert McNamara, JFK's Secretary of Defence. However, The Strategic Air Command greatly resented McNamara's presence and almost as soon as he left, the code to launch the missile's, all 50 of them, was set to 00000000. > Oh, and in case you actually did forget the code, it was handily written down on a checklist handed out to the soldiers.
- hellbanner 10y agoThis is good. And for the software engineers, great. I've heard from people doing the grunt work at amazon -- warehouse staff -- that Amazon incentivises employees to rat out each other for mishandling, late time etc, fostering intense competition
- mabbo 10y agoI spent time in the fulfillment centers, writing software for them. I definitely didn't see that sort of thing. There's no need- the software tracked everything they did. Low performer would be found and retained or 'promoted to customer' without the need for anyone to 'rat out'. Plus, managing humans in a 'rat out' system would be incredibly inefficient. Now you need lots of employees just to listen to the ratting!
- djsumdog 10y agoWhich is really the point of automation and configuration management. When a manager asks you, "How are you going to prevent this in the future?" You can say, "We added a check so n must be less than x% of the total number of cluster members," or "We added additional unit tests for the missing area of coverage" or "We added new integration tests that will pick up on this." Tests and configuration scripts don't prevent all breakage. But when you have them, you can say, "We missed that, let's add it," or "That failed, but it's a false positive. Let's add this edge case to this test." If you have no automation, tests or auditing systems around running deployments, you can't do any of this.
- jegoodwin3 10y agoI agree testing and automation are good. I think they need to go beyond this to formal verification, for something on this scale and reliability. NASA doesn't make these sorts of mistakes. By the way - this is not just Amazon's problem now. We know the internet has a single point of failure. So does a lot of IoT. When will we experience the first Suicide DevOps?
- gmac 10y agoNASA doesn't make these sorts of mistakes https://www.wired.com/2010/11/1110mars-climate-observer-report/ https://www.wired.com/2010/11/1110mars-climate-observer-repo...
- lisper 10y agohttps://www.youtube.com/watch?v=6OalIW1yL-k https://www.youtube.com/watch?v=6OalIW1yL-k (Specifically https://www.youtube.com/watch?v=6OalIW1yL-k#t=3m https://www.youtube.com/watch?v=6OalIW1yL-k#t=3m but it's worth watching the whole clip (or even the whole movie) if you haven't seen it before. It's from Terry Gilliam's "Brazil".)
- icebraining 10y agoAlmost twenty years ago, though.
- liquidise 10y agoThis is the major basis of the CMM Levels [1]. At higher levels of maturity and necessity, systems and processes are designed to increasingly prevent errors from reaching a production environment. Amazon is taking the right approach here. The fact that a system as complex and important as S3 can be taken down is a failure of the system, not the person who took it down accidentally. 1. https://en.wikipedia.org/wiki/Capability_Maturity_Model#Levels https://en.wikipedia.org/wiki/Capability_Maturity_Model#Leve...
- geodel 10y agoA lot of IT vendors I have worked with, they all were CMM/CMMi level 5. But the crappiness in their work development/process/deployment etc make me wonder if all their efforts go in attaining those certifications as oppose to doing something better.
- vaishaksuresh 10y agoAs someone who worked for an IT vendor with certification and as someone who was part of the certification team at another place, I can assure you that you're right. The certification is more for the organization/unit and the people working do not realize what they are for. Another thing that usually becomes a problem is the rigidity of the certification. Saying you need X, Y and Z documented is easy, but it doesn't work for projects that maybe don't have Y. So people make up documentation and process just to be compliant, this soon becomes a hinderance to the work. At this point people either abandon the process or follow it and the work suffers.
- _ao789 10y agoThank you for adding this comment. I am glad there are more people out there that aren't afraid to be honest about some of the nonsense 'follow the process no matter what' stuff that I have experienced over the years.
- groby_b 10y agoCMM level 5 ==> You have a well-documented, repeatable, and still horrible process that declares all errors statistically uncommon by "augmenting" the root cause with random factors. Insta-certification. (I lied about the "insta" part)
- Bartweiss 10y agoAny time I see "we're going to train everyone better" or "we're going to fire the guy who did it", all I can read is "this will happen again". You can't actually solve the problem of user error with training, and it's good to see Amazon not playing that game.
- problems 10y agoThis is true in some cases, but not when mitigations aren't practiced properly - it's not the fat fingered user who should be fired or retrained, but the designer or maintainer of the system that allowed it to become a serious issue. Look at the recent GitLab incident - one guy messed up and nuked a server. Okay, that happens sometimes, go to backups. Uh oh, all the backups are broken. Minor momentary problem just turned into a major multi-day one. That's a problem and one which could be preventable with training (or, arguably, firing and hiring). Maintaining your backups properly should be someone's duty, designing and testing systems to minimize impact of user error should be too.
- Bartweiss 10y agoFair enough. I guess what I meant was specifically using training or punishment to combat "momentary lapse" issues. If someone doesn't test their backups, you train them to test backups. If someone lies about testing the backups, maybe you fire them. But if someone trips and shatters the only backup disk, you don't yell at them - you create backups that an instant of clumsiness can't ruin. I did overstate, training is perfectly reasonable, but I often see it cited exactly when it shouldn't be, as a solution to errors like typos or forgetfulness.
- hibikir 10y agoTraining about testing backups is still a bad idea: Why make someone do a job that purely verification? Those jobs eventually stop getting done, and it's hard to keep people doing them. Instead, you make a machine verify the backups simply by using the backups all the time. For example, at work I feed part of our data pipeline with backups: Those processes have no access to the live data. If the backups break, those processes would provide bad information to the users, and people would come complaining in a matter of minutes. Just like when you have a set of backup servers, you don't leave them collecting dust, or tell someone to go look at them every once in a while: you just route 1% of the traffic through them. They are still extra capacity, you can still do all kinds of things to them without too much trouble, but you know they are always working. Never, ever, force people to do things they don't gain anything from. Their discipline would fade, just like it fades when you force them to a project management tool they get no value from.
- antaviana 10y agoI presume this is well entrenched in the Amazon culture. Jeff Bezos once said: "Good intentions never work, you need good mechanisms to make anything happen"
- mabbo 10y agoThat's exactly it. Amazon doesn't like sharing all that much, but I wish they'd publicly release that video.
- refulgentis 10y agoHow laudable is this, really? I've had the privilege of either working for myself, the company that acquired mine and let me run the dev, or at Google. From that perspective, and what I understand about ops, the rarity is not having the attitude mentioned in the parent.
- simplify 10y agoAre you suggesting we take their behavior for granted? Positive behavior needs to be praised – it's part of how society influences its members.
- refulgentis 10y agoNo
- jordache 10y agothat's just a public statement. how do you whether the individual was reprimanded
- mabbo 10y agoBecause in my five years as an Amazon dev, that's exactly the attitude I witnessed. People are trying their best, so firing them won't help.
- mrep 10y agoI believe Jeff once said something along the lines of "why would I fire an employee that made an honest mistake? I just spent a bunch of money teaching him a lesson"
- philtrem00 10y agoYeh, makes sense to make changes to the system rather than do nothing and just blame someone. Errors happen, it's something you can't avoid.
- gigatexal 10y agoYup. And for that mercy the engineer is going to be that much more careful, and loyal. I would be, that's for sure.
- flamedoge 10y agolol what part of "these things should be done proactively and tested over and over in CI" does not make sense to management?
- mulmen 10y agoAgreed, especially regarding the culture but isn't this pretty much the same explanation they gave a few years ago when something similar happened? I seem to recall an EC2 or S3 outage a few years ago that boiled down to an engineer pushing out a patch that broke an entire region when it was supposed to be a phased deployment. I could be mis-remembering that but it's important that these lessons be applied across the whole company (at least AWS) so it would be a bigger mark against AWS if this is a result of similar tooling to what caused a previous outage.
- cjbprime 10y agoPretty sure that one was a Microsoft Azure outage. (Source: am a self-identified post-mortems connoisseur. :)
- evilDagmar 10y agoNot a bad plan. If you don't make enough mistakes on your own, ya gotta learn from the mistakes of others as a preventative.
- sah2ed 10y agoDo you by chance keep a public log of your postmortem collection :)?
- cjbprime 10y agoI don't, but danluu does! https://github.com/danluu/post-mortems https://github.com/danluu/post-mortems
- bobzimuta 10y agoYeah an EC2 engineer switched over traffic to a backup network connection that had significantly less bandwidth, triggering cascading failures.