5 ms·

I wouldn't want to be the person who wrote the wrong command! Sheesh.

by ct0 10y ago

tyingq 10y ago
Meh, that's a process problem, not a people problem. Playbooks that have you retype commands with complex options, with no confirmation, etc, are inviting that sort of thing.
makira 10y ago
That person is the least likely to make that mistake again!
- oblio 10y ago
  No! Off with this head!!! On a more serious note, if you've never done something like this, you haven't had enough interesting projects. I've had a decent career and I still managed to: * re-deploy the current application version in all our data centers, instead of the new version, in a period when our deployment wasn't a 0-downtime one * rename all the Jenkins jobs on the server to the same name, thus deleting hundreds of Jenkins jobs in one fell swoop "Let him who is without sin cast the first stone" and all that :)
  - ckozlowski 10y ago
    There's two things I've come to believe in my IT career regarding operations: 1. No organization anywhere is a paragon of excellence, and everyone can benefit from improvement. 2. Every organization is made up of humans just like you. With all that entails. Some things which seem blatantly obvious after the fact are easily overlooked when the pressure to deliver is high and other issues are taking precedence. I completely agree with your statement. In fact, when I do interviews, one of my favorite and most insightful questions to ask is, basically, "tell me about a time you screwed the pooch." If they don't have a story and they worked in ops, then it can suggest they didn't really do much. The really sharp ones I've interviewed have a good story or two (and can tell it in excruciating detail. =) * At a prior company I once tried appending to the list of NFS exports, but dropped the "no-root-squash" option, and instantly denied write permissions to our entire VMware farm. You can imagine what then happened to all of the VMs for this mission critical customer. =P
  - tetha 10y ago
    As I keep saying, there are people who screw up big time and people who are too scared to touch the system. I managed to gun down three productive clusters by deploying NTP by accident along with a tiny change. Kerpow, 12 minutes of downtime, full network outage due to DHCP and such. Great fun.
- Alupis 10y ago
  Reminds me of the story about Bob Hoover.[1] He took off in his piston engine plane, only to lose power during the climb and was forced to make a crash landing. It turned out the airplane was fueled with jet fuel instead of regular gasoline (the ground crewman mistakenly thought the plane was a turbo prop). Instead of yelling at or firing the ground crewman, Hoover had this to say[2]: "There isn't a man alive who hasn't made a mistake. But I'm positive you'll never make this mistake again. That's why I want to make sure that you're the only one to refuel my plane tomorrow. I won't let anyone else on the field touch it." [1] https://www.aopa.org/news-and-media/all-news/2014/july/pilot/pilots https://www.aopa.org/news-and-media/all-news/2014/july/pilot... [2] http://www.squawkpoint.com/2014/01/criticism/ http://www.squawkpoint.com/2014/01/criticism/
dahdum 10y ago
At these scales it's the fault of the system, not the individual, so hopefully they don't come down hard on them.
- notheguyouthink 10y ago
  Agreed. They also seemed to acknowledge that in the post, as they mentioned improving the tool to not allow such destructive options.
  - joezydeco 10y ago
    You mean like range checking the input parameters to the command? =) SHUT DOWN S3? ARE YOU SURE? (y/N) :
    - unit91 10y ago
      More like: SHUT DOWN S3? ARE YOU SURE? ("I'm absolutely positive."/n) :
    - fixermark 10y ago
      Were it so easy. Likely more like "Shut down 73 servers? Are you sure? (y/N)": "Seventy-three? Wow, I hadn't realized our system grew that much. Probably a new backend dependency got added that I'm not familiar with yet; I'll look into it later." (Y)
      - TheDauthi 10y ago
        Every sysadmin at my previous job (a Fortune 500) would stop the moment that number is off the expected by a few machines, just long enough to verify it's correct. That may be due to having made a mistake like this once. I know that's true in several other large shops, as well. Source: I'm was the one the one they would call for our team... usually at 4:00 AM because one of our team members (which was also frequently me) didn't document something correctly.
    - flamedoge 10y ago
Johnny555 10y ago
I brought down our production system after a typo in a command once... the dev team took the blame for allowing an illegal parameter to bring down the system.
- alexvy86 10y ago
  Kudos to the dev team for that, I think most people wouldn't own that kind of issue
  - Johnny555 10y ago
    It was a very well-run engineering department where taking blame was not a career ending decision. I took full blame for the typo (at 3am trying to resolve a customer issue), but the dev team accepted full responsibility for letting it take down the system. Every mistake was used as a learning opportunity to ensure that the same and similar mistakes can't be repeated.
- unit91 10y ago
  I did once, too. Unfortunately, it was company of 4 people so I was also the dev team. :-) Never did it again though (yet).
- vhost- 10y ago
  Me too. I accidentally SSH'd into the wrong data center and ran a for loop restarting every database server once. Fixed that by putting the DC domain in red on the prompt.
omni 10y ago
They just spent millions of dollars training that person to never make this mistake again, should definitely keep them around.
- DaiPlusPlus 10y ago
  The wording of the article implies Amazon is shifting the blame entirely on the individual who typo'd: they indemnify themselves with "an authorized S3 team member using an established playbook..." ("don't blame us, our process is perfect!") There are process-fixes for this, such as requiring a two-person rule when at a production shell and modifying tooling to detect potentially unintentional commands (e.g. a SQL UPDATE without a WHERE) - but given what I know about Amazon's internal practices (i.e. the brutality) it wouldn't surprise me if they did terminate the unfortunate operator - not because they want to, but because AWS simply has too many large-scale customers who would demand immediate action like that.
  - unit91 10y ago
    Maybe so. I didn't take it that way. I took it as, "nobody panic, we weren't hacked."
    - fixermark 10y ago
      It's both. This isn't "our system was compromised by attack;" it's "SNAFU." Everyone who's had operations experience knows that there will be, as time approaches infinity, more than zero SNAFU. That's why companies offer five nines of uptime, not 100% uptime.
  - ckozlowski 10y ago
    Not at all. The "an authorized S3 team member using an established playbook..." is to establish that: 1. They were authorized to be doing it. 2. They were following an established process, and not winging it. The reason for that wording is to illustrate that operational changes are made in accordance with established CM. Mistakes happen. The system didn't catch the error. That's why the mitigation is "fix the system", and not "fire the team member". =)
asciimo 10y ago
I would love to hear their recollection of the day. Would make me feel better about my mistakes.

I wouldn't want to be the person who wrote the wrong command! Sheesh.

by ct0 10y ago

I wouldn't want to be the person who wrote the wrong command! Sheesh.

Meh, that's a process problem, not a people problem. Playbooks that have you retype commands with complex options, with no confirmation, etc, are inviting that sort of thing.

makira 10y ago

That person is the least likely to make that mistake again!

oblio 10y ago
No! Off with this head!!! On a more serious note, if you've never done something like this, you haven't had enough interesting projects. I've had a decent career and I still managed to: * re-deploy the current application version in all our data centers, instead of the new version, in a period when our deployment wasn't a 0-downtime one * rename all the Jenkins jobs on the server to the same name, thus deleting hundreds of Jenkins jobs in one fell swoop "Let him who is without sin cast the first stone" and all that :)
- ckozlowski 10y ago
  There's two things I've come to believe in my IT career regarding operations: 1. No organization anywhere is a paragon of excellence, and everyone can benefit from improvement. 2. Every organization is made up of humans just like you. With all that entails. Some things which seem blatantly obvious after the fact are easily overlooked when the pressure to deliver is high and other issues are taking precedence. I completely agree with your statement. In fact, when I do interviews, one of my favorite and most insightful questions to ask is, basically, "tell me about a time you screwed the pooch." If they don't have a story and they worked in ops, then it can suggest they didn't really do much. The really sharp ones I've interviewed have a good story or two (and can tell it in excruciating detail. =) * At a prior company I once tried appending to the list of NFS exports, but dropped the "no-root-squash" option, and instantly denied write permissions to our entire VMware farm. You can imagine what then happened to all of the VMs for this mission critical customer. =P
- tetha 10y ago
  As I keep saying, there are people who screw up big time and people who are too scared to touch the system. I managed to gun down three productive clusters by deploying NTP by accident along with a tiny change. Kerpow, 12 minutes of downtime, full network outage due to DHCP and such. Great fun.
Alupis 10y ago
Reminds me of the story about Bob Hoover.[1] He took off in his piston engine plane, only to lose power during the climb and was forced to make a crash landing. It turned out the airplane was fueled with jet fuel instead of regular gasoline (the ground crewman mistakenly thought the plane was a turbo prop). Instead of yelling at or firing the ground crewman, Hoover had this to say[2]: "There isn't a man alive who hasn't made a mistake. But I'm positive you'll never make this mistake again. That's why I want to make sure that you're the only one to refuel my plane tomorrow. I won't let anyone else on the field touch it." [1] https://www.aopa.org/news-and-media/all-news/2014/july/pilot/pilots https://www.aopa.org/news-and-media/all-news/2014/july/pilot... [2] http://www.squawkpoint.com/2014/01/criticism/ http://www.squawkpoint.com/2014/01/criticism/

dahdum 10y ago

At these scales it's the fault of the system, not the individual, so hopefully they don't come down hard on them.

notheguyouthink 10y ago
Agreed. They also seemed to acknowledge that in the post, as they mentioned improving the tool to not allow such destructive options.
- joezydeco 10y ago
  You mean like range checking the input parameters to the command? =) SHUT DOWN S3? ARE YOU SURE? (y/N) :
  - unit91 10y ago
    More like: SHUT DOWN S3? ARE YOU SURE? ("I'm absolutely positive."/n) :
  - fixermark 10y ago
    Were it so easy. Likely more like "Shut down 73 servers? Are you sure? (y/N)": "Seventy-three? Wow, I hadn't realized our system grew that much. Probably a new backend dependency got added that I'm not familiar with yet; I'll look into it later." (Y)
    - TheDauthi 10y ago
      Every sysadmin at my previous job (a Fortune 500) would stop the moment that number is off the expected by a few machines, just long enough to verify it's correct. That may be due to having made a mistake like this once. I know that's true in several other large shops, as well. Source: I'm was the one the one they would call for our team... usually at 4:00 AM because one of our team members (which was also frequently me) didn't document something correctly.
  - flamedoge 10y ago

Johnny555 10y ago

I brought down our production system after a typo in a command once... the dev team took the blame for allowing an illegal parameter to bring down the system.

alexvy86 10y ago
Kudos to the dev team for that, I think most people wouldn't own that kind of issue
- Johnny555 10y ago
  It was a very well-run engineering department where taking blame was not a career ending decision. I took full blame for the typo (at 3am trying to resolve a customer issue), but the dev team accepted full responsibility for letting it take down the system. Every mistake was used as a learning opportunity to ensure that the same and similar mistakes can't be repeated.
unit91 10y ago
I did once, too. Unfortunately, it was company of 4 people so I was also the dev team. :-) Never did it again though (yet).
vhost- 10y ago
Me too. I accidentally SSH'd into the wrong data center and ran a for loop restarting every database server once. Fixed that by putting the DC domain in red on the prompt.

omni 10y ago

They just spent millions of dollars training that person to never make this mistake again, should definitely keep them around.

DaiPlusPlus 10y ago
The wording of the article implies Amazon is shifting the blame entirely on the individual who typo'd: they indemnify themselves with "an authorized S3 team member using an established playbook..." ("don't blame us, our process is perfect!") There are process-fixes for this, such as requiring a two-person rule when at a production shell and modifying tooling to detect potentially unintentional commands (e.g. a SQL UPDATE without a WHERE) - but given what I know about Amazon's internal practices (i.e. the brutality) it wouldn't surprise me if they did terminate the unfortunate operator - not because they want to, but because AWS simply has too many large-scale customers who would demand immediate action like that.
- unit91 10y ago
  Maybe so. I didn't take it that way. I took it as, "nobody panic, we weren't hacked."
  - fixermark 10y ago
    It's both. This isn't "our system was compromised by attack;" it's "SNAFU." Everyone who's had operations experience knows that there will be, as time approaches infinity, more than zero SNAFU. That's why companies offer five nines of uptime, not 100% uptime.
- ckozlowski 10y ago
  Not at all. The "an authorized S3 team member using an established playbook..." is to establish that: 1. They were authorized to be doing it. 2. They were following an established process, and not winging it. The reason for that wording is to illustrate that operational changes are made in accordance with established CM. Mistakes happen. The system didn't catch the error. That's why the mitigation is "fix the system", and not "fire the team member". =)

asciimo 10y ago

I would love to hear their recollection of the day. Would make me feel better about my mistakes.