3 ms·

That person is the least likely to make that mistake again!

by makira 10y ago

oblio 10y ago
No! Off with this head!!! On a more serious note, if you've never done something like this, you haven't had enough interesting projects. I've had a decent career and I still managed to: * re-deploy the current application version in all our data centers, instead of the new version, in a period when our deployment wasn't a 0-downtime one * rename all the Jenkins jobs on the server to the same name, thus deleting hundreds of Jenkins jobs in one fell swoop "Let him who is without sin cast the first stone" and all that :)
- ckozlowski 10y ago
  There's two things I've come to believe in my IT career regarding operations: 1. No organization anywhere is a paragon of excellence, and everyone can benefit from improvement. 2. Every organization is made up of humans just like you. With all that entails. Some things which seem blatantly obvious after the fact are easily overlooked when the pressure to deliver is high and other issues are taking precedence. I completely agree with your statement. In fact, when I do interviews, one of my favorite and most insightful questions to ask is, basically, "tell me about a time you screwed the pooch." If they don't have a story and they worked in ops, then it can suggest they didn't really do much. The really sharp ones I've interviewed have a good story or two (and can tell it in excruciating detail. =) * At a prior company I once tried appending to the list of NFS exports, but dropped the "no-root-squash" option, and instantly denied write permissions to our entire VMware farm. You can imagine what then happened to all of the VMs for this mission critical customer. =P
- tetha 10y ago
  As I keep saying, there are people who screw up big time and people who are too scared to touch the system. I managed to gun down three productive clusters by deploying NTP by accident along with a tiny change. Kerpow, 12 minutes of downtime, full network outage due to DHCP and such. Great fun.
Alupis 10y ago
Reminds me of the story about Bob Hoover.[1] He took off in his piston engine plane, only to lose power during the climb and was forced to make a crash landing. It turned out the airplane was fueled with jet fuel instead of regular gasoline (the ground crewman mistakenly thought the plane was a turbo prop). Instead of yelling at or firing the ground crewman, Hoover had this to say[2]: "There isn't a man alive who hasn't made a mistake. But I'm positive you'll never make this mistake again. That's why I want to make sure that you're the only one to refuel my plane tomorrow. I won't let anyone else on the field touch it." [1] https://www.aopa.org/news-and-media/all-news/2014/july/pilot/pilots https://www.aopa.org/news-and-media/all-news/2014/july/pilot... [2] http://www.squawkpoint.com/2014/01/criticism/ http://www.squawkpoint.com/2014/01/criticism/

That person is the least likely to make that mistake again!

by makira 10y ago

That person is the least likely to make that mistake again!

oblio 10y ago

No! Off with this head!!! On a more serious note, if you've never done something like this, you haven't had enough interesting projects. I've had a decent career and I still managed to: * re-deploy the current application version in all our data centers, instead of the new version, in a period when our deployment wasn't a 0-downtime one * rename all the Jenkins jobs on the server to the same name, thus deleting hundreds of Jenkins jobs in one fell swoop "Let him who is without sin cast the first stone" and all that :)

ckozlowski 10y ago
There's two things I've come to believe in my IT career regarding operations: 1. No organization anywhere is a paragon of excellence, and everyone can benefit from improvement. 2. Every organization is made up of humans just like you. With all that entails. Some things which seem blatantly obvious after the fact are easily overlooked when the pressure to deliver is high and other issues are taking precedence. I completely agree with your statement. In fact, when I do interviews, one of my favorite and most insightful questions to ask is, basically, "tell me about a time you screwed the pooch." If they don't have a story and they worked in ops, then it can suggest they didn't really do much. The really sharp ones I've interviewed have a good story or two (and can tell it in excruciating detail. =) * At a prior company I once tried appending to the list of NFS exports, but dropped the "no-root-squash" option, and instantly denied write permissions to our entire VMware farm. You can imagine what then happened to all of the VMs for this mission critical customer. =P
tetha 10y ago
As I keep saying, there are people who screw up big time and people who are too scared to touch the system. I managed to gun down three productive clusters by deploying NTP by accident along with a tiny change. Kerpow, 12 minutes of downtime, full network outage due to DHCP and such. Great fun.

Alupis 10y ago

Reminds me of the story about Bob Hoover.[1] He took off in his piston engine plane, only to lose power during the climb and was forced to make a crash landing. It turned out the airplane was fueled with jet fuel instead of regular gasoline (the ground crewman mistakenly thought the plane was a turbo prop). Instead of yelling at or firing the ground crewman, Hoover had this to say[2]: "There isn't a man alive who hasn't made a mistake. But I'm positive you'll never make this mistake again. That's why I want to make sure that you're the only one to refuel my plane tomorrow. I won't let anyone else on the field touch it." [1] https://www.aopa.org/news-and-media/all-news/2014/july/pilot/pilots https://www.aopa.org/news-and-media/all-news/2014/july/pilot... [2] http://www.squawkpoint.com/2014/01/criticism/ http://www.squawkpoint.com/2014/01/criticism/