4 ms·

I've seen individual engineers make a mistake that caused 100 engineer's salary worth of damage.

alexchamberlain 6y ago
I suspect we all have. The question is rather: how much would it have cost to prevent that mistake? The flip side is the insurance argument: given a big enough organisation, a mistake like that will happen, even if the probability of each individual mistake is very low. So assuming one of these will happen, what can you do to reduce the impact or to reduce the latency of fixing such a mistake?
gjvc 6y ago
I would love to know more about the details of this example :-)
- gwd 6y ago
  So the company I work for made the front page of HN last year for a security breach. I won't go into details, but I will say one factor was a particular instance where security was overlooked for many years due to convenience. In response, the company has now sort of swung the other way: lots of things that have a perfectly valid reason and are reasonably safe have been forbidden or suddenly have needed to be justified in the face of extreme skepticism. That wasn't a case of a single engineer causing 100x engineers worth of damage, but it was the case where something made hundreds of people's jobs a tiny bit easier for two decades, then caused a massive public loss of face and who knows how much reputational damage. And I'd be willing to bet that a fair number of HN readers work at a company which has at least one system with similar properties (a bit more convenient, a bit less secure) to the one implicated in our security breach; but failure is a once-a-decade event, and their company isn't nearly that old yet. Would those engineers defend that setup using the argument from this article?
  - dtech 6y ago
    Was it the Twitter internal account management tool?
    - gwd 6y ago
      Nope. :-)
- DaiPlusPlus 6y ago
  Here's an example: https://news.ycombinator.com/item?id=24196131 https://news.ycombinator.com/item?id=24196131
- cecilpl2 6y ago
  Here's another famous example where the damage was thousands of person-years of salary: https://www.henricodolfing.com/2019/06/project-failure-case-study-knight-capital.html https://www.henricodolfing.com/2019/06/project-failure-case-...
moritonal 6y ago
"Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. The servers that were inadvertently removed supported two other S3 subsystems." [1] 1. https://aws.amazon.com/message/41926/?ascsubtag=e7c5af5933bd8f1cdce5a10744b6fdc6a7575617&tag=gizmodoamzn-20 https://aws.amazon.com/message/41926/?ascsubtag=e7c5af5933bd...
Aeolun 6y ago
Indeed, but did those 100 engineers increase the company bottom line by 10% more because they weren’t hindered by too much red tape, and is that value a multiple of their salary? Also, do you expect mistakes of that magnitude to keep being made?
leprechaun1066 6y ago
The Knight Capital error really messed them up. https://www.reuters.com/article/us-knightcapital-results/knight-capital-posts-389-9-million-loss-on-trading-glitch-idUSBRE89G0HI20121017 https://www.reuters.com/article/us-knightcapital-results/kni...

I suspect we all have. The question is rather: how much would it have cost to prevent that mistake? The flip side is the insurance argument: given a big enough organisation, a mistake like that will happen, even if the probability of each individual mistake is very low. So assuming one of these will happen, what can you do to reduce the impact or to reduce the latency of fixing such a mistake?

gjvc 6y ago

I would love to know more about the details of this example :-)

gwd 6y ago
So the company I work for made the front page of HN last year for a security breach. I won't go into details, but I will say one factor was a particular instance where security was overlooked for many years due to convenience. In response, the company has now sort of swung the other way: lots of things that have a perfectly valid reason and are reasonably safe have been forbidden or suddenly have needed to be justified in the face of extreme skepticism. That wasn't a case of a single engineer causing 100x engineers worth of damage, but it was the case where something made hundreds of people's jobs a tiny bit easier for two decades, then caused a massive public loss of face and who knows how much reputational damage. And I'd be willing to bet that a fair number of HN readers work at a company which has at least one system with similar properties (a bit more convenient, a bit less secure) to the one implicated in our security breach; but failure is a once-a-decade event, and their company isn't nearly that old yet. Would those engineers defend that setup using the argument from this article?
- dtech 6y ago
  Was it the Twitter internal account management tool?
  - gwd 6y ago
    Nope. :-)
DaiPlusPlus 6y ago
Here's an example: https://news.ycombinator.com/item?id=24196131 https://news.ycombinator.com/item?id=24196131
cecilpl2 6y ago
Here's another famous example where the damage was thousands of person-years of salary: https://www.henricodolfing.com/2019/06/project-failure-case-study-knight-capital.html https://www.henricodolfing.com/2019/06/project-failure-case-...

moritonal 6y ago

"Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. The servers that were inadvertently removed supported two other S3 subsystems." [1] 1. https://aws.amazon.com/message/41926/?ascsubtag=e7c5af5933bd8f1cdce5a10744b6fdc6a7575617&tag=gizmodoamzn-20 https://aws.amazon.com/message/41926/?ascsubtag=e7c5af5933bd...

Aeolun 6y ago

Indeed, but did those 100 engineers increase the company bottom line by 10% more because they weren’t hindered by too much red tape, and is that value a multiple of their salary? Also, do you expect mistakes of that magnitude to keep being made?

leprechaun1066 6y ago

The Knight Capital error really messed them up. https://www.reuters.com/article/us-knightcapital-results/knight-capital-posts-389-9-million-loss-on-trading-glitch-idUSBRE89G0HI20121017 https://www.reuters.com/article/us-knightcapital-results/kni...