6 ms·

Why would you fire an engineer you have just spent millions to train?

by kroolik 5y ago

goldcd 5y ago
I'd guess because Europe-wide outages are costing more than millions
- kreeben 5y ago
  But now you have someone in your team who will never, ever make that same mistake again and should be your new go-to guy for all X related changes (X being DNS or what-have-you). Firing someone with that type of experience does not lead to success. 100% of all devs make huge mistakes, at least once.
  - KronisLV 5y ago
    > But now you have someone in your team who will never, ever make that same mistake again and that should be your new go-to guy for all DNS related changes. I'm not entirely sure that's always true. For example, i've seen people introduce N+1 issues into a codebase, spend evenings fixing them and refactoring code to fix production issues... just to later introduce those very same types of issues. Sure, you can learn from mistakes, have post-mortems and so on (provided that your org even does those and that anyone listens and cares about the conclusions from those), but to me it feels like the most foolproof way is to ensure that no-one can make these mistakes again, be it with a checklist (which tend to be ignored, honestly), or better yet, an automated CI step or a new test suite. In my eyes, it's basically the same as with unit tests - everyone agrees that you need them, but people rarely write enough of them. So if you introduce something to prevent them from not doing what they should, e.g. a quality gate within a CI step which will disallow a merge once the coverage falls below a set margin, suddenly things are a lot better in the long run.
    - kreeben 5y ago
      N+1 issues aren't nearly as devastating as N^2. Commend them for not putting your systems to a complete halt, then teach them how to reason about this properly. >> a quality gate Yes, this, also.
      - KronisLV 5y ago
        > N+1 issues aren't nearly as devastating as N^2. Depends on the project, i guess: if you're unlucky enough to be working on a monolith and suddenly a page takes 5'000 SQL queries to load as opposed to 100, because someone thought that initializing data through service/DB calls in a loop is "easier" than writing views in the DB, it might still kill the entire system anyways, depending on the count of users. And once this data initialization is sufficiently complicated and convoluted for you not to be able to rewrite it and them not wanting to rewrite it, all while "the business" is breathing down on your neck, you might either want to introduce caching (and possibly run into cache invalidation problems down the road), or just freshen up your CV. I guess i'd also like to expand on the previous suggestion and advise others to consider performance/load testing as well, especially when coupled with APM solutions like Skywalking or even Matomo analytics, both of which can allow you to aggregate the historical page load times, CPM and overall performance of your applications, to figure out what went wrong when.
- kroolik 5y ago
  Still, that engineer (if its engineer's fault) is extremely unlikely to make that issue again. IMHO, the problem is systemic in that why the system allows such errors (if its human error) to happen. Given Google's scale I think a lot of the generally known scenarios are covered and what you see is tens of services interacting in not obvious ways. Those unobvious scenarios manifest in situations like this.
- cranekam 5y ago
  But firing someone doesn't undo an incident. It just introduces other weird incentives. People become afraid to change things for fear of breaking something, or when something does break they try to hide it rather than feeling like they can immediately ask for and receive help to fix it. The only time someone should be fired for causing an outage is if they're negligent or sloppy or mess things up all the time. This is rare. Almost always outages in large systems are the combination of many factors — latent bugs, design flaws, abnormal load, etc, any one or two of which wouldn't take the site down. But when the combine in a perfect storm that nobody foresaw things fall over.
chrisseaton 5y ago
What tech company spends millions on training anyone?
- dukeyukey 5y ago
  Someone who just cost their company millions in revenues is gonna be _extremely_ careful not to make the same mistake again. Hence, million-dollsr training.
- l33t2328 5y ago
  The implication is that millions were spent training the person who made the mistake when they cost the company millions by making that mistake.
- Ueland 5y ago
  An outage can be pretty expensive but it is training for those whom triggered it and/or those that fixes it.
- pestaa 5y ago
  It's reframing lost revenue, not talking about literal training cost.
- 1ibsq 5y ago
  The other comments already explained it, but I'm wondering how you haven't come across this 'saying' before. It's so overused and also cheesy in my opinion.
  - chrisseaton 5y ago
    Lol maybe if people actually did spend millions training their people up front we could do better?
    - exikyut 5y ago
      139,995 employees at Google * 1,000,000 = $139,995,000,000 $140 billion dollars. On training. On the one hand... you know what, I'd love to work in an environment like that. Seriously. On the other hand... what's the argument you make to the CFO in support of this? Honest question, interested to hear answers.
deleted 5y ago
[deleted]

Why would you fire an engineer you have just spent millions to train?

goldcd 5y ago

I'd guess because Europe-wide outages are costing more than millions

kreeben 5y ago
But now you have someone in your team who will never, ever make that same mistake again and should be your new go-to guy for all X related changes (X being DNS or what-have-you). Firing someone with that type of experience does not lead to success. 100% of all devs make huge mistakes, at least once.
- KronisLV 5y ago
  > But now you have someone in your team who will never, ever make that same mistake again and that should be your new go-to guy for all DNS related changes. I'm not entirely sure that's always true. For example, i've seen people introduce N+1 issues into a codebase, spend evenings fixing them and refactoring code to fix production issues... just to later introduce those very same types of issues. Sure, you can learn from mistakes, have post-mortems and so on (provided that your org even does those and that anyone listens and cares about the conclusions from those), but to me it feels like the most foolproof way is to ensure that no-one can make these mistakes again, be it with a checklist (which tend to be ignored, honestly), or better yet, an automated CI step or a new test suite. In my eyes, it's basically the same as with unit tests - everyone agrees that you need them, but people rarely write enough of them. So if you introduce something to prevent them from not doing what they should, e.g. a quality gate within a CI step which will disallow a merge once the coverage falls below a set margin, suddenly things are a lot better in the long run.
  - kreeben 5y ago
    N+1 issues aren't nearly as devastating as N^2. Commend them for not putting your systems to a complete halt, then teach them how to reason about this properly. >> a quality gate Yes, this, also.
    - KronisLV 5y ago
      > N+1 issues aren't nearly as devastating as N^2. Depends on the project, i guess: if you're unlucky enough to be working on a monolith and suddenly a page takes 5'000 SQL queries to load as opposed to 100, because someone thought that initializing data through service/DB calls in a loop is "easier" than writing views in the DB, it might still kill the entire system anyways, depending on the count of users. And once this data initialization is sufficiently complicated and convoluted for you not to be able to rewrite it and them not wanting to rewrite it, all while "the business" is breathing down on your neck, you might either want to introduce caching (and possibly run into cache invalidation problems down the road), or just freshen up your CV. I guess i'd also like to expand on the previous suggestion and advise others to consider performance/load testing as well, especially when coupled with APM solutions like Skywalking or even Matomo analytics, both of which can allow you to aggregate the historical page load times, CPM and overall performance of your applications, to figure out what went wrong when.
kroolik 5y ago
Still, that engineer (if its engineer's fault) is extremely unlikely to make that issue again. IMHO, the problem is systemic in that why the system allows such errors (if its human error) to happen. Given Google's scale I think a lot of the generally known scenarios are covered and what you see is tens of services interacting in not obvious ways. Those unobvious scenarios manifest in situations like this.
cranekam 5y ago
But firing someone doesn't undo an incident. It just introduces other weird incentives. People become afraid to change things for fear of breaking something, or when something does break they try to hide it rather than feeling like they can immediately ask for and receive help to fix it. The only time someone should be fired for causing an outage is if they're negligent or sloppy or mess things up all the time. This is rare. Almost always outages in large systems are the combination of many factors — latent bugs, design flaws, abnormal load, etc, any one or two of which wouldn't take the site down. But when the combine in a perfect storm that nobody foresaw things fall over.

chrisseaton 5y ago

What tech company spends millions on training anyone?

dukeyukey 5y ago
Someone who just cost their company millions in revenues is gonna be _extremely_ careful not to make the same mistake again. Hence, million-dollsr training.
l33t2328 5y ago
The implication is that millions were spent training the person who made the mistake when they cost the company millions by making that mistake.
Ueland 5y ago
An outage can be pretty expensive but it is training for those whom triggered it and/or those that fixes it.
pestaa 5y ago
It's reframing lost revenue, not talking about literal training cost.
1ibsq 5y ago
The other comments already explained it, but I'm wondering how you haven't come across this 'saying' before. It's so overused and also cheesy in my opinion.
- chrisseaton 5y ago
  Lol maybe if people actually did spend millions training their people up front we could do better?
  - exikyut 5y ago
    139,995 employees at Google * 1,000,000 = $139,995,000,000 $140 billion dollars. On training. On the one hand... you know what, I'd love to work in an environment like that. Seriously. On the other hand... what's the argument you make to the CFO in support of this? Honest question, interested to hear answers.

deleted 5y ago

[deleted]