4 ms·

> Heads should probably roll here, I disagree. It's obviously a major balls up by multiple people, but at the end of the day this is an organizational failure.

by mdoms 5y ago

> Heads should probably roll here, I disagree. It's obviously a major balls up by multiple people, but at the end of the day this is an organizational failure. You don't fire people for this, you learn from it and fix the organizational holes that caused it.

sneak 5y ago
Someone is responsible for the organizational units and their processes. By that logic, almost no individual failure is a firing offense. Many people in the organization are responsible for ensuring that the processes themselves overlap in ways that exclude the possibility of failures like this. I'm talking about the meta-failure in management's oversight of the processes. There is belt person, and there is suspenders person, and there is a third person who makes sure there are both belt person and suspenders person on staff and that they don't go on vacation at the same time. Somewhere above the team lead and below the CEO there is someone who didn't do their job.
- nicoburns 5y ago
  > By that logic, almost no individual failure is a firing offence I'd probably agree with that. Firing should be for extreme negligence, or sustained underperformance.
  - idiotsecant 5y ago
    >extreme negligence If this doesn't qualify, what does?
    - sneak 5y ago
      I would say this is negligence, but perhaps not extremely so.
      - anyfoo 5y ago
        You are all very quick to judge. I don't work for Google so I don't know what happened, but consider the possibility that this may have been a freak accident. That piece of code for example might only be on a path that is not activated when testing, or it may be executed but not have the same fatal effect. And before anyone comes along and says "well, every possible path should be tested then": That would be absolutely impossible even if the halting problem wasn't a thing (which it very much is). Even if you somehow have utopian "100% code coverage", that does not mean you have tested every possible input resulting in every possible state combination. I bet every one of us has a story about an old bug that only manifested years[1] later because of the confluence of many unfortunate things, one that takes half an hour to tell. [1] Or even decades, in some cases that then made it to the HN front page.
        nicoburns 5y ago
        This is the sort of thing that most linting tools will catch, so it's a bit worse than that I think. You shouldn't need any test coverage to catch this.
        Veserv 5y ago
        There are freak accidents and then there are complete lapses is basic critical functionality that demonstrate a total lack of competence. The article indicates that every person who updated to the latest version could not login. This is a bug that clearly presents itself, every time, in essentially every use case, and is perfectly reproducible. It affects critical functionality that any layperson can recognize is one of the most critical components of a general purpose computing system. The greatest degree of scrutiny should be applied to this system, yet what we see is the functional equivalent of a car factory forgetting to put a wheel on every car it produces. Sure, there could be a freak accident that makes the 4th wheel robot malfunction, but other stages of the process should catch such an egregious visible failure. Any process that would allow such basic errors to occur without checking or fixing them elsewhere is terrible on its face. If that is too abstract, imagine you hired a construction company to build a house and they forgot to put in an entire wall. Maybe you just got unlucky and you were the 1 in 1 billion person who gets such a stupid error to happen to them, but the much more likely theory is that they are grossly incompetent, or at the very least far less reliable than you were led to believe.
    - vlovich123 5y ago
      This isn’t extreme negligence of any single individual. If there are linters/compiler warnings that should have caught this, this is an organizational failure that can be remedied and the manager/TL should be taking a bit of a hit here as having those enabled is best practices. Since it’s ChromeOS I imagine there’s a dedicated team managing the build infra and it’s not using the base set of things that Google3 protects you with, so the fault may lie in that organization a bit as an oversight. Then there’s obviously the automated testing strategy, manual testing strategy and sanitizer strategy that missed this. The net result is extreme negligence but to me this kind of failure speaks to several layers going wrong at once rather than one person fucking up badly. Organizationally you try to defend against these kinds of failures with multiple layers of defenses. Now if this was a malicious internal attack that should be investigated but I’d hate to have to seriously reprimand anyone who typo’ed & instead of &&.
      - ectopod 5y ago
        The GGGP is talking about failures at a higher level. Not reprimanding the coder who missed an ampersand, but the manager who was ultimately responsibly for ensuring there was a functioning process for catching this sort of nonsense.
        nicoburns 5y ago
        Reprimanding is one thing and firing another though. I certainly think the manager in charge deserves a reprimand.
        ascagnel_ 5y ago
        I wouldn't even say a reprimand is necessary here. Programming safety is like safety in auto racing or rock climbing: you build the safest thing you can, but the activity you're doing is inherently unsafe, and it will generate failures despite the presence safety measures, so you do your best to study those failures to mitigate them the best you can. The only time adverse action like a reprimand is necessary is if the manager or tech lead don't take any action to prevent future issues. The only time a firing is necessary is if the security issue was created with intent, rather than accidentally.
- admax88q 5y ago
  The way you fix organizational/process failures is not by firing the person in charge of them every time they fail to catch something. How many breaking bugs have _NOT_ shipped due to the current processes. Your approach suggests that there is some perfect process manager out there who will let no bugs like this ship (whilst not dropping any other business priorities mind you) that they just haven't hired yet into this position. This isn't as big of a bug as people on HN want it to be. A few months from few people will remember, and even fewer will care.
- yupper32 5y ago
  > By that logic, almost no individual failure is a firing offense. That's pretty close to how it should be. That's the core premise of the no-blame post-mortem culture that's great about some companies. As long as you go through the port-mortem process, which includes doing your best to prevent this from happening again (which may include changing organizational processes, testing strategies, etc), then everyone can move on with their lives. Firing happens when you have a history of performing below your level. Never for a single engineering failure.
  - sneak 5y ago
    This isn't an engineering failure. Everyone writes bugs.
    - yupper32 5y ago
      Engineering process is still engineering in my book, but if it helps: Firing happens when you have a history of performing below your level. Never for a single process failure.
- AnimalMuppet 5y ago
  There is a story (perhaps apocryphal) of a salesman who lost a $10 million deal for IBM. He was sure he was going to be fired. Some bigwig (Watson himself, maybe?) talked him through what had gone wrong. The salesman asked, "Aren't you going to fire me?" The bigwig said, "I just spent 10 million dollars training you."
- mdoms 5y ago
  > By that logic, almost no individual failure is a firing offense. Correct.
aeturnum 5y ago
I don't think we know enough from this to say. It's possible that this bug made it into the wild because the organizational structure made remedies impossible. In that case you probably don't fire anyone, though maybe you reassign some folks. It's also possible that the organizational structure indicates that a person or a group of people is responsible for this and they failed to do that job. If that's the case then it might very well make sense to fire folks. It also might not. I think it's important for poor performance to have consequences[1] and I also think that people leave jobs all the time and it's fine to decide that someone should leave their place in the org with a positive reference and look elsewhere for work (perhaps in another part of goog). [1] for moral and company culture reasons if nothing else.

by mdoms 5y ago

sneak 5y ago

Someone is responsible for the organizational units and their processes. By that logic, almost no individual failure is a firing offense. Many people in the organization are responsible for ensuring that the processes themselves overlap in ways that exclude the possibility of failures like this. I'm talking about the meta-failure in management's oversight of the processes. There is belt person, and there is suspenders person, and there is a third person who makes sure there are both belt person and suspenders person on staff and that they don't go on vacation at the same time. Somewhere above the team lead and below the CEO there is someone who didn't do their job.

nicoburns 5y ago
> By that logic, almost no individual failure is a firing offence I'd probably agree with that. Firing should be for extreme negligence, or sustained underperformance.
- idiotsecant 5y ago
  >extreme negligence If this doesn't qualify, what does?
  - sneak 5y ago
    I would say this is negligence, but perhaps not extremely so.
    - anyfoo 5y ago
      You are all very quick to judge. I don't work for Google so I don't know what happened, but consider the possibility that this may have been a freak accident. That piece of code for example might only be on a path that is not activated when testing, or it may be executed but not have the same fatal effect. And before anyone comes along and says "well, every possible path should be tested then": That would be absolutely impossible even if the halting problem wasn't a thing (which it very much is). Even if you somehow have utopian "100% code coverage", that does not mean you have tested every possible input resulting in every possible state combination. I bet every one of us has a story about an old bug that only manifested years[1] later because of the confluence of many unfortunate things, one that takes half an hour to tell. [1] Or even decades, in some cases that then made it to the HN front page.
      - nicoburns 5y ago
        This is the sort of thing that most linting tools will catch, so it's a bit worse than that I think. You shouldn't need any test coverage to catch this.
      - Veserv 5y ago
        There are freak accidents and then there are complete lapses is basic critical functionality that demonstrate a total lack of competence. The article indicates that every person who updated to the latest version could not login. This is a bug that clearly presents itself, every time, in essentially every use case, and is perfectly reproducible. It affects critical functionality that any layperson can recognize is one of the most critical components of a general purpose computing system. The greatest degree of scrutiny should be applied to this system, yet what we see is the functional equivalent of a car factory forgetting to put a wheel on every car it produces. Sure, there could be a freak accident that makes the 4th wheel robot malfunction, but other stages of the process should catch such an egregious visible failure. Any process that would allow such basic errors to occur without checking or fixing them elsewhere is terrible on its face. If that is too abstract, imagine you hired a construction company to build a house and they forgot to put in an entire wall. Maybe you just got unlucky and you were the 1 in 1 billion person who gets such a stupid error to happen to them, but the much more likely theory is that they are grossly incompetent, or at the very least far less reliable than you were led to believe.
  - vlovich123 5y ago
    This isn’t extreme negligence of any single individual. If there are linters/compiler warnings that should have caught this, this is an organizational failure that can be remedied and the manager/TL should be taking a bit of a hit here as having those enabled is best practices. Since it’s ChromeOS I imagine there’s a dedicated team managing the build infra and it’s not using the base set of things that Google3 protects you with, so the fault may lie in that organization a bit as an oversight. Then there’s obviously the automated testing strategy, manual testing strategy and sanitizer strategy that missed this. The net result is extreme negligence but to me this kind of failure speaks to several layers going wrong at once rather than one person fucking up badly. Organizationally you try to defend against these kinds of failures with multiple layers of defenses. Now if this was a malicious internal attack that should be investigated but I’d hate to have to seriously reprimand anyone who typo’ed & instead of &&.
    - ectopod 5y ago
      The GGGP is talking about failures at a higher level. Not reprimanding the coder who missed an ampersand, but the manager who was ultimately responsibly for ensuring there was a functioning process for catching this sort of nonsense.
      - nicoburns 5y ago
        Reprimanding is one thing and firing another though. I certainly think the manager in charge deserves a reprimand.
        ascagnel_ 5y ago
        I wouldn't even say a reprimand is necessary here. Programming safety is like safety in auto racing or rock climbing: you build the safest thing you can, but the activity you're doing is inherently unsafe, and it will generate failures despite the presence safety measures, so you do your best to study those failures to mitigate them the best you can. The only time adverse action like a reprimand is necessary is if the manager or tech lead don't take any action to prevent future issues. The only time a firing is necessary is if the security issue was created with intent, rather than accidentally.
admax88q 5y ago
The way you fix organizational/process failures is not by firing the person in charge of them every time they fail to catch something. How many breaking bugs have _NOT_ shipped due to the current processes. Your approach suggests that there is some perfect process manager out there who will let no bugs like this ship (whilst not dropping any other business priorities mind you) that they just haven't hired yet into this position. This isn't as big of a bug as people on HN want it to be. A few months from few people will remember, and even fewer will care.
yupper32 5y ago
> By that logic, almost no individual failure is a firing offense. That's pretty close to how it should be. That's the core premise of the no-blame post-mortem culture that's great about some companies. As long as you go through the port-mortem process, which includes doing your best to prevent this from happening again (which may include changing organizational processes, testing strategies, etc), then everyone can move on with their lives. Firing happens when you have a history of performing below your level. Never for a single engineering failure.
- sneak 5y ago
  This isn't an engineering failure. Everyone writes bugs.
  - yupper32 5y ago
    Engineering process is still engineering in my book, but if it helps: Firing happens when you have a history of performing below your level. Never for a single process failure.
AnimalMuppet 5y ago
There is a story (perhaps apocryphal) of a salesman who lost a $10 million deal for IBM. He was sure he was going to be fired. Some bigwig (Watson himself, maybe?) talked him through what had gone wrong. The salesman asked, "Aren't you going to fire me?" The bigwig said, "I just spent 10 million dollars training you."
mdoms 5y ago
> By that logic, almost no individual failure is a firing offense. Correct.

aeturnum 5y ago

I don't think we know enough from this to say. It's possible that this bug made it into the wild because the organizational structure made remedies impossible. In that case you probably don't fire anyone, though maybe you reassign some folks. It's also possible that the organizational structure indicates that a person or a group of people is responsible for this and they failed to do that job. If that's the case then it might very well make sense to fire folks. It also might not. I think it's important for poor performance to have consequences[1] and I also think that people leave jobs all the time and it's fine to decide that someone should leave their place in the org with a positive reference and look elsewhere for work (perhaps in another part of goog). [1] for moral and company culture reasons if nothing else.