3 ms·

It's quite impressive it kept working for that long, and even more amazing someone was capable of saying "no, let's not cut the power, there might be a problem

It's quite impressive it kept working for that long, and even more amazing someone was capable of saying "no, let's not cut the power, there might be a problem if we do"

Aeolun 2y ago
Honestly, that doesn’t surprise me at all. This seems perfectly in line with “I don’t care what it is, or how well I perform it, as long as I have a job.”
- hinkley 2y ago
  Few things in my career have granted me more power faster than taking ownership of decisions. Some people are so terrified of having to say "I was wrong" that they either will do nothing, or just happily follow decisions that someone else makes. It's one thing when that's a line employee behaving that way, but there are managers so terrified of making a mistake that all they really are is scribes, tracking who said what and who decided what. These weasels don't lead. They don't contribute anything. They just have a quick answer when someone above them asks how this fuckup happened. Well that was Jameson. He decided to do X so clearly it's his fault things broke. I was in a room once with a guy who was so caught up in the idea of catching me in a gotcha that he had no idea what to do when I said, "yep, that's wrong, and this is how we fix it." His face went through the same roller coaster of emotions that happens when you drop your ice cream cone. I got to say most of what I needed to say before he even recovered. I think I was proudest of keeping a straight face during the entire conversation. Where has intellectual honesty gone that just being transparent verges on a superpower? I feel like I'm taking crazy pills.
  - refulgentis 2y ago
    Taking ownership of decisions in this way breeds another problem: people love to police when they perceive other people taking authority that's not rightfully theirs. I had your view until 7 years at Google. Showing this to 80% of people there would garner tsktsking and quips about delusions of grandeur and seeming unbalanced. In performance reviews, you could count on this person saying you collaborate poorly and don't take others concerns into account, because they felt humiliated and retconned a reason why it's your fault. (To be fair, given your vivid understanding of their face at the time, it sounds like you understand they wouldn't feel well after you did this) In the higher rooms, I saw at least a couple times everyone living this worldview and it ending up in bickering and extreme insistence the other person was missing fundamentals while nothing got done for months: which in the more Machiavellian parlance, is still good if you're good, because most people won't do the work, just argue over the 20% of the problem they understand, and someone has to win eventually I keep editing this comment because I get the vibe it disagrees with you, but it doesn't, at all. If anything, your framing helps sharpen why it felt like an odd place, in a bad way, to me from day 30.
    - exmadscientist 2y ago
      I've known enough people at Google that I've gotten a decent view into the broken-ass culture there, from the outside looking in, and I'd say that's a pretty decent summary of some of their worst problems.
    - nine_k 2y ago
      One of the things that impressed me at my time at Google (2011-2015) was the culture of blameless post-mortems, and of general transparency of many (not all) things. Sadly, this culture must have deteriorated; my friends who are still at Google say that it's become literally the stuff of Dilbert comic strips :-(
      - refulgentis 2y ago
        Hah, somewhat perversely, I'm glad to hear that: it's exactly how I sum things up in a sentence in my head, then I think I'm being overdramatic / personalizing it, because I kinda grew up learning about tech via Dilbert, I was a big fan of Dogbert at 7 years old. But I'm not. I did Aug 2016-2023. Of course it's a complicated story, and you've heard a lot already I'm sure, but when you mention blameless postmortems, it remind me of my pet peeve: promo committees done within teams (basically 2 levels up, i.e. skip is doling out promo, with no recourse or real outside input) was a horrible, wretched idea. There was this odd obsession with saving time on perf, but they kinda just needed to tell people to chill and recognize that it really is important for management. With all promos being judged by people next to you, what we might call "being agreeable" became a necessity. I distinctly remember in 2018 being told if things were how they had been, that I should just do a blameless postmortem, but with things as they are, that was a quick way to end up on the bench for the rest of my career.
    - msteffen 2y ago
      Interestingly, I was at Google during I think the very end of its golden years (2011-2016). I loved it and when I heard about how sclerotic it had gotten was pretty surprised. Looking back, IMO the seeds of its downfall had already been planted by the time I was there, including the culture of blameless postmortems. There was this idea that any problem—any outage or whatever —reflected not stupidity on the part of the perpetrator (Google hires the smartest people after all) but a process failure. Blameless postmortems avoided blame by ending with a process recommendation. I’ve worked at startups since, and I do think that creating an environment where people are comfortable taking risks is just a fundamentally hard culture problem, and you don’t even have to be big to have it (though it doesn’t help). I’ve seen 15-person startups where engineers wouldn’t fix something because “I’m just an engineer and this is really a product decision” while the PM says “we need to get the engineers together in a meeting and come to a consensus on this.” IMO Google’s hack of giving people space to take risks by letting them pass off failure as a process problem did sort of work, to their credit, until they got to where it sounds like they are now, where you can’t do anything without being accused of bypassing “good process,” which means going to endless lengths to get everyone’s opinions and placating everyone who disagrees with you. It seems like at Meta they solved the problem by regarding failure as good and just. At my current company, to do something risky, you have to write a design doc and have a comment period, and then you’re free to do whatever the doc proposes. Idk what other approaches there are—I’d pay to read a book about it, though.
  - intelVISA 2y ago
    Whilst I fully agree - you have to remember a line employee has limited incentive to take said risks as there's little to gain and a lot to lose.
- kazinator 2y ago
  I don't see how that manager expected to have a job by deciding not to cut power to submerged equipment, and have it flushed of the brine and cleaned, ASAP. It looks completely idiotic/incompetent from where I'm sitting.
  - varjag 2y ago
    There are plenty people who never changed a lightbulb in their life, and some of them manage other people.
  - close04 2y ago
    Different industries prioritize different things. I'm speculating but being the military they may consider that the enemy is planning an incursion exactly when some critical detection systems are expected to be down. So they'd rather kill some equipment while they mobilize a contingency measure. No idea what was in the manager's head or if the threat model makes sense but things aren't always super obvious and don't lend themselves to generalization.
    - Aeolun 2y ago
      Potentially, but if your intelligence is so terrible you have no idea of that attack beforehand, or your radar coverage so sloppy that you can’t take any of them offline without losing that coverage, you probably failed already.
- silisili 2y ago
  Because that's what companies incentivize for, sadly. The handbook or SOP somewhere probably does call for a risk assessment. So they have two choices. Choice A is to do what's obvious, cut power and get to work. That has two branches. One branch is saving the day and potentially millions of dollars. For what? A side mention at the next all hands, maybe. Branch 2 is something goes wrong, and the company says you didn't follow the handbook, and you're fired or worse, found liable. Choice B is following the handbook, knowing everything will be ruined. But, you keep your job, and maybe the handbook gets updated. Both choices provide no reward. Only one provides no risk.
  - Spivak 2y ago
    Ding ding ding! As an employee you're not there to do good work, you're there to what you're employer asks of you which is sometimes good work. When its your own electronics you can do whatever you want but as a rando employee you're not in a position to accept the risk.
inkyoto 2y ago
It is only impressive coming from the consumer grade electronics. Microchips, capacitors, PCB's and other stuff like manufactured for defence have much wider and higher tolerances. One won't find the cutting edge 2 nm processes (depending on the specific area of defence, 200-300 µm would still be cutting edge today for them), materials used are different, plus all sorts of chemical and physical hardening is employed. Most of the time, the chips can't use the consumer grade cutting edge designs and processes as they have to withstand not just abuse but also stronger EM fields (in radar equipment) or EM pulses (e.g. a nuclear explosion). After electronics have been manufactured, they proceed onto the QA phase where chips are literally baked in ovens for a day or a few, then retrieved and tested. They may get irradiated, soaked for periods of time etc. Most people would call it abuse – from a civilian perspective. The surviving ones are deemed to have passed the QA and are used to make the actual equipment.
- klyrs 2y ago
  > 200-300 µm would still be cutting edge today for them Sorry, I think you're off by a few orders of magnitude, do you mean 200-300nm?
  - inkyoto 2y ago
    No, I am not, although 200-300 µm is not a commonplace. Smaller processes result in chips either becoming very unstable or plainly kicking the bucket when exposed to strong electromagnetic fields and electromagnetic radiation. Radar equipment generates such strong EM radiation that is lethal to humans and to microelectronics no matter how well shielded the chips are. Then there are highly charged particles released during a nuclear explosion, so the electronics have to survive at least for some time at the battlefield. Chips produced for defence and military use some of the most conservative design and manufacturing processes, and even the analogue processes may still be used somewhere (I have not checked recently though).

It's quite impressive it kept working for that long, and even more amazing someone was capable of saying "no, let's not cut the power, there might be a problem if we do"

Aeolun 2y ago

Honestly, that doesn’t surprise me at all. This seems perfectly in line with “I don’t care what it is, or how well I perform it, as long as I have a job.”

hinkley 2y ago
Few things in my career have granted me more power faster than taking ownership of decisions. Some people are so terrified of having to say "I was wrong" that they either will do nothing, or just happily follow decisions that someone else makes. It's one thing when that's a line employee behaving that way, but there are managers so terrified of making a mistake that all they really are is scribes, tracking who said what and who decided what. These weasels don't lead. They don't contribute anything. They just have a quick answer when someone above them asks how this fuckup happened. Well that was Jameson. He decided to do X so clearly it's his fault things broke. I was in a room once with a guy who was so caught up in the idea of catching me in a gotcha that he had no idea what to do when I said, "yep, that's wrong, and this is how we fix it." His face went through the same roller coaster of emotions that happens when you drop your ice cream cone. I got to say most of what I needed to say before he even recovered. I think I was proudest of keeping a straight face during the entire conversation. Where has intellectual honesty gone that just being transparent verges on a superpower? I feel like I'm taking crazy pills.
- refulgentis 2y ago
  Taking ownership of decisions in this way breeds another problem: people love to police when they perceive other people taking authority that's not rightfully theirs. I had your view until 7 years at Google. Showing this to 80% of people there would garner tsktsking and quips about delusions of grandeur and seeming unbalanced. In performance reviews, you could count on this person saying you collaborate poorly and don't take others concerns into account, because they felt humiliated and retconned a reason why it's your fault. (To be fair, given your vivid understanding of their face at the time, it sounds like you understand they wouldn't feel well after you did this) In the higher rooms, I saw at least a couple times everyone living this worldview and it ending up in bickering and extreme insistence the other person was missing fundamentals while nothing got done for months: which in the more Machiavellian parlance, is still good if you're good, because most people won't do the work, just argue over the 20% of the problem they understand, and someone has to win eventually I keep editing this comment because I get the vibe it disagrees with you, but it doesn't, at all. If anything, your framing helps sharpen why it felt like an odd place, in a bad way, to me from day 30.
  - exmadscientist 2y ago
    I've known enough people at Google that I've gotten a decent view into the broken-ass culture there, from the outside looking in, and I'd say that's a pretty decent summary of some of their worst problems.
  - nine_k 2y ago
    One of the things that impressed me at my time at Google (2011-2015) was the culture of blameless post-mortems, and of general transparency of many (not all) things. Sadly, this culture must have deteriorated; my friends who are still at Google say that it's become literally the stuff of Dilbert comic strips :-(
    - refulgentis 2y ago
      Hah, somewhat perversely, I'm glad to hear that: it's exactly how I sum things up in a sentence in my head, then I think I'm being overdramatic / personalizing it, because I kinda grew up learning about tech via Dilbert, I was a big fan of Dogbert at 7 years old. But I'm not. I did Aug 2016-2023. Of course it's a complicated story, and you've heard a lot already I'm sure, but when you mention blameless postmortems, it remind me of my pet peeve: promo committees done within teams (basically 2 levels up, i.e. skip is doling out promo, with no recourse or real outside input) was a horrible, wretched idea. There was this odd obsession with saving time on perf, but they kinda just needed to tell people to chill and recognize that it really is important for management. With all promos being judged by people next to you, what we might call "being agreeable" became a necessity. I distinctly remember in 2018 being told if things were how they had been, that I should just do a blameless postmortem, but with things as they are, that was a quick way to end up on the bench for the rest of my career.
  - msteffen 2y ago
    Interestingly, I was at Google during I think the very end of its golden years (2011-2016). I loved it and when I heard about how sclerotic it had gotten was pretty surprised. Looking back, IMO the seeds of its downfall had already been planted by the time I was there, including the culture of blameless postmortems. There was this idea that any problem—any outage or whatever —reflected not stupidity on the part of the perpetrator (Google hires the smartest people after all) but a process failure. Blameless postmortems avoided blame by ending with a process recommendation. I’ve worked at startups since, and I do think that creating an environment where people are comfortable taking risks is just a fundamentally hard culture problem, and you don’t even have to be big to have it (though it doesn’t help). I’ve seen 15-person startups where engineers wouldn’t fix something because “I’m just an engineer and this is really a product decision” while the PM says “we need to get the engineers together in a meeting and come to a consensus on this.” IMO Google’s hack of giving people space to take risks by letting them pass off failure as a process problem did sort of work, to their credit, until they got to where it sounds like they are now, where you can’t do anything without being accused of bypassing “good process,” which means going to endless lengths to get everyone’s opinions and placating everyone who disagrees with you. It seems like at Meta they solved the problem by regarding failure as good and just. At my current company, to do something risky, you have to write a design doc and have a comment period, and then you’re free to do whatever the doc proposes. Idk what other approaches there are—I’d pay to read a book about it, though.
- intelVISA 2y ago
  Whilst I fully agree - you have to remember a line employee has limited incentive to take said risks as there's little to gain and a lot to lose.
kazinator 2y ago
I don't see how that manager expected to have a job by deciding not to cut power to submerged equipment, and have it flushed of the brine and cleaned, ASAP. It looks completely idiotic/incompetent from where I'm sitting.
- varjag 2y ago
  There are plenty people who never changed a lightbulb in their life, and some of them manage other people.
- close04 2y ago
  Different industries prioritize different things. I'm speculating but being the military they may consider that the enemy is planning an incursion exactly when some critical detection systems are expected to be down. So they'd rather kill some equipment while they mobilize a contingency measure. No idea what was in the manager's head or if the threat model makes sense but things aren't always super obvious and don't lend themselves to generalization.
  - Aeolun 2y ago
    Potentially, but if your intelligence is so terrible you have no idea of that attack beforehand, or your radar coverage so sloppy that you can’t take any of them offline without losing that coverage, you probably failed already.
silisili 2y ago
Because that's what companies incentivize for, sadly. The handbook or SOP somewhere probably does call for a risk assessment. So they have two choices. Choice A is to do what's obvious, cut power and get to work. That has two branches. One branch is saving the day and potentially millions of dollars. For what? A side mention at the next all hands, maybe. Branch 2 is something goes wrong, and the company says you didn't follow the handbook, and you're fired or worse, found liable. Choice B is following the handbook, knowing everything will be ruined. But, you keep your job, and maybe the handbook gets updated. Both choices provide no reward. Only one provides no risk.
- Spivak 2y ago
  Ding ding ding! As an employee you're not there to do good work, you're there to what you're employer asks of you which is sometimes good work. When its your own electronics you can do whatever you want but as a rando employee you're not in a position to accept the risk.

inkyoto 2y ago

It is only impressive coming from the consumer grade electronics. Microchips, capacitors, PCB's and other stuff like manufactured for defence have much wider and higher tolerances. One won't find the cutting edge 2 nm processes (depending on the specific area of defence, 200-300 µm would still be cutting edge today for them), materials used are different, plus all sorts of chemical and physical hardening is employed. Most of the time, the chips can't use the consumer grade cutting edge designs and processes as they have to withstand not just abuse but also stronger EM fields (in radar equipment) or EM pulses (e.g. a nuclear explosion). After electronics have been manufactured, they proceed onto the QA phase where chips are literally baked in ovens for a day or a few, then retrieved and tested. They may get irradiated, soaked for periods of time etc. Most people would call it abuse – from a civilian perspective. The surviving ones are deemed to have passed the QA and are used to make the actual equipment.

klyrs 2y ago
> 200-300 µm would still be cutting edge today for them Sorry, I think you're off by a few orders of magnitude, do you mean 200-300nm?
- inkyoto 2y ago
  No, I am not, although 200-300 µm is not a commonplace. Smaller processes result in chips either becoming very unstable or plainly kicking the bucket when exposed to strong electromagnetic fields and electromagnetic radiation. Radar equipment generates such strong EM radiation that is lethal to humans and to microelectronics no matter how well shielded the chips are. Then there are highly charged particles released during a nuclear explosion, so the electronics have to survive at least for some time at the battlefield. Chips produced for defence and military use some of the most conservative design and manufacturing processes, and even the analogue processes may still be used somewhere (I have not checked recently though).