5 ms·

I was once involved in a similar (unintentional) experiment with a multi-billion dollar defence radar. A several metres deep underground bunker, containing sev

• rcxdude 2y ago
  It's quite impressive it kept working for that long, and even more amazing someone was capable of saying "no, let's not cut the power, there might be a problem if we do"
  • Aeolun 2y ago
    Honestly, that doesn’t surprise me at all. This seems perfectly in line with “I don’t care what it is, or how well I perform it, as long as I have a job.”
    • hinkley 2y ago
      Few things in my career have granted me more power faster than taking ownership of decisions. Some people are so terrified of having to say "I was wrong" that they either will do nothing, or just happily follow decisions that someone else makes. It's one thing when that's a line employee behaving that way, but there are managers so terrified of making a mistake that all they really are is scribes, tracking who said what and who decided what. These weasels don't lead. They don't contribute anything. They just have a quick answer when someone above them asks how this fuckup happened. Well that was Jameson. He decided to do X so clearly it's his fault things broke. I was in a room once with a guy who was so caught up in the idea of catching me in a gotcha that he had no idea what to do when I said, "yep, that's wrong, and this is how we fix it." His face went through the same roller coaster of emotions that happens when you drop your ice cream cone. I got to say most of what I needed to say before he even recovered. I think I was proudest of keeping a straight face during the entire conversation. Where has intellectual honesty gone that just being transparent verges on a superpower? I feel like I'm taking crazy pills.
      • refulgentis 2y ago
        Taking ownership of decisions in this way breeds another problem: people love to police when they perceive other people taking authority that's not rightfully theirs. I had your view until 7 years at Google. Showing this to 80% of people there would garner tsktsking and quips about delusions of grandeur and seeming unbalanced. In performance reviews, you could count on this person saying you collaborate poorly and don't take others concerns into account, because they felt humiliated and retconned a reason why it's your fault. (To be fair, given your vivid understanding of their face at the time, it sounds like you understand they wouldn't feel well after you did this) In the higher rooms, I saw at least a couple times everyone living this worldview and it ending up in bickering and extreme insistence the other person was missing fundamentals while nothing got done for months: which in the more Machiavellian parlance, is still good if you're good, because most people won't do the work, just argue over the 20% of the problem they understand, and someone has to win eventually I keep editing this comment because I get the vibe it disagrees with you, but it doesn't, at all. If anything, your framing helps sharpen why it felt like an odd place, in a bad way, to me from day 30.
        • exmadscientist 2y ago
          I've known enough people at Google that I've gotten a decent view into the broken-ass culture there, from the outside looking in, and I'd say that's a pretty decent summary of some of their worst problems.
        • nine_k 2y ago
          One of the things that impressed me at my time at Google (2011-2015) was the culture of blameless post-mortems, and of general transparency of many (not all) things. Sadly, this culture must have deteriorated; my friends who are still at Google say that it's become literally the stuff of Dilbert comic strips :-(
          • refulgentis 2y ago
            Hah, somewhat perversely, I'm glad to hear that: it's exactly how I sum things up in a sentence in my head, then I think I'm being overdramatic / personalizing it, because I kinda grew up learning about tech via Dilbert, I was a big fan of Dogbert at 7 years old. But I'm not. I did Aug 2016-2023. Of course it's a complicated story, and you've heard a lot already I'm sure, but when you mention blameless postmortems, it remind me of my pet peeve: promo committees done within teams (basically 2 levels up, i.e. skip is doling out promo, with no recourse or real outside input) was a horrible, wretched idea. There was this odd obsession with saving time on perf, but they kinda just needed to tell people to chill and recognize that it really is important for management. With all promos being judged by people next to you, what we might call "being agreeable" became a necessity. I distinctly remember in 2018 being told if things were how they had been, that I should just do a blameless postmortem, but with things as they are, that was a quick way to end up on the bench for the rest of my career.
    • kazinator 2y ago
      I don't see how that manager expected to have a job by deciding not to cut power to submerged equipment, and have it flushed of the brine and cleaned, ASAP. It looks completely idiotic/incompetent from where I'm sitting.
      • varjag 2y ago
        There are plenty people who never changed a lightbulb in their life, and some of them manage other people.
      • close04 2y ago
        Different industries prioritize different things. I'm speculating but being the military they may consider that the enemy is planning an incursion exactly when some critical detection systems are expected to be down. So they'd rather kill some equipment while they mobilize a contingency measure. No idea what was in the manager's head or if the threat model makes sense but things aren't always super obvious and don't lend themselves to generalization.
        • Aeolun 2y ago
          Potentially, but if your intelligence is so terrible you have no idea of that attack beforehand, or your radar coverage so sloppy that you can’t take any of them offline without losing that coverage, you probably failed already.
    • silisili 2y ago
      Because that's what companies incentivize for, sadly. The handbook or SOP somewhere probably does call for a risk assessment. So they have two choices. Choice A is to do what's obvious, cut power and get to work. That has two branches. One branch is saving the day and potentially millions of dollars. For what? A side mention at the next all hands, maybe. Branch 2 is something goes wrong, and the company says you didn't follow the handbook, and you're fired or worse, found liable. Choice B is following the handbook, knowing everything will be ruined. But, you keep your job, and maybe the handbook gets updated. Both choices provide no reward. Only one provides no risk.
      • Spivak 2y ago
        Ding ding ding! As an employee you're not there to do good work, you're there to what you're employer asks of you which is sometimes good work. When its your own electronics you can do whatever you want but as a rando employee you're not in a position to accept the risk.
  • inkyoto 2y ago
    It is only impressive coming from the consumer grade electronics. Microchips, capacitors, PCB's and other stuff like manufactured for defence have much wider and higher tolerances. One won't find the cutting edge 2 nm processes (depending on the specific area of defence, 200-300 µm would still be cutting edge today for them), materials used are different, plus all sorts of chemical and physical hardening is employed. Most of the time, the chips can't use the consumer grade cutting edge designs and processes as they have to withstand not just abuse but also stronger EM fields (in radar equipment) or EM pulses (e.g. a nuclear explosion). After electronics have been manufactured, they proceed onto the QA phase where chips are literally baked in ovens for a day or a few, then retrieved and tested. They may get irradiated, soaked for periods of time etc. Most people would call it abuse – from a civilian perspective. The surviving ones are deemed to have passed the QA and are used to make the actual equipment.
    • klyrs 2y ago
      > 200-300 µm would still be cutting edge today for them Sorry, I think you're off by a few orders of magnitude, do you mean 200-300nm?
      • inkyoto 2y ago
        No, I am not, although 200-300 µm is not a commonplace. Smaller processes result in chips either becoming very unstable or plainly kicking the bucket when exposed to strong electromagnetic fields and electromagnetic radiation. Radar equipment generates such strong EM radiation that is lethal to humans and to microelectronics no matter how well shielded the chips are. Then there are highly charged particles released during a nuclear explosion, so the electronics have to survive at least for some time at the battlefield. Chips produced for defence and military use some of the most conservative design and manufacturing processes, and even the analogue processes may still be used somewhere (I have not checked recently though).
• deleted 2y ago
  [deleted]
• BikiniPrince 2y ago
  The navy used to throw the boards in the cleaner and repairs the failures. Now we have better cleansing solutions, but damn if that isn’t nightmare fuel. I would have pumped and dehumidified the room until parts could be pulled/repaired. These days they just scrap everything. It’s terrible.
• m463 2y ago
  I wonder although wasteful, isn't this better than accelerated failures taking the defense radar offline at random times in the future? Reminds me of a friend of mine who somehow blew a shift in his car and did something bad to his engine. Instead of pulling over and stopping, he just drove it home and ended up destroying the engine. So he got a whole new engine instead of having insurance pay to dismantle the engine and replace a few valves. It might even have been a similar price, but a much more robust solution.
  • nine_k 2y ago
    But the action that has actually been taken just guaranteed the prompt destruction of the radar equipment, and a prolonged outage to replace it. Unless an attack was underway that the radar was tracking, the right course of action was to power it off immediately, and start cleaning it. After the cleaning, it would go online promptly, and then parts of it could be assessed for damage, taken offline when the replacement is ready, and replaced, likely without even stopping the operation. This kind of paralysis and fear of making decisions, especially when time is of essence, would be exactly what I'd like to instill in my enemies.
    • sakjur 2y ago
      It would explain a surprising amount of events at least here in Sweden if BCG and McKinsey were part of a conspiracy to weaken the crisis preparedness of states worldwide.
  • OptionOfT 2y ago
    If your friend did a money shift the damage happened at the shift moment and driving home didn't change the outcome.
    • jpc0 2y ago
      I reread the comment a few times and I think he agrees with you. His friend got the better outcome by not stopping immediately is what he was implying.
• fch42 2y ago
  I have a similar experience from waaaaay back (now near 30 years). We had a water leak in the basement of the institute of physics. Nothing terribly major, ran for a few hours shorted power downstairs eventually (that's how it was found). Once fixed, pumped out, all that remained was a musty smell for a few days. Except four days later network connection failed in the building. Tracing the cabling eventually lead us to the basement; some under-stairs / slightly below floor level comms cabinet, empty other than the building's outbound router. 48V Telco, no mains connect, had sat under"water" (well some rather murky stuff) for almost a week and done well. Once we got it washed (with clear water) and dried, it continued fine (till replacement a few weeks after). We were a little amazed, to be honest, physicists or no. Brave device, that.
• appplication 2y ago
  I also worked at a very large multi-billion dollar defense radar, once upon a time. Though by your description maybe a different one.