4 ms·

I remember talking to my ex's dad about his job, which involved planning refuels of a large nuclear-powered generation station in the Lower Midwest. The words

I remember talking to my ex's dad about his job, which involved planning refuels of a large nuclear-powered generation station in the Lower Midwest. The words "it's a miracle it works at all" routinely popped up in those conversations, which is... something you don't want to hear about any sort of power generation - especially not nuclear - but it's true. It's a system basically built to produce "common accidents". It's amazing that it doesn't on a regular basis.

tpmoney 2y ago
> The words "it's a miracle it works at all" routinely popped up in those conversations, which is... something you don't want to hear about any sort of power generation - especially not nuclear - but it's true. Funny thing is, those are the exact words I use when talking to people about networking. And realistically anytime I dig deep into the underlying details of any big enough system I walk away with that impression. At scale, I think any system is less “controlled and planned precision” and more “harnessed chaos with a lot of resiliency to the unpredictability of that chaos”
- lisper 2y ago
  > those are the exact words I use when talking to people about networking Or the U.S. financial system. Or civilization in general.
  - psunavy03 2y ago
    It ultimately comes down to shared norms, shared expectations, and trust.
    - _kb 2y ago
      Which is why the long tail impact of current times is frankly terrifying.
      - chairmansteve 2y ago
        Yes. The preppers are starting to look sane.
        organsnyder 2y ago
        The preppers can only buy themselves a small amount of time, though—no more than a year or two. Eventually, their stockpiled supplies will run out, or some piece of equipment will need a replacement part. I'd much rather focus on "prepping" by building social resiliency, instead. The local community I'm plugged into is much stronger together than anything I could possibly build individually.
    - somenameforme 2y ago
      A bit of a tangent, but I don't think this is it. There are plenty of species with plenty of shared norms, expectations, and trust - but no civilization. And, vice versa, many of the greatest societies have been riddled with completely incompatible worldviews yet created amazing civilizations. Consider that Sparta and Athens were separated by only 130 miles, yet couldn't possibly have been further apart! The reason people work together is fundamentally the same reason you go to work - self interest. You're rarely there because you genuinely believe in the mission or product - mostly you just want to get paid and then go do your own thing. And that's basically the gears of society in a nutshell. But you need the intelligence to understand the bigger picture of things. For instance Chimps have intricate little societies that at their peak have reached upwards of 200 chimps. They even wage war over them and in efforts to expand them or control their territory. This [1] war was something that revolutionized our understanding of primates behaviors, which had been excessively idealized beforehand. But they lack the intelligence to understand how to bring their little societies up in scale. They understand full well how to kill the other tribe and "integrate" their females, but they never think to e.g. enslave the males, let alone higher order forms of expansion with vassalage, negotiated treaties, and so on. All of which over time trend towards where we are today, where it turns out giving somebody a little slice of your pie and letting him otherwise roam free is way more effective than just trying to dominate him. [1] - https://en.wikipedia.org/wiki/Gombe_Chimpanzee_War https://en.wikipedia.org/wiki/Gombe_Chimpanzee_War
      - namaria 2y ago
        > There are plenty of species with plenty of shared norms, expectations, and trust Citation needed on that one. > Consider that Sparta and Athens were separated by only 130 miles, yet couldn't possibly have been further apart! They spoke the same language, shared the same literature, practiced the same religion, had a long history of diplomatic ties. When the Persians razed Athens, they took refuge with the Spartans. > For instance Chimps have intricate little societies that at their peak have reached upwards of 200 chimps. Again, I don't think this claim stands to evidence. The so called chimp war you mention is about a group of about a dozen and a huge fight that broke out among them. That doesn't support the idea that they are capable of 200-strong 'intricate' groupings.
  - BrandoElFollito 2y ago
    For me it is the financial system as a whole. I am an ex-scientist and an engineer and had a look at the books of my son who studies finance in the best finance school in the world (I am saying this to highlight that he will be one of the perpetrators, possibly with influence, of this mess) The things in there are crazy. There are whole blocks that are obvious but made to sound complicated. I spent some time on a graph just to realize that they ultimately talk about solving a set of two linear equations (midfle school level). Some pieces were not comprehensible because they did not make sense. And then bam! A random differential equation and explanation as it was the answer to the universe. With an incorrect interpretation. And then there are statistics that would make "sociology science" blush. Yes, they are so bad that even the, ahem, experts who do stats in sociology would be ashamed (no hate for sociology, everyone needs to eat, it is just that I was several times reviewer of thesises there and I have trauma afterwards). The fact that finance works is because we have some kind of magical "local minimum of finance energy" from which the Trumps of this world somehow did not maybe to break from (fingers crossed) by disrupting the world too much.
- r3trohack3r 2y ago
  This is one of the key insights in my early SRE career that changed how I viewed software engineering at scale. Components aren’t reliable. The whole thing might be duct tape and popsicle sticks. But the trick for SRE work is to create stability from unreliable components by isolating and routing around failures. It’s part of what made chaos engineering so effective. From randomly slowing down disk/network speed to unplugging server racks to making entire datacenters go dark - you intentionally introduce all sorts of crazy failure modes to intentionally break things and make sure the system remains metastable.
  - ForOldHack 2y ago
    Message on a mug: "if carpenters built houses the way programmers write software, a woodpecker could destroy civilization." The syncronasation of a power grid ... Wow.
    - vkou 2y ago
      If houses could be torn down and recreated with the press of a key, we probably wouldn't have a housing shortage.
      - alpaca128 2y ago
        We would instead have HaaS, with monthly subscriptions for a license to use the house. Which can be randomly revoked at any moment if the company doesn't feel like supporting it is profitable enough, or if an AI thinks your electricity usage is suspicious and permabans you from using a home in the entire town.
        organsnyder 2y ago
        Isn't that called "renting"?
        vkou 2y ago
        So, in short, being a tenant in a low-regulatory environment. Tell me more about this paradise.
- gerdesj 2y ago
  "Funny thing is, those are the exact words I use when talking to people about networking" Computer networking is not the same. Our networks will not explode. I will grant you that they can be shite if not designed properly but they end up running slowly or not at all, but it will not combust nor explode. If you get the basics right for ethernet then it works rather well as a massive network. You could describe it as an internetwork. Basically, keep your layer 1 to around 200 odd maximum devices per VLAN - that works fine for IPv4. You might have to tune MAC tables for IPv6 for obvious reasons. Your fancier switches will have some funky memory for tables of one address to other address translation eg MAC to IP n VLAN and that. That memory will be shared with other databases too, perhaps iSCSI, so you have to decide how to manage that lot.
  - lillecarl 2y ago
    You tried to nerdsnipe someone without mentioning L2 is effectively dead within datacenters since VXLAN became hardware accelerated in both Broadcom and "NVIDIA"(Mellanox) gear. And for those that don't need/care about L2 they don't even bother and run L3 all the way. EVPN uses BGP to advertise MAC addresses in VXLAN networks which solves looping without magic packets, scales better and is easier to introspect. And we didn't even get into the provider side which has been using MPLS for decades. A problem with high bandwidth networking over fiber is that since light refracts within the fiber some light will take a longer path than other, if the widow is too short and you have too much scattering you will drop packets. So hopefully someone doesn't bend your 100G fiber too much, if that isn't finicky idk what is, DAC cables with twinax solve it short-range for cheaper however.
- chasd00 2y ago
  I did a lot of work for a major airline earlier in my career and came away with same impression. I just couldn’t see how they kept planes in the air based on my experiences through out the organization. I think in a big enough org the sheer momentum keeps things moving despite all the fires happening constantly.
0xbadcafebee 2y ago
If people knew how crappy, insecure, and unreliable nuclear computer systems were, there'd be a lot more existential dread about cyber security
- mikewarot 2y ago
  Almost all computers are insecure, not just the systems in nuclear stations. Most operating systems are based on ambient authority, which is just a disaster waiting to happen.
  - theteapot 2y ago
    What's the alternative?
- amirhirsch 2y ago
  I built control computers for nuclear reactors. Those machines are not connected to a network and are guarded by multiple stages of men with automatic machine guns. It was designed to flawlessly run 3x boards each with triple-modular-redundant processors in FPGA fabric all nine processors instruction-synced with ECC down to the Registers (including cycling the three areas of programmable fabric on the FPGAs). They cycle and test each board every month. What’s your source?
  - photonthug 2y ago
    Well, the news says that doge randos are potentially exfiltrating the details of systems like that as well as financial details of many Americans, including those who hold machine guns and probably suffer from substandard pay and bad economic prospects/job security as much as anyone else does. Perhaps the safest assumption is that system reliability ultimately depends on quite a lot of factors that are not purely about careful engineering.
    - raverbashing 2y ago
      Nothing like a special commando of people doing your more malicious biddings while also being expendable
bradgessler 2y ago
My definition of technology is, “something built by humans that barely works”
- throwaway2037 2y ago
  Modern aircraft? Those are excellent and work well. I am thinking of a B787 and A350. More: How about medical implants, like a heart pacemaker?
  - lazide 2y ago
    The regulations around parts sourcing, required maintenance, and training has more to do with how well/safe modern aviation is than anything else. If those aren’t done properly, all sorts of weird things start happening. Pretty much the only reason aerospace safety records aren’t worse in third world countries is because of how obviously bad the consequences are quickly - and even then….

tpmoney 2y ago

> The words "it's a miracle it works at all" routinely popped up in those conversations, which is... something you don't want to hear about any sort of power generation - especially not nuclear - but it's true. Funny thing is, those are the exact words I use when talking to people about networking. And realistically anytime I dig deep into the underlying details of any big enough system I walk away with that impression. At scale, I think any system is less “controlled and planned precision” and more “harnessed chaos with a lot of resiliency to the unpredictability of that chaos”

lisper 2y ago
> those are the exact words I use when talking to people about networking Or the U.S. financial system. Or civilization in general.
- psunavy03 2y ago
  It ultimately comes down to shared norms, shared expectations, and trust.
  - _kb 2y ago
    Which is why the long tail impact of current times is frankly terrifying.
    - chairmansteve 2y ago
      Yes. The preppers are starting to look sane.
      - organsnyder 2y ago
        The preppers can only buy themselves a small amount of time, though—no more than a year or two. Eventually, their stockpiled supplies will run out, or some piece of equipment will need a replacement part. I'd much rather focus on "prepping" by building social resiliency, instead. The local community I'm plugged into is much stronger together than anything I could possibly build individually.
  - somenameforme 2y ago
    A bit of a tangent, but I don't think this is it. There are plenty of species with plenty of shared norms, expectations, and trust - but no civilization. And, vice versa, many of the greatest societies have been riddled with completely incompatible worldviews yet created amazing civilizations. Consider that Sparta and Athens were separated by only 130 miles, yet couldn't possibly have been further apart! The reason people work together is fundamentally the same reason you go to work - self interest. You're rarely there because you genuinely believe in the mission or product - mostly you just want to get paid and then go do your own thing. And that's basically the gears of society in a nutshell. But you need the intelligence to understand the bigger picture of things. For instance Chimps have intricate little societies that at their peak have reached upwards of 200 chimps. They even wage war over them and in efforts to expand them or control their territory. This [1] war was something that revolutionized our understanding of primates behaviors, which had been excessively idealized beforehand. But they lack the intelligence to understand how to bring their little societies up in scale. They understand full well how to kill the other tribe and "integrate" their females, but they never think to e.g. enslave the males, let alone higher order forms of expansion with vassalage, negotiated treaties, and so on. All of which over time trend towards where we are today, where it turns out giving somebody a little slice of your pie and letting him otherwise roam free is way more effective than just trying to dominate him. [1] - https://en.wikipedia.org/wiki/Gombe_Chimpanzee_War https://en.wikipedia.org/wiki/Gombe_Chimpanzee_War
    - namaria 2y ago
      > There are plenty of species with plenty of shared norms, expectations, and trust Citation needed on that one. > Consider that Sparta and Athens were separated by only 130 miles, yet couldn't possibly have been further apart! They spoke the same language, shared the same literature, practiced the same religion, had a long history of diplomatic ties. When the Persians razed Athens, they took refuge with the Spartans. > For instance Chimps have intricate little societies that at their peak have reached upwards of 200 chimps. Again, I don't think this claim stands to evidence. The so called chimp war you mention is about a group of about a dozen and a huge fight that broke out among them. That doesn't support the idea that they are capable of 200-strong 'intricate' groupings.
- BrandoElFollito 2y ago
  For me it is the financial system as a whole. I am an ex-scientist and an engineer and had a look at the books of my son who studies finance in the best finance school in the world (I am saying this to highlight that he will be one of the perpetrators, possibly with influence, of this mess) The things in there are crazy. There are whole blocks that are obvious but made to sound complicated. I spent some time on a graph just to realize that they ultimately talk about solving a set of two linear equations (midfle school level). Some pieces were not comprehensible because they did not make sense. And then bam! A random differential equation and explanation as it was the answer to the universe. With an incorrect interpretation. And then there are statistics that would make "sociology science" blush. Yes, they are so bad that even the, ahem, experts who do stats in sociology would be ashamed (no hate for sociology, everyone needs to eat, it is just that I was several times reviewer of thesises there and I have trauma afterwards). The fact that finance works is because we have some kind of magical "local minimum of finance energy" from which the Trumps of this world somehow did not maybe to break from (fingers crossed) by disrupting the world too much.
r3trohack3r 2y ago
This is one of the key insights in my early SRE career that changed how I viewed software engineering at scale. Components aren’t reliable. The whole thing might be duct tape and popsicle sticks. But the trick for SRE work is to create stability from unreliable components by isolating and routing around failures. It’s part of what made chaos engineering so effective. From randomly slowing down disk/network speed to unplugging server racks to making entire datacenters go dark - you intentionally introduce all sorts of crazy failure modes to intentionally break things and make sure the system remains metastable.
- ForOldHack 2y ago
  Message on a mug: "if carpenters built houses the way programmers write software, a woodpecker could destroy civilization." The syncronasation of a power grid ... Wow.
  - vkou 2y ago
    If houses could be torn down and recreated with the press of a key, we probably wouldn't have a housing shortage.
    - alpaca128 2y ago
      We would instead have HaaS, with monthly subscriptions for a license to use the house. Which can be randomly revoked at any moment if the company doesn't feel like supporting it is profitable enough, or if an AI thinks your electricity usage is suspicious and permabans you from using a home in the entire town.
      - organsnyder 2y ago
        Isn't that called "renting"?
      - vkou 2y ago
        So, in short, being a tenant in a low-regulatory environment. Tell me more about this paradise.
gerdesj 2y ago
"Funny thing is, those are the exact words I use when talking to people about networking" Computer networking is not the same. Our networks will not explode. I will grant you that they can be shite if not designed properly but they end up running slowly or not at all, but it will not combust nor explode. If you get the basics right for ethernet then it works rather well as a massive network. You could describe it as an internetwork. Basically, keep your layer 1 to around 200 odd maximum devices per VLAN - that works fine for IPv4. You might have to tune MAC tables for IPv6 for obvious reasons. Your fancier switches will have some funky memory for tables of one address to other address translation eg MAC to IP n VLAN and that. That memory will be shared with other databases too, perhaps iSCSI, so you have to decide how to manage that lot.
- lillecarl 2y ago
  You tried to nerdsnipe someone without mentioning L2 is effectively dead within datacenters since VXLAN became hardware accelerated in both Broadcom and "NVIDIA"(Mellanox) gear. And for those that don't need/care about L2 they don't even bother and run L3 all the way. EVPN uses BGP to advertise MAC addresses in VXLAN networks which solves looping without magic packets, scales better and is easier to introspect. And we didn't even get into the provider side which has been using MPLS for decades. A problem with high bandwidth networking over fiber is that since light refracts within the fiber some light will take a longer path than other, if the widow is too short and you have too much scattering you will drop packets. So hopefully someone doesn't bend your 100G fiber too much, if that isn't finicky idk what is, DAC cables with twinax solve it short-range for cheaper however.
chasd00 2y ago
I did a lot of work for a major airline earlier in my career and came away with same impression. I just couldn’t see how they kept planes in the air based on my experiences through out the organization. I think in a big enough org the sheer momentum keeps things moving despite all the fires happening constantly.

0xbadcafebee 2y ago

If people knew how crappy, insecure, and unreliable nuclear computer systems were, there'd be a lot more existential dread about cyber security

mikewarot 2y ago
Almost all computers are insecure, not just the systems in nuclear stations. Most operating systems are based on ambient authority, which is just a disaster waiting to happen.
- theteapot 2y ago
  What's the alternative?
amirhirsch 2y ago
I built control computers for nuclear reactors. Those machines are not connected to a network and are guarded by multiple stages of men with automatic machine guns. It was designed to flawlessly run 3x boards each with triple-modular-redundant processors in FPGA fabric all nine processors instruction-synced with ECC down to the Registers (including cycling the three areas of programmable fabric on the FPGAs). They cycle and test each board every month. What’s your source?
- photonthug 2y ago
  Well, the news says that doge randos are potentially exfiltrating the details of systems like that as well as financial details of many Americans, including those who hold machine guns and probably suffer from substandard pay and bad economic prospects/job security as much as anyone else does. Perhaps the safest assumption is that system reliability ultimately depends on quite a lot of factors that are not purely about careful engineering.
  - raverbashing 2y ago
    Nothing like a special commando of people doing your more malicious biddings while also being expendable

bradgessler 2y ago

My definition of technology is, “something built by humans that barely works”

throwaway2037 2y ago
Modern aircraft? Those are excellent and work well. I am thinking of a B787 and A350. More: How about medical implants, like a heart pacemaker?
- lazide 2y ago
  The regulations around parts sourcing, required maintenance, and training has more to do with how well/safe modern aviation is than anything else. If those aren’t done properly, all sorts of weird things start happening. Pretty much the only reason aerospace safety records aren’t worse in third world countries is because of how obviously bad the consequences are quickly - and even then….