4 ms·

It’s great that nuclear power safety events are so well managed that nearly 100% of events go completely unnoticed by the public. Unironically this is the kind

It’s great that nuclear power safety events are so well managed that nearly 100% of events go completely unnoticed by the public. Unironically this is the kind of headline I’m expecting in 2040 about one or more major AGI systems running an electrical grid, factory or similar.

stefantalpalaru 2mo ago
[dead]
bigyabai 2mo ago
I doubt AGI will be used for life-or-death systems like that if it resembles anything like the LLMs of today. Systems at that scale are about risk management, and everything we know about AI suggests that it cannot mitigate risk at all. All AI is seemingly a net-positive risk factor. Really, the "big AI story" will be when the next Snowden defects from the NRO and tells us what SENTIENT is doing. Until then it's just going to be a bunch of pie-in-the-sky wishcasting about what we want the future to look like. It's getting a little tiring hearing people insist that abundance is around the corner while surveillance is here-and-now.
- mlyle 2mo ago
  The way I read it was that he expects large industrial systems with AI to have safeguards to "fail closed." Which: it seems like we're already getting to the point of capability vs. unexpected behavior that having the ability to "trip" a complex system may be an important design criterion. I mean, there's plenty of harm these systems can do besides trying to turn us all into paperclips.
  - AndrewKemendo 2mo ago
    Correct interpretation
    - bigyabai 2mo ago
      So, which do we choose? A) Let AI override safety mechanisms (increases uptime while endangering humans) B) Let AI impose additional safeties (decreases uptime with questionable safety uplift) If you intend to go with option B) then there's basically zero benefit to investing in that kind of automation. AGI will be fallible, and even with all the safeties and safeguards it's hard to see what tangible benefit AGI would add to the system. It'll be a hard sell for critical infrastructure of any kind. I understand that it's hard to get a man to understand something when his AGI Society executive credentials depend on it, but it feels glaringly obvious why AI isn't masterminding these sorts of systems. Even accounting for unprecedented advancement in AI, it's hard to see what AI could contribute to managing these things.
      - mlyle 2mo ago
        > It'll be a hard sell for critical infrastructure of any kind Remember, critical infrastructure attached SCADA systems to the internet and has had trouble removing them all despite 30 years of warnings and audits. There's definite advantages to having a LLM in the loop. The best we can hope for is that it's architected in a way that it fails safe and damages are limited, so we get some of the advantages but less of the downside.
        bigyabai 2mo ago
        That's a configuration issue, not people preferring a bad idea. SCADA itself is not a hard sell because it's ostensibly deterministic. If AI cannot provide the same baseline promises while only bring right a fraction of the time, then it's not the right solution. I don't want my car's AC to ask Anthropic what the right temperature is; it's a waste of tokens that isn't optimizing jack shit 99.99% of the time. If a system isn't failing currently, then it's probably in the same boat as car ACs. Shoehorning AI into it won't solve anything.
      - AndrewKemendo 2mo ago
        There are more than two options
- mmooss 2mo ago
  That assumes people mistrust LLM output, which is a perspective inside the HN bubble. Outside, almost everyone I know - including highly educated people - takes even Google's search LLM output as fact, without a second thought, and even regarding serious issues (e.g., medical problems). Nobody raises a doubt or an eyebrow. A recent article says members of US Congress use it extensively for serious work. Lawyers use it to file cases. Doctors use it to summarize essential medical information. I even hear that software developers use it to create high-risk, high-value code. In the public people think computers are superior to humans - more authoritative, objective, perfectly logical and error-proof. SV pushes that narrative, for example saying computer-driven cars are safer because computers don't have human flaws (that old sales technique - compare only the competition's unique flaws and don't mention your own; computers have their own flaws that humans lack). And of course, that one is popular on HN. (IMHO the way to educate the public is to spread the narrative that every computer is created (designed, programmed) by normal, flawed, biased, etc. humans, and controlled by other flawed humans for their own agenda. And that computers have their own flaws.)
  - AndrewKemendo 2mo ago
    The fact that you conflate LLMs and AI means you’re unqualified to be part of the discussion
  - AtlasBarfed 2mo ago
    The Elon buttlicker types praise llms as the new knowledge gods. The general public distrusts them cuz they know that they're coming for their jobs. But they will use them out of convenience
- MBCook 2mo ago
  I’m absolutely willing to believe there are some people that stupid. I doubt anyone who works on reactors is that stupid. But some idiot somewhere is going to put some industrial stuff under AI control pretty soon and murder a few people at minimum.
- jmalicki 2mo ago
  AI labs are already working on getting LLMs to control nuclear reactors, and hiring experienced reactor operators to help train them: https://www.reddit.com/r/nuclear/comments/1qnzxvg/this_job_posting_is_insane/#lightbox https://www.reddit.com/r/nuclear/comments/1qnzxvg/this_job_p...
iwontberude 2mo ago
[dead]
delichon 2mo ago
This was a power safety event in the sense that safety was accidentally added. Like oops, we accidentally put the pin back in the grenade.
- AndrewKemendo 2mo ago
  Which indicates that there’s so much understanding of how it works and the power that it uses that compensatory systems have been created to increase its reliability what precisely is your point of describing how quality production engineering for infrastructure works?
dist-epoch 2mo ago
> nearly 100% of events go completely unnoticed by the public "Nearly" is doing a lot of heavy lifting there. Those "nearly" 0% of events which DO go noticed by the public, like you know, Fukushima, tend to be world news for weeks at a time.

stefantalpalaru 2mo ago

[dead]

bigyabai 2mo ago

I doubt AGI will be used for life-or-death systems like that if it resembles anything like the LLMs of today. Systems at that scale are about risk management, and everything we know about AI suggests that it cannot mitigate risk at all. All AI is seemingly a net-positive risk factor. Really, the "big AI story" will be when the next Snowden defects from the NRO and tells us what SENTIENT is doing. Until then it's just going to be a bunch of pie-in-the-sky wishcasting about what we want the future to look like. It's getting a little tiring hearing people insist that abundance is around the corner while surveillance is here-and-now.

mlyle 2mo ago
The way I read it was that he expects large industrial systems with AI to have safeguards to "fail closed." Which: it seems like we're already getting to the point of capability vs. unexpected behavior that having the ability to "trip" a complex system may be an important design criterion. I mean, there's plenty of harm these systems can do besides trying to turn us all into paperclips.
- AndrewKemendo 2mo ago
  Correct interpretation
  - bigyabai 2mo ago
    So, which do we choose? A) Let AI override safety mechanisms (increases uptime while endangering humans) B) Let AI impose additional safeties (decreases uptime with questionable safety uplift) If you intend to go with option B) then there's basically zero benefit to investing in that kind of automation. AGI will be fallible, and even with all the safeties and safeguards it's hard to see what tangible benefit AGI would add to the system. It'll be a hard sell for critical infrastructure of any kind. I understand that it's hard to get a man to understand something when his AGI Society executive credentials depend on it, but it feels glaringly obvious why AI isn't masterminding these sorts of systems. Even accounting for unprecedented advancement in AI, it's hard to see what AI could contribute to managing these things.
    - mlyle 2mo ago
      > It'll be a hard sell for critical infrastructure of any kind Remember, critical infrastructure attached SCADA systems to the internet and has had trouble removing them all despite 30 years of warnings and audits. There's definite advantages to having a LLM in the loop. The best we can hope for is that it's architected in a way that it fails safe and damages are limited, so we get some of the advantages but less of the downside.
      - bigyabai 2mo ago
        That's a configuration issue, not people preferring a bad idea. SCADA itself is not a hard sell because it's ostensibly deterministic. If AI cannot provide the same baseline promises while only bring right a fraction of the time, then it's not the right solution. I don't want my car's AC to ask Anthropic what the right temperature is; it's a waste of tokens that isn't optimizing jack shit 99.99% of the time. If a system isn't failing currently, then it's probably in the same boat as car ACs. Shoehorning AI into it won't solve anything.
    - AndrewKemendo 2mo ago
      There are more than two options
mmooss 2mo ago
That assumes people mistrust LLM output, which is a perspective inside the HN bubble. Outside, almost everyone I know - including highly educated people - takes even Google's search LLM output as fact, without a second thought, and even regarding serious issues (e.g., medical problems). Nobody raises a doubt or an eyebrow. A recent article says members of US Congress use it extensively for serious work. Lawyers use it to file cases. Doctors use it to summarize essential medical information. I even hear that software developers use it to create high-risk, high-value code. In the public people think computers are superior to humans - more authoritative, objective, perfectly logical and error-proof. SV pushes that narrative, for example saying computer-driven cars are safer because computers don't have human flaws (that old sales technique - compare only the competition's unique flaws and don't mention your own; computers have their own flaws that humans lack). And of course, that one is popular on HN. (IMHO the way to educate the public is to spread the narrative that every computer is created (designed, programmed) by normal, flawed, biased, etc. humans, and controlled by other flawed humans for their own agenda. And that computers have their own flaws.)
- AndrewKemendo 2mo ago
  The fact that you conflate LLMs and AI means you’re unqualified to be part of the discussion
- AtlasBarfed 2mo ago
  The Elon buttlicker types praise llms as the new knowledge gods. The general public distrusts them cuz they know that they're coming for their jobs. But they will use them out of convenience
MBCook 2mo ago
I’m absolutely willing to believe there are some people that stupid. I doubt anyone who works on reactors is that stupid. But some idiot somewhere is going to put some industrial stuff under AI control pretty soon and murder a few people at minimum.
jmalicki 2mo ago
AI labs are already working on getting LLMs to control nuclear reactors, and hiring experienced reactor operators to help train them: https://www.reddit.com/r/nuclear/comments/1qnzxvg/this_job_posting_is_insane/#lightbox https://www.reddit.com/r/nuclear/comments/1qnzxvg/this_job_p...

iwontberude 2mo ago

[dead]

delichon 2mo ago

This was a power safety event in the sense that safety was accidentally added. Like oops, we accidentally put the pin back in the grenade.

AndrewKemendo 2mo ago
Which indicates that there’s so much understanding of how it works and the power that it uses that compensatory systems have been created to increase its reliability what precisely is your point of describing how quality production engineering for infrastructure works?

dist-epoch 2mo ago

> nearly 100% of events go completely unnoticed by the public "Nearly" is doing a lot of heavy lifting there. Those "nearly" 0% of events which DO go noticed by the public, like you know, Fukushima, tend to be world news for weeks at a time.