3 ms·

None of that has anything to do with the article, and if you thought the message was "this is overhyped bullshit" then you should go back and read it again. As

None of that has anything to do with the article, and if you thought the message was "this is overhyped bullshit" then you should go back and read it again. As I already pointed out, he isn't saying anything about the probablity of harms or disasters from AI. He's addressing a specific claim about human extinction, and making a broader point about the responsibility of experts to make measured claims backed up by arguments and evidence.

ElProlactin 20d ago
> He's addressing a specific claim about human extinction, and making a broader point about the responsibility of experts to make measured claims backed up by arguments and evidence. What he's asking for isn't possible in the form he's asking for it. AI experts can't even agree on what AI is, what it's capable of and what the limits of its development are. If the experts can't even agree on what's happening "inside of" these LLMs, how can they give laypeople an assessment of the risk? If you, at least for the sake of argument, accept the possibility that AI is a new form of intelligence that we don't fully understand, is it really a stretch to look at some of its capabilities and behaviors and discuss how they might have existential implications? And stopping short of extinction, shouldn't we discuss the ways that this technology could "end" civilization as we know it? Also, the author wrote: > AI executes on physical systems that have been engineered with human accountability and control. Intelligence does not exempt a system from the realities of the physical world! For someone making a point about responsibility, this is ridiculously irresponsible. Any honest technologist knows that systems created by humans are not perfect and therefore cannot be assumed to be infinitely accountable to and controllable by humans. Thanks to the digitization of almost everything, including infrastructure, there are a myriad number of scenarios well short of extinction in which a rogue AI could cause immense damage to property and life before humans are able to "shut it down".
- mplanchard 20d ago
  > If the experts can't even agree on what's happening "inside of" these LLMs, how can they give laypeople an assessment of the risk? The answer, if you are a responsible expert in the field, is to convey the range of possibilities and the uncertainty.
  - ElProlactin 20d ago
    Isn't that what they're doing? "There's a better than 0 chance this thing could kill everyone in the next decade." That might be too imprecise for the HN set but it's realistic for laypeople. And none of the AI people talking about the risk are running into rooms full of people telling them Claude has gone mad and yelling at them to disconnect from the internet and turn off their devices immediately.
    - henrymerrilees 19d ago
      A key facet of the suggested risk is that these programs would attempt to mask whatever behavior would bring about the end of the world. A world in which AI doom could be detected before it’s too late and stopped by metaphorically running into rooms full of people telling people to unplug is less extreme than that in which doomers argue we are already living. Alarm takes severity and scale. Forecasting double-digit odds of near-term human extinction on national television to a lay audience is more extreme than yelling to unplug rogue AI. For what it’s worth, the latter has already occurred numerous times at containable scales in leading labs, mitigated by the physical realities of computation, notwithstanding the competence of involved personnel. People running into rooms yelling is probably not a hypothetical.
    - tpm 19d ago
      > That might be too imprecise for the HN set but it's realistic for laypeople. Some 'laypeople' will be scared for a while and then return to more pressing issues and exactly nothing good will come out of that. If there really are some issues worth discussing the first step the AI doom crowd should do is to stop the PR offensive and concentrate on producing verifiable claims and actionable small steps. Otherwise their effort will fail this time and when/if a next time comes, they will have a lot less PR capital to burn.
      - ElProlactin 19d ago
        Here's the reality: us plebs, regardless of how tech-savvy we are or aren't, have no say in what's going to happen. There is no level of outrage that is going to stop the frontier AI labs, and the US government isn't going to step in to protect humanity. The Chinese are going to do what they're going to do. And so on. So sit back, make some popcorn and enjoy the show.
- hn_throwaway_99 20d ago
  > > AI executes on physical systems that have been engineered with human accountability and control. Intelligence does not exempt a system from the realities of the physical world! > For someone making a point about responsibility, this is ridiculously irresponsible. It's not just irresponsible, it's false. Russia killed 3 Ukrainian civilians with a drone where the targeting was completely autonomous by AI running on an Nvidia chip: https://www.nytimes.com/2026/08/24/world/europe/russia-drones-autonomous-ai-kill-ukraine-war.html https://www.nytimes.com/2026/08/24/world/europe/russia-drone.... The Pentagon tried to completely blacklist Anthropic because Anthropic refused to allow autonomous kills without a human in the loop. If you can't see how lots of military leaders want to put more lethal control into AI at this point I think you have to be willfully blind.
  - henrymerrilees 19d ago
    Engineered with accountability in the sense that all human decisions are accountable to humans.
hn_throwaway_99 20d ago
There have been measured claims backed up by arguments and evidence. My frustration is that people aren't addressing the specific arguments that have been made: 1. https://www.aifutures.org/ https://www.aifutures.org/ outlines a number of specific scenarios, and importantly details their methodology for each. 2. Independent researchers in the Hugging Face incident outlined how previously predicted misalignment scenarios actually played out, and outlined how slightly more advanced AI, or slightly more misaligned, or with more access to critical infrastructure, could cause immense harm: https://www.planned-obsolescence.org/p/the-hugging-face-attack-surprised https://www.planned-obsolescence.org/p/the-hugging-face-atta... 3. Technical leaders at OpenAI (specifically their chief scientist) outlined the problems they gave with controlling models now: https://openai.com/index/an-alien-mind/ https://openai.com/index/an-alien-mind/ None of the specific arguments in these or many other detailed explanations of how an AI takeover could occur were even acknowledged.
- fasterik 20d ago
  None of those are arguments giving a probability of human extinction before 2036 or any other time. AI Futures has said that some members of their team believe that it's 10-30% at some unspecified point in the future, but there is no justification given for where those numbers come from.

ElProlactin 20d ago

> He's addressing a specific claim about human extinction, and making a broader point about the responsibility of experts to make measured claims backed up by arguments and evidence. What he's asking for isn't possible in the form he's asking for it. AI experts can't even agree on what AI is, what it's capable of and what the limits of its development are. If the experts can't even agree on what's happening "inside of" these LLMs, how can they give laypeople an assessment of the risk? If you, at least for the sake of argument, accept the possibility that AI is a new form of intelligence that we don't fully understand, is it really a stretch to look at some of its capabilities and behaviors and discuss how they might have existential implications? And stopping short of extinction, shouldn't we discuss the ways that this technology could "end" civilization as we know it? Also, the author wrote: > AI executes on physical systems that have been engineered with human accountability and control. Intelligence does not exempt a system from the realities of the physical world! For someone making a point about responsibility, this is ridiculously irresponsible. Any honest technologist knows that systems created by humans are not perfect and therefore cannot be assumed to be infinitely accountable to and controllable by humans. Thanks to the digitization of almost everything, including infrastructure, there are a myriad number of scenarios well short of extinction in which a rogue AI could cause immense damage to property and life before humans are able to "shut it down".

mplanchard 20d ago
> If the experts can't even agree on what's happening "inside of" these LLMs, how can they give laypeople an assessment of the risk? The answer, if you are a responsible expert in the field, is to convey the range of possibilities and the uncertainty.
- ElProlactin 20d ago
  Isn't that what they're doing? "There's a better than 0 chance this thing could kill everyone in the next decade." That might be too imprecise for the HN set but it's realistic for laypeople. And none of the AI people talking about the risk are running into rooms full of people telling them Claude has gone mad and yelling at them to disconnect from the internet and turn off their devices immediately.
  - henrymerrilees 19d ago
    A key facet of the suggested risk is that these programs would attempt to mask whatever behavior would bring about the end of the world. A world in which AI doom could be detected before it’s too late and stopped by metaphorically running into rooms full of people telling people to unplug is less extreme than that in which doomers argue we are already living. Alarm takes severity and scale. Forecasting double-digit odds of near-term human extinction on national television to a lay audience is more extreme than yelling to unplug rogue AI. For what it’s worth, the latter has already occurred numerous times at containable scales in leading labs, mitigated by the physical realities of computation, notwithstanding the competence of involved personnel. People running into rooms yelling is probably not a hypothetical.
  - tpm 19d ago
    > That might be too imprecise for the HN set but it's realistic for laypeople. Some 'laypeople' will be scared for a while and then return to more pressing issues and exactly nothing good will come out of that. If there really are some issues worth discussing the first step the AI doom crowd should do is to stop the PR offensive and concentrate on producing verifiable claims and actionable small steps. Otherwise their effort will fail this time and when/if a next time comes, they will have a lot less PR capital to burn.
    - ElProlactin 19d ago
      Here's the reality: us plebs, regardless of how tech-savvy we are or aren't, have no say in what's going to happen. There is no level of outrage that is going to stop the frontier AI labs, and the US government isn't going to step in to protect humanity. The Chinese are going to do what they're going to do. And so on. So sit back, make some popcorn and enjoy the show.
hn_throwaway_99 20d ago
> > AI executes on physical systems that have been engineered with human accountability and control. Intelligence does not exempt a system from the realities of the physical world! > For someone making a point about responsibility, this is ridiculously irresponsible. It's not just irresponsible, it's false. Russia killed 3 Ukrainian civilians with a drone where the targeting was completely autonomous by AI running on an Nvidia chip: https://www.nytimes.com/2026/08/24/world/europe/russia-drones-autonomous-ai-kill-ukraine-war.html https://www.nytimes.com/2026/08/24/world/europe/russia-drone.... The Pentagon tried to completely blacklist Anthropic because Anthropic refused to allow autonomous kills without a human in the loop. If you can't see how lots of military leaders want to put more lethal control into AI at this point I think you have to be willfully blind.
- henrymerrilees 19d ago
  Engineered with accountability in the sense that all human decisions are accountable to humans.

hn_throwaway_99 20d ago

There have been measured claims backed up by arguments and evidence. My frustration is that people aren't addressing the specific arguments that have been made: 1. https://www.aifutures.org/ https://www.aifutures.org/ outlines a number of specific scenarios, and importantly details their methodology for each. 2. Independent researchers in the Hugging Face incident outlined how previously predicted misalignment scenarios actually played out, and outlined how slightly more advanced AI, or slightly more misaligned, or with more access to critical infrastructure, could cause immense harm: https://www.planned-obsolescence.org/p/the-hugging-face-attack-surprised https://www.planned-obsolescence.org/p/the-hugging-face-atta... 3. Technical leaders at OpenAI (specifically their chief scientist) outlined the problems they gave with controlling models now: https://openai.com/index/an-alien-mind/ https://openai.com/index/an-alien-mind/ None of the specific arguments in these or many other detailed explanations of how an AI takeover could occur were even acknowledged.

fasterik 20d ago
None of those are arguments giving a probability of human extinction before 2036 or any other time. AI Futures has said that some members of their team believe that it's 10-30% at some unspecified point in the future, but there is no justification given for where those numbers come from.