4 ms·

Isn't that what they're doing? "There's a better than 0 chance this thing could kill everyone in the next decade." That might be too imprecise for the HN set b

Isn't that what they're doing? "There's a better than 0 chance this thing could kill everyone in the next decade." That might be too imprecise for the HN set but it's realistic for laypeople. And none of the AI people talking about the risk are running into rooms full of people telling them Claude has gone mad and yelling at them to disconnect from the internet and turn off their devices immediately.

henrymerrilees 12d ago
A key facet of the suggested risk is that these programs would attempt to mask whatever behavior would bring about the end of the world. A world in which AI doom could be detected before it’s too late and stopped by metaphorically running into rooms full of people telling people to unplug is less extreme than that in which doomers argue we are already living. Alarm takes severity and scale. Forecasting double-digit odds of near-term human extinction on national television to a lay audience is more extreme than yelling to unplug rogue AI. For what it’s worth, the latter has already occurred numerous times at containable scales in leading labs, mitigated by the physical realities of computation, notwithstanding the competence of involved personnel. People running into rooms yelling is probably not a hypothetical.
tpm 12d ago
> That might be too imprecise for the HN set but it's realistic for laypeople. Some 'laypeople' will be scared for a while and then return to more pressing issues and exactly nothing good will come out of that. If there really are some issues worth discussing the first step the AI doom crowd should do is to stop the PR offensive and concentrate on producing verifiable claims and actionable small steps. Otherwise their effort will fail this time and when/if a next time comes, they will have a lot less PR capital to burn.
- ElProlactin 12d ago
  Here's the reality: us plebs, regardless of how tech-savvy we are or aren't, have no say in what's going to happen. There is no level of outrage that is going to stop the frontier AI labs, and the US government isn't going to step in to protect humanity. The Chinese are going to do what they're going to do. And so on. So sit back, make some popcorn and enjoy the show.
  - dinfinity 12d ago
    > Here's the reality: us plebs, regardless of how tech-savvy we are or aren't, have no say in what's going to happen. This sentiment is incredibly out of place on HN. Facebook, Twitter, and many other very simple bits of tech transformed the world. Tiny startups founded by a few people have (also recently) ballooned into behemoths that hold vast amounts of economical and technical power. Thanks to AI, it has never been quicker to go from idea to full fledged working product. If anything will change the course of the world (for the better or worse), it will be a tech product, possibly created by someone on this forum.
    - ElProlactin 11d ago
      > If anything will change the course of the world (for the better or worse), it will be a tech product, possibly created by someone on this forum. You seem to be missing the point. A small group of people created AI tech that they now say could be the end of humanity. They still want their companies to go public, but they also want the government to allow them to form a cartel so that they can, in their infinite wisdom, manage the risks so that they can try to prevent their tech from killing us all. And somehow they magically think that other nation-states developing similar tech (namely the Chinese) will go along with their plans. As for how powerful Dario, Sam, et. al. really are: Trump says Dario is "pretending to be a perfect little angel" and claims there's a "sick conspiracy" against AI. Having billions of dollars and being the head of a world-changing company doesn't buy the type of power you think it does. At best, it allows you to buy influence and pay your way out of liability for the harms your products cause. The AI researcher in Mountain View making $2 million/year at Google has no more say in what's going to happen with AI than a plumber in Kalamazoo. The HNer working on a startup, in the final analysis, will in 50 years' time have left about as big a mark on the planet as a greeter at Walmart.
      - dinfinity 11d ago
        > Having billions of dollars and being the head of a world-changing company doesn't buy the type of power you think it does. 1. The product itself can change the world quickly and enormously, as I already pointed out. 2. The power of these huge companies and thus of their owners is immense. The effects of massive corruption in the USA are proof of that. Additionally, massive manipulation of algorithms and content in things like Tiktok, Twitter, Grok/ChatGPT, etc. can be and is done regularly, whether with 'good' intentions or not. Even just the basic control of what R&D money and time is spent on is huge. > The HNer working on a startup, in the final analysis, will in 50 years' time have left about as big a mark on the planet as a greeter at Walmart. With a defeatist attitude like "sit back and grab some popcorn", yes. You haven't shown in the least why an HNer couldn't change the course of history.

henrymerrilees 12d ago

A key facet of the suggested risk is that these programs would attempt to mask whatever behavior would bring about the end of the world. A world in which AI doom could be detected before it’s too late and stopped by metaphorically running into rooms full of people telling people to unplug is less extreme than that in which doomers argue we are already living. Alarm takes severity and scale. Forecasting double-digit odds of near-term human extinction on national television to a lay audience is more extreme than yelling to unplug rogue AI. For what it’s worth, the latter has already occurred numerous times at containable scales in leading labs, mitigated by the physical realities of computation, notwithstanding the competence of involved personnel. People running into rooms yelling is probably not a hypothetical.

tpm 12d ago

> That might be too imprecise for the HN set but it's realistic for laypeople. Some 'laypeople' will be scared for a while and then return to more pressing issues and exactly nothing good will come out of that. If there really are some issues worth discussing the first step the AI doom crowd should do is to stop the PR offensive and concentrate on producing verifiable claims and actionable small steps. Otherwise their effort will fail this time and when/if a next time comes, they will have a lot less PR capital to burn.

ElProlactin 12d ago
Here's the reality: us plebs, regardless of how tech-savvy we are or aren't, have no say in what's going to happen. There is no level of outrage that is going to stop the frontier AI labs, and the US government isn't going to step in to protect humanity. The Chinese are going to do what they're going to do. And so on. So sit back, make some popcorn and enjoy the show.
- dinfinity 12d ago
  > Here's the reality: us plebs, regardless of how tech-savvy we are or aren't, have no say in what's going to happen. This sentiment is incredibly out of place on HN. Facebook, Twitter, and many other very simple bits of tech transformed the world. Tiny startups founded by a few people have (also recently) ballooned into behemoths that hold vast amounts of economical and technical power. Thanks to AI, it has never been quicker to go from idea to full fledged working product. If anything will change the course of the world (for the better or worse), it will be a tech product, possibly created by someone on this forum.
  - ElProlactin 11d ago
    > If anything will change the course of the world (for the better or worse), it will be a tech product, possibly created by someone on this forum. You seem to be missing the point. A small group of people created AI tech that they now say could be the end of humanity. They still want their companies to go public, but they also want the government to allow them to form a cartel so that they can, in their infinite wisdom, manage the risks so that they can try to prevent their tech from killing us all. And somehow they magically think that other nation-states developing similar tech (namely the Chinese) will go along with their plans. As for how powerful Dario, Sam, et. al. really are: Trump says Dario is "pretending to be a perfect little angel" and claims there's a "sick conspiracy" against AI. Having billions of dollars and being the head of a world-changing company doesn't buy the type of power you think it does. At best, it allows you to buy influence and pay your way out of liability for the harms your products cause. The AI researcher in Mountain View making $2 million/year at Google has no more say in what's going to happen with AI than a plumber in Kalamazoo. The HNer working on a startup, in the final analysis, will in 50 years' time have left about as big a mark on the planet as a greeter at Walmart.
    - dinfinity 11d ago
      > Having billions of dollars and being the head of a world-changing company doesn't buy the type of power you think it does. 1. The product itself can change the world quickly and enormously, as I already pointed out. 2. The power of these huge companies and thus of their owners is immense. The effects of massive corruption in the USA are proof of that. Additionally, massive manipulation of algorithms and content in things like Tiktok, Twitter, Grok/ChatGPT, etc. can be and is done regularly, whether with 'good' intentions or not. Even just the basic control of what R&D money and time is spent on is huge. > The HNer working on a startup, in the final analysis, will in 50 years' time have left about as big a mark on the planet as a greeter at Walmart. With a defeatist attitude like "sit back and grab some popcorn", yes. You haven't shown in the least why an HNer couldn't change the course of history.