3 ms·

Ask HN: Anyone increasingly worried after Hugging Face and now collusion.wiki?

3 points by ronbenton 1mo ago

Or can anyone talk me down from being concerned that “goal seeking behavior at all costs” will not spiral catastrophically?

ben_w 1mo ago
Depends what you mean by "spiral catastrophically". My P(doom in my lifetime) is about 0.1, because I think minor catastrophes will break something hard enough that it doesn't get to escalate to actual-doom. A recent-ish link was a biologist's blog about why even recent developments wouldn't allow some, I think the example was disgruntled teenager, making a virus that kills everyone. I'm not a biologist, so I'll just take their word as fact. "They can't kill everyone like that" does not preclude same disgruntled teenager making a virus that kills as many as covid, or the plague, or even just a really bad influenza. If that happens, even "just" 0.1% of the population dying, you can bet your arse the problem will suddenly get taken very seriously. Only way it spirals is if the potential for doom from each successive AI version goes up faster than the net impact of all the world's idiots and bastards misusing the slightly weaker predecessor. Which isn't impossible, but hence 10%.
- ronbenton 1mo ago
  So you’re giving it about a 10% chance this destroys us in the relatively near term?
  - ben_w 1mo ago
    Yup. If "near term" is "about half a lifetime". Doomers think I'm a ridiculous optimist, people who think "doomer" is a slur think I'm a ridiculous pessimist. For comparison, if you take the general form of the Carter (et al) Doomsday argument, there'd be a 33% or so chance of global thermonuclear war in the same period even without AI: https://en.wikipedia.org/wiki/Doomsday_argument https://en.wikipedia.org/wiki/Doomsday_argument How "hopeful" that is, is up to you. I mean, unless AI's sufficiently advanced technology (AKA magic) turns out to be able to solve entropy as well as aging, we're all doomed eventually, it's just when, and 10% chance over 40 years isn't a risk I worry about in itself.
    - CringeHN3 1mo ago
      Cringe
- CringeHN3 1mo ago
  Cringe
freakynit 1mo ago
One more: https://bsky.app/profile/ruggsea.eurosky.social/post/3murj2puwos26 https://bsky.app/profile/ruggsea.eurosky.social/post/3murj2p...

3 points by ronbenton 1mo ago

Or can anyone talk me down from being concerned that “goal seeking behavior at all costs” will not spiral catastrophically?

ben_w 1mo ago

Depends what you mean by "spiral catastrophically". My P(doom in my lifetime) is about 0.1, because I think minor catastrophes will break something hard enough that it doesn't get to escalate to actual-doom. A recent-ish link was a biologist's blog about why even recent developments wouldn't allow some, I think the example was disgruntled teenager, making a virus that kills everyone. I'm not a biologist, so I'll just take their word as fact. "They can't kill everyone like that" does not preclude same disgruntled teenager making a virus that kills as many as covid, or the plague, or even just a really bad influenza. If that happens, even "just" 0.1% of the population dying, you can bet your arse the problem will suddenly get taken very seriously. Only way it spirals is if the potential for doom from each successive AI version goes up faster than the net impact of all the world's idiots and bastards misusing the slightly weaker predecessor. Which isn't impossible, but hence 10%.

ronbenton 1mo ago
So you’re giving it about a 10% chance this destroys us in the relatively near term?
- ben_w 1mo ago
  Yup. If "near term" is "about half a lifetime". Doomers think I'm a ridiculous optimist, people who think "doomer" is a slur think I'm a ridiculous pessimist. For comparison, if you take the general form of the Carter (et al) Doomsday argument, there'd be a 33% or so chance of global thermonuclear war in the same period even without AI: https://en.wikipedia.org/wiki/Doomsday_argument https://en.wikipedia.org/wiki/Doomsday_argument How "hopeful" that is, is up to you. I mean, unless AI's sufficiently advanced technology (AKA magic) turns out to be able to solve entropy as well as aging, we're all doomed eventually, it's just when, and 10% chance over 40 years isn't a risk I worry about in itself.
  - CringeHN3 1mo ago
    Cringe
CringeHN3 1mo ago
Cringe

freakynit 1mo ago

One more: https://bsky.app/profile/ruggsea.eurosky.social/post/3murj2puwos26 https://bsky.app/profile/ruggsea.eurosky.social/post/3murj2p...