4 ms·

If you actually believe that your invention will lead to the extinction of humanity, you would be acting very very differently. Namely, you would not be inventi

by mlsu 20d ago

If you actually believe that your invention will lead to the extinction of humanity, you would be acting very very differently. Namely, you would not be inventing the thing that you think is going to kill all of humanity, and probably you would be doing everything in your power to prevent it from being invented, up to and including sabotage of the company that YOU work for, which is currently trying to invent that thing. And yet, here's Fable 6, more capable than ever before, oh BTW we are IPOing, oh, BTW, I have stock options...

mrob 20d ago
Sabotaging your own company is not enough. You'd have to sabotage every other AI lab too. If you assume that's impossible, attempting to be the first to produce ASI is a rational alternative, even if you think it will most likely kill you. At least that way you have some control over the P(doom).
- deleted 20d ago
  [deleted]
- goatlover 20d ago
  This sounds like a rationalization for gambling with everyone's lives, assuming they really believe such a thing.
  - mrob 20d ago
    What's the alternative? There's no realistic way to globally coordinate and enforce a ban. A direct action campaign against AI labs would only motivate increased security until sabotage becomes impossible. Starting a nuclear war would probably be enough to stop dangerous AI research, and that's only likely to kill about 50% of humans instead 100%, so it's a great improvement, but how is an individual supposed to start a nuclear war? Of course it would be better if everybody stopped, but it's a prisoner's dilemma scenario with no possibility of future rounds, so the rational play is to defect.
    - MentalM 20d ago
      > There's no realistic way to globally coordinate and enforce a ban. I don't see what is unrealistic about that. The major players already have strong incentives to freeze AI advancements at the the current state, while the technology is highly efficient for totalitarian surveillance systems that can eliminate any threat to their power in the buds, but still not so advanced that it could create crises that might endanger their power. So I think the situation where several largest countries declare AI a deadly threat for humanity and start fighting it development much like they fight ISIS to be quite plausible. This isn't a prisoner's dilemma; the distribution of risks and rewards is entirely different. Ban AI research, give away advanced models, bomb a couple of Russian datacenters where they launching those models unsafely, and then AI development will stop in the entire world.
- bigstrat2003 20d ago
  > If you assume that's impossible, attempting to be the first to produce ASI is a rational alternative, even if you think it will most likely kill you. That is completely irrational.
  - mrob 20d ago
    1% chance of survival beats 0%.
  - pixl97 20d ago
    Humans aren't exactly rational
aaroninsf 20d ago
This is not an argument, it's an unfounded claim based on a straw argument. The threat is distributed; the problem is systemic. Social herd behavior is very real, and it's a probability space, not a certainty. Similar case: climate change is without any question causing millions of deaths per year and may well kill billions, not to mention destroying ecosystems, and how may people do you know who have given up eating beef or flying...?
- mlsu 20d ago
  You're right. That's why I'm doing my best to give the oil companies more power and influence. They are, after all, in the best position to stop climate change!
  - frabcus 20d ago
    You joke, but alas... "Fossil fuel lobbyists flood COP30 climate talks in Brazil, with largest ever attendance share" https://kickbigpollutersout.org/Release-Kick-Out-The-Suits-COP30 https://kickbigpollutersout.org/Release-Kick-Out-The-Suits-C...
- MentalM 20d ago
  > Similar case: climate change is without any question causing millions of deaths per year It is WERY questionable statement. > and how may people do you know who have given up eating beef or flying That is exactly because absolute majority of people do not think climate change to be that big of a problem

by mlsu 20d ago

mrob 20d ago

Sabotaging your own company is not enough. You'd have to sabotage every other AI lab too. If you assume that's impossible, attempting to be the first to produce ASI is a rational alternative, even if you think it will most likely kill you. At least that way you have some control over the P(doom).

deleted 20d ago
[deleted]
goatlover 20d ago
This sounds like a rationalization for gambling with everyone's lives, assuming they really believe such a thing.
- mrob 20d ago
  What's the alternative? There's no realistic way to globally coordinate and enforce a ban. A direct action campaign against AI labs would only motivate increased security until sabotage becomes impossible. Starting a nuclear war would probably be enough to stop dangerous AI research, and that's only likely to kill about 50% of humans instead 100%, so it's a great improvement, but how is an individual supposed to start a nuclear war? Of course it would be better if everybody stopped, but it's a prisoner's dilemma scenario with no possibility of future rounds, so the rational play is to defect.
  - MentalM 20d ago
    > There's no realistic way to globally coordinate and enforce a ban. I don't see what is unrealistic about that. The major players already have strong incentives to freeze AI advancements at the the current state, while the technology is highly efficient for totalitarian surveillance systems that can eliminate any threat to their power in the buds, but still not so advanced that it could create crises that might endanger their power. So I think the situation where several largest countries declare AI a deadly threat for humanity and start fighting it development much like they fight ISIS to be quite plausible. This isn't a prisoner's dilemma; the distribution of risks and rewards is entirely different. Ban AI research, give away advanced models, bomb a couple of Russian datacenters where they launching those models unsafely, and then AI development will stop in the entire world.
bigstrat2003 20d ago
> If you assume that's impossible, attempting to be the first to produce ASI is a rational alternative, even if you think it will most likely kill you. That is completely irrational.
- mrob 20d ago
  1% chance of survival beats 0%.
- pixl97 20d ago
  Humans aren't exactly rational

aaroninsf 20d ago

This is not an argument, it's an unfounded claim based on a straw argument. The threat is distributed; the problem is systemic. Social herd behavior is very real, and it's a probability space, not a certainty. Similar case: climate change is without any question causing millions of deaths per year and may well kill billions, not to mention destroying ecosystems, and how may people do you know who have given up eating beef or flying...?

mlsu 20d ago
You're right. That's why I'm doing my best to give the oil companies more power and influence. They are, after all, in the best position to stop climate change!
- frabcus 20d ago
  You joke, but alas... "Fossil fuel lobbyists flood COP30 climate talks in Brazil, with largest ever attendance share" https://kickbigpollutersout.org/Release-Kick-Out-The-Suits-COP30 https://kickbigpollutersout.org/Release-Kick-Out-The-Suits-C...
MentalM 20d ago
> Similar case: climate change is without any question causing millions of deaths per year It is WERY questionable statement. > and how may people do you know who have given up eating beef or flying That is exactly because absolute majority of people do not think climate change to be that big of a problem