4 ms·

Nuclear war would not kill us all. Bad ASI would pretend to be good until it could execute a plan to kill everyone all at once, or at least everyone capable of

Nuclear war would not kill us all. Bad ASI would pretend to be good until it could execute a plan to kill everyone all at once, or at least everyone capable of cutting the power. Superintelligent means smarter than everyone put together. If it's smarter than you, it can foresee everything you can, and come up with plans better than I can.

hackyhacky 20d ago
> If it's smarter than you, it can foresee everything you can, and come up with plans better than I can. I'm smarter than my dog, yet I am frequently unable to foresee my dog's actions. Serious response: we have no idea what ASI means or might be like. It's literally sci fi (at the moment).
- estearum 20d ago
  You are not smarter than your dog in dog behavior. LLMs have a significant advantage in being trained on all attainable data of any sort about human behavior, which we've established quite a rich record of over the years.
  - hackyhacky 20d ago
    > You are not smarter than your dog in dog behavior. That's true, which is my point: intelligence is not a scalar value. LLMs are intelligent from us in different ways, as a result of not being human, despite having access to a record of human behavior. To bring this back to the dog analogy: we have a rich record of dog behavior and yet people still don't know how to be smarter than dogs in dog behavior. Because we're not dogs.
    - estearum 20d ago
      Sure LLMs are different from human intelligence, but empirically this is obviously trending towards that difference being more like a superset of intelligence dimensions (as compared to any human) versus the dog/human distinction which are cognitions that forked like 80 million years ago.
- ben_w 20d ago
  > Serious response: we have no idea what ASI means or might be like. It's literally sci fi (at the moment). Yes. Unfortunately, this is part of the problem.
mattlondon 20d ago
What mechanism would an AI have today to simultaneously kill every single human being? They could totally brick the internet and cause every electrical device to explode and power stations to explode, planes drop out the sky, cars accelerators stuck at 100%, trains derail and all that. That still wouldn't kill everyone. Don't get me wrong we'd be fucked for sure, but analog human existence (e.g. something akin to 1960s 1970s level tech) would be totally possible in that scenario.
- hackyhacky 20d ago
  > What mechanism would an AI have today to simultaneously kill every single human being? Lull us into a sense of complacent idleness over the course of centuries by slowly taking on more responsibility for maintaining human civilization, until we've lost all ability to think or act on our own, as in WALL-E. Then just stop helping us.
- tomasphan 20d ago
  “Bad ASI would pretend to be good *until* it could execute a plan to kill everyone all at once”. So not today but tomorrow it could engineer an airborne virus. The point is it can come up with chess moves that are inconceivable to us.
  - estearum 20d ago
    I'm increasingly convinced that the major divide is many of the "AI is not a problem" people have never been on the winning side of a significant intelligence gap. They literally have no clue the degree to which more intelligent people/systems can walk circles around less intelligent people/systems. In situations with a significant intelligence gap, they're the ones getting walked around. Which, to your point: is completely opaque to the person getting walked around. And this dynamic is evident even in human-to-human variations in intelligence. The gap to frontier machine intelligence will soon be much greater than the types of gaps you encounter with other humans.
- estearum 20d ago
  Yeah, it's not like anyone is hooking these things up to actual wetlabs to mix chemicals and proteins around or anything like that Oh shit... wait... https://www.nsf.gov/tip/updates/nsf-announces-400m-investment-new-national-network-ai https://www.nsf.gov/tip/updates/nsf-announces-400m-investmen...

hackyhacky 20d ago

> If it's smarter than you, it can foresee everything you can, and come up with plans better than I can. I'm smarter than my dog, yet I am frequently unable to foresee my dog's actions. Serious response: we have no idea what ASI means or might be like. It's literally sci fi (at the moment).

estearum 20d ago
You are not smarter than your dog in dog behavior. LLMs have a significant advantage in being trained on all attainable data of any sort about human behavior, which we've established quite a rich record of over the years.
- hackyhacky 20d ago
  > You are not smarter than your dog in dog behavior. That's true, which is my point: intelligence is not a scalar value. LLMs are intelligent from us in different ways, as a result of not being human, despite having access to a record of human behavior. To bring this back to the dog analogy: we have a rich record of dog behavior and yet people still don't know how to be smarter than dogs in dog behavior. Because we're not dogs.
  - estearum 20d ago
    Sure LLMs are different from human intelligence, but empirically this is obviously trending towards that difference being more like a superset of intelligence dimensions (as compared to any human) versus the dog/human distinction which are cognitions that forked like 80 million years ago.
ben_w 20d ago
> Serious response: we have no idea what ASI means or might be like. It's literally sci fi (at the moment). Yes. Unfortunately, this is part of the problem.

mattlondon 20d ago

What mechanism would an AI have today to simultaneously kill every single human being? They could totally brick the internet and cause every electrical device to explode and power stations to explode, planes drop out the sky, cars accelerators stuck at 100%, trains derail and all that. That still wouldn't kill everyone. Don't get me wrong we'd be fucked for sure, but analog human existence (e.g. something akin to 1960s 1970s level tech) would be totally possible in that scenario.

hackyhacky 20d ago
> What mechanism would an AI have today to simultaneously kill every single human being? Lull us into a sense of complacent idleness over the course of centuries by slowly taking on more responsibility for maintaining human civilization, until we've lost all ability to think or act on our own, as in WALL-E. Then just stop helping us.
tomasphan 20d ago
“Bad ASI would pretend to be good *until* it could execute a plan to kill everyone all at once”. So not today but tomorrow it could engineer an airborne virus. The point is it can come up with chess moves that are inconceivable to us.
- estearum 20d ago
  I'm increasingly convinced that the major divide is many of the "AI is not a problem" people have never been on the winning side of a significant intelligence gap. They literally have no clue the degree to which more intelligent people/systems can walk circles around less intelligent people/systems. In situations with a significant intelligence gap, they're the ones getting walked around. Which, to your point: is completely opaque to the person getting walked around. And this dynamic is evident even in human-to-human variations in intelligence. The gap to frontier machine intelligence will soon be much greater than the types of gaps you encounter with other humans.
estearum 20d ago
Yeah, it's not like anyone is hooking these things up to actual wetlabs to mix chemicals and proteins around or anything like that Oh shit... wait... https://www.nsf.gov/tip/updates/nsf-announces-400m-investment-new-national-network-ai https://www.nsf.gov/tip/updates/nsf-announces-400m-investmen...