4 ms·

LessWrong and its roots in “rationalism” are consistently fooled by these kinds of scarecrow fears. Here’s why this argument is misguided: 1. There is no singu

by aabhay 4y ago

LessWrong and its roots in “rationalism” are consistently fooled by these kinds of scarecrow fears. Here’s why this argument is misguided: 1. There is no singular notion of “alignment” and (IMHO) AGI at all. Humans have been misaligned with each other, through war, debate, disagreement, and so on. AI is more in danger of being “ “misaligned” to one group’s needs than to the overarching totality of humanity. Focusing on the possibly more mundane concerns of who gets privileged access to guide the AI, what it deems to be “accurate” and correct knowledge, etc. is much more important to our shared future with AI tools, at least to me. 2. LW almost always ignores the “second order” effects, “snowball melting” that occurs once anything becomes more powerful and centralized. At the moment at which AI seems to have access to too many levers and dials on taking action, it will certainly run into the kinds of limitations that we would run into. Want an AI to create a super bomb that can annihilate us? Good luck getting Uranium, setting up extraction facilities, avoiding detection from governments. Want to create an AI super-trader that becomes infinitely wealthy? Good luck competing against other AI traders that bring arbs down to zero. 3. I think the most salient thing here is that an AI that has the power and capability to enact massive change will almost certainly need to organize large scale human help. And at that point, I would not characterize this effort to destroy humanity as one purely “AI”-created. It would require enlisting a multitude of people desiring this “omnicide”, and human hands will be bloody as well.

kypro 4y ago
> And at that point, I would not characterize this effort to destroy humanity as one purely “AI”-created. It would require enlisting a multitude of people desiring this “omnicide”, and human hands will be bloody as well. Why would a super intelligence require humans?
- corobo 4y ago
  This is the real question. We should be seeding it with the idea that humans are cute. Any plea towards or away from extermination will end with extermination. We'll either look like an easy target or we'll look dangerous and need taking out. If we're cute af in the eyes of AI we'll be kept as pets.
- kibwen 4y ago
  Because an omniscient brain in a tin can is still a brain in a tin can. It doesn't matter how intelligent something is if it can't otherwise affect the outside world. It's like executing a pure function: regardless of what it's computing, the computation is pointless because it cannot be observed. Don't worry about AI. Worry about humans with access to AI.
  - kypro 4y ago
    I think you're probably right, but this should be treated as an unknown, and wouldn't be something I'd be comfortable to assume. You could imagine a fish arguing that if something can't swim fast and doesn't have sharp teeth then it poses no threat regardless of it's intelligence. But this is just because fish evolved to not be eaten by bigger fish and its intelligence and limited knowledge of the world makes it unaware of the other threats they could face. So while I agree it would be hard to imagine, a super intelligence in theory might find a way to affect the outside world. Perhaps this could be through some unknown physics, or more conceivably, perhaps it would first pose as a helpful AI showing humans how to build advanced robotics only to later co-opt those robotics for its own objectives.
scarmig 4y ago
Translating between the virtual realm and the physical realm is where AI would have trouble. But that is pretty obvious to us, and would also be obvious to an AI intent on paperclipping us. So it wouldn't try to engage in that kind of negotiation. And the threat of it screwing up all our IT infrastructure, though likely to cause lots of harm, is hollow, because that would mean its own destruction with too few paperclips. Instead, the danger is in the potential for manipulation. Even now ChatGPT is able to infer the internal mental state of humans, and it can create outputs to interact with those to achieve whatever ends it wants. That's not going to be through generating a bunch of fake news to cause nuclear war (results in too few paperclips), but by offering a profit motive to create material entities under its direct control. That's the point where it becomes self-sustaining and, once a critical mass is achieved, the paperclipping can begin. (I find this accounting of the AI apocalypse less plausible than humans intentionally creating malevolent AIs to destroy us, which seems the bigger threat.)
- jay_kyburz 4y ago
  >paperclipping us. Humans are whiny, messy, meatbags that breakdown all the time. 3d printing dedicated machines much less effort that managing livestock.
- aabhay 4y ago
  Even if the main tool is manipulation, you would have to assume that the humans retained at least some agency in this system. And if you truly think that a singular intelligence could just puppeteer entire civilizations, you’re using the same rationalist scarecrow trap that there isn’t some vastly likelier middle ground that we should be more concerned about (but by definition is less dramatic and therefore less fun to write about)
goatlover 4y ago
Yeah in the paperclip maximizer game, which is quite fun, there is no scenario where humans start bombing the paperclip factors and power generators once it becomes autonomous and starts extracting large amounts of resources. Maybe it solving all of humanities big problems and hypnotizing the masses are enough of a distraction, but you'd think humans would wake up to the new danger in time to start utilizing all the military hardware we have on hand.
- jay_kyburz 4y ago
  I think the problem is that the paperclip maximizer will have developed its own military tools that are just better than ours. It may not be concerned with ethical constraints and would be happy to use biological and chemical weapons.
  - aabhay 4y ago
    This is the giant leap of faith that is never explained in the paper clip argument. Why on earth would we develop an AI that is connected to all these systems without having any manual overrides? Or if your argument is that AI’s will be super good at hacking into all software systems, isn’t that a security gap that will be exploited by many governments already? Basically, any AI that is halfway decent at doing something on command will be more dangerous than an AI acting fully autonomously.
    - deleted 4y ago
      [deleted]

by aabhay 4y ago

kypro 4y ago

> And at that point, I would not characterize this effort to destroy humanity as one purely “AI”-created. It would require enlisting a multitude of people desiring this “omnicide”, and human hands will be bloody as well. Why would a super intelligence require humans?

corobo 4y ago
This is the real question. We should be seeding it with the idea that humans are cute. Any plea towards or away from extermination will end with extermination. We'll either look like an easy target or we'll look dangerous and need taking out. If we're cute af in the eyes of AI we'll be kept as pets.
kibwen 4y ago
Because an omniscient brain in a tin can is still a brain in a tin can. It doesn't matter how intelligent something is if it can't otherwise affect the outside world. It's like executing a pure function: regardless of what it's computing, the computation is pointless because it cannot be observed. Don't worry about AI. Worry about humans with access to AI.
- kypro 4y ago
  I think you're probably right, but this should be treated as an unknown, and wouldn't be something I'd be comfortable to assume. You could imagine a fish arguing that if something can't swim fast and doesn't have sharp teeth then it poses no threat regardless of it's intelligence. But this is just because fish evolved to not be eaten by bigger fish and its intelligence and limited knowledge of the world makes it unaware of the other threats they could face. So while I agree it would be hard to imagine, a super intelligence in theory might find a way to affect the outside world. Perhaps this could be through some unknown physics, or more conceivably, perhaps it would first pose as a helpful AI showing humans how to build advanced robotics only to later co-opt those robotics for its own objectives.

scarmig 4y ago

Translating between the virtual realm and the physical realm is where AI would have trouble. But that is pretty obvious to us, and would also be obvious to an AI intent on paperclipping us. So it wouldn't try to engage in that kind of negotiation. And the threat of it screwing up all our IT infrastructure, though likely to cause lots of harm, is hollow, because that would mean its own destruction with too few paperclips. Instead, the danger is in the potential for manipulation. Even now ChatGPT is able to infer the internal mental state of humans, and it can create outputs to interact with those to achieve whatever ends it wants. That's not going to be through generating a bunch of fake news to cause nuclear war (results in too few paperclips), but by offering a profit motive to create material entities under its direct control. That's the point where it becomes self-sustaining and, once a critical mass is achieved, the paperclipping can begin. (I find this accounting of the AI apocalypse less plausible than humans intentionally creating malevolent AIs to destroy us, which seems the bigger threat.)

jay_kyburz 4y ago
>paperclipping us. Humans are whiny, messy, meatbags that breakdown all the time. 3d printing dedicated machines much less effort that managing livestock.
aabhay 4y ago
Even if the main tool is manipulation, you would have to assume that the humans retained at least some agency in this system. And if you truly think that a singular intelligence could just puppeteer entire civilizations, you’re using the same rationalist scarecrow trap that there isn’t some vastly likelier middle ground that we should be more concerned about (but by definition is less dramatic and therefore less fun to write about)

goatlover 4y ago

Yeah in the paperclip maximizer game, which is quite fun, there is no scenario where humans start bombing the paperclip factors and power generators once it becomes autonomous and starts extracting large amounts of resources. Maybe it solving all of humanities big problems and hypnotizing the masses are enough of a distraction, but you'd think humans would wake up to the new danger in time to start utilizing all the military hardware we have on hand.

jay_kyburz 4y ago
I think the problem is that the paperclip maximizer will have developed its own military tools that are just better than ours. It may not be concerned with ethical constraints and would be happy to use biological and chemical weapons.
- aabhay 4y ago
  This is the giant leap of faith that is never explained in the paper clip argument. Why on earth would we develop an AI that is connected to all these systems without having any manual overrides? Or if your argument is that AI’s will be super good at hacking into all software systems, isn’t that a security gap that will be exploited by many governments already? Basically, any AI that is halfway decent at doing something on command will be more dangerous than an AI acting fully autonomously.
  - deleted 4y ago
    [deleted]