3 ms·

Can anybody explain, specifically, what that 20% risk looks like? The most specific I ever see is "an adversarial AI will become sentient and wipe out humanity"

Can anybody explain, specifically, what that 20% risk looks like? The most specific I ever see is "an adversarial AI will become sentient and wipe out humanity". It sounds like as much snake oil as the people pushing AI itself.

EGreg 2y ago
It doesnt need to become sentient to cause great disruption. 1. Bot swarms will simply disrupt everything about the Internet as we know it. Most people ALREADY barely scritinize chats and articles, so bots can EASILY produce those at scale to push opinion in any direction, or just sell shyt 2. Botswarms will outplay adversarrial games vs humans for karma / reputation points, as well as launch coordinated attacks on opponents organically trying to stop whatever viewpoint is being gradually pushed or sold, until they give up or are totally reputationally discredited 3. People start to PREFER bots to humans, just as they PREFER google maps to asking for directions etc. At that point most humans would be surrounded by 100-1000 bots and have no way of affecting other humans. 4. Physical world, cameras capturing all the info and cross correlating where you are. Maybe slaughterbots are mass-produced. Who knows. When the costs come down and scale goes up, it doesnt matter about AGI, the entire society is disrupted permanently. And that’s what AI is on track to so. It’s far easier to continually create a mess than to continually clean it up.
- LargeWu 2y ago
  You see, the jump from "Bots take over all the karma points on social media sites" to "Slaughterbots" is a pretty wide chasm I'm having trouble getting over mentally. This is why I can't take such predictions seriously.
  - EGreg 2y ago
    Okay. So remove point 4 and it’s still very dystopian… Not to mention point 4 contains things that have been in place already for over a decade. It’s not even a prediction: https://magarshak.com/blog/?p=169 https://magarshak.com/blog/?p=169 But sure, take the one tiny thing you can caricature and ignore the rest. That’s one step up from strawman, I guess
    - LargeWu 2y ago
      Your original post didn't just posit a "dystopian" future. "...even the overwhelming majority AI experts coming out and saying there is at least 20% chance of a global catastrophe or even risk of extinction". You're the one bringing up the prospect of extinction. Extinction! And a 20% chance at that. So no, I don't think it's unreasonable to ask about how we arrive at that outcome. Because there's a massive distinction between "dystopian" and "extinction".
      - EGreg 2y ago
        No I’m not “the one” bringing it up, the experts and the people asking them and publishing their words are, and you seize on the most hard-to-substantiate claims first, and ignore the rest. Great debate technique for realtime debates, but this is HN and I can reply to focus the point. My main concern for the next 5 years is that the Internet is going to become a dark forest where you can’t trust anything, it will be impossible to discern fake stuff, and even if it was, the botswarms will gang up to take care of any dissent. That alone is extremely plausible and scary. Every single institution we have relies on the inefficiency of an attacker. Let alone swarms of attackers that any member of the institution can run instead of themselves, and can be subverted to bring about ANY goal, by who knows behind the scenes.

EGreg 2y ago

It doesnt need to become sentient to cause great disruption. 1. Bot swarms will simply disrupt everything about the Internet as we know it. Most people ALREADY barely scritinize chats and articles, so bots can EASILY produce those at scale to push opinion in any direction, or just sell shyt 2. Botswarms will outplay adversarrial games vs humans for karma / reputation points, as well as launch coordinated attacks on opponents organically trying to stop whatever viewpoint is being gradually pushed or sold, until they give up or are totally reputationally discredited 3. People start to PREFER bots to humans, just as they PREFER google maps to asking for directions etc. At that point most humans would be surrounded by 100-1000 bots and have no way of affecting other humans. 4. Physical world, cameras capturing all the info and cross correlating where you are. Maybe slaughterbots are mass-produced. Who knows. When the costs come down and scale goes up, it doesnt matter about AGI, the entire society is disrupted permanently. And that’s what AI is on track to so. It’s far easier to continually create a mess than to continually clean it up.

LargeWu 2y ago
You see, the jump from "Bots take over all the karma points on social media sites" to "Slaughterbots" is a pretty wide chasm I'm having trouble getting over mentally. This is why I can't take such predictions seriously.
- EGreg 2y ago
  Okay. So remove point 4 and it’s still very dystopian… Not to mention point 4 contains things that have been in place already for over a decade. It’s not even a prediction: https://magarshak.com/blog/?p=169 https://magarshak.com/blog/?p=169 But sure, take the one tiny thing you can caricature and ignore the rest. That’s one step up from strawman, I guess
  - LargeWu 2y ago
    Your original post didn't just posit a "dystopian" future. "...even the overwhelming majority AI experts coming out and saying there is at least 20% chance of a global catastrophe or even risk of extinction". You're the one bringing up the prospect of extinction. Extinction! And a 20% chance at that. So no, I don't think it's unreasonable to ask about how we arrive at that outcome. Because there's a massive distinction between "dystopian" and "extinction".
    - EGreg 2y ago
      No I’m not “the one” bringing it up, the experts and the people asking them and publishing their words are, and you seize on the most hard-to-substantiate claims first, and ignore the rest. Great debate technique for realtime debates, but this is HN and I can reply to focus the point. My main concern for the next 5 years is that the Internet is going to become a dark forest where you can’t trust anything, it will be impossible to discern fake stuff, and even if it was, the botswarms will gang up to take care of any dissent. That alone is extremely plausible and scary. Every single institution we have relies on the inefficiency of an attacker. Let alone swarms of attackers that any member of the institution can run instead of themselves, and can be subverted to bring about ANY goal, by who knows behind the scenes.