6 ms·

It doesn't sound like you gave serious thought to the arguments. The AGI doesn't need to hack robots. It has superhuman persuasion, by definition; it can "hack"

It doesn't sound like you gave serious thought to the arguments. The AGI doesn't need to hack robots. It has superhuman persuasion, by definition; it can "hack" (enough of) the humans to achieve its goals.

CamperBob2 3y ago
Then it's just a matter of evolution in action. And while it doesn't take a God to start evolution, it would take a God to stop it.
- hollerith 3y ago
  You might be OK with suddenly dying along with all your friends and family, but I am not even if it is "evolution in action".
  - CamperBob2 3y ago
    Historically governments haven't needed computers or AI to do that. They've always managed just fine. Punched cards helped, though, I guess...
    - FeepingCreature 2y ago
      gestures at the human population graph wordlessly
      - CamperBob2 2y ago
        Agent Smith smiles mirthlessly
stale2002 3y ago
AI mind control abilities are also on the level of an extraordinary claim, that requires extraordinary evidence. It's on the level of "we better regulate wooden sticks so Voldemort doesn't use the imperious curse on us!". That's how I treat such claims. I treat them the same as someone literally talking about magic from Harry potter. There isn't nothing that would make me believe that. But it requires actual evidence and not thought experiments.
- the8472 3y ago
  Voldemort is fictional and so are bumbling wizard apprentices. Toy-level, not-yet-harmful AIs on the other hand are real. And so are efforts to make them more powerful. So the proposition that more powerful AIs will exist in the future is far more likely than an evil super wizard coming into existence. And I don't think literal 5-word-magic-incantation mind control is essential for an AI to be dangerous. More subtle or elaborate manipulation will be sufficient. Employees already have been duped into financial transactions by faked video calls with what they assumed to be their CEOs[0], and this didn't require superhuman general intelligence, only one single superhuman capability (realtime video manipulation). [0] https://edition.cnn.com/2024/02/04/asia/deepfake-cfo-scam-hong-kong-intl-hnk/index.html https://edition.cnn.com/2024/02/04/asia/deepfake-cfo-scam-ho...
  - stale2002 3y ago
    > Toy-level, not-yet-harmful AIs on the other hand are real. A computer that can cause harm is much different than the absurd claims that I am disagreeing with. The extraordinary claims that are equivalent to saying that the imperious curse exists would be the magic computers that create diamond nanobots and mind control humans. > that more powerful AIs will exist in the future Bad argument. Non safe Boxes exist in real life. People are trying to make more and better boxes. Therefore it is rational to be worried about Pandora's box being created and ending the world. That is the equivalent argument to what you just made. And it is absurd when talking about world ending box technology, even though Yes dangerous boxes exist, just as much as it is absurd to claim that world ending AI could exist.
    - the8472 3y ago
      Instead of gesturing at flawed analogies, let's return to the actual issue at hand. Do you think that agents more intelligent than humans are impossible or at least extremely unlikely to come into existence in the future? Or that such super-human intelligent agents are unlikely to have goals that are dangerous to humans? Or that they would be incapable of pursuing such goals? Also, it seems obvious that the standard of evidence that "AI could cause extinction" can't be observing an extinction level event, because at that point it would be too late. Considering that preventive measures would take time and safety margin, which level of evidence would be sufficient to motivate serious countermeasures?

CamperBob2 3y ago

Then it's just a matter of evolution in action. And while it doesn't take a God to start evolution, it would take a God to stop it.

hollerith 3y ago
You might be OK with suddenly dying along with all your friends and family, but I am not even if it is "evolution in action".
- CamperBob2 3y ago
  Historically governments haven't needed computers or AI to do that. They've always managed just fine. Punched cards helped, though, I guess...
  - FeepingCreature 2y ago
    gestures at the human population graph wordlessly
    - CamperBob2 2y ago
      Agent Smith smiles mirthlessly

stale2002 3y ago

AI mind control abilities are also on the level of an extraordinary claim, that requires extraordinary evidence. It's on the level of "we better regulate wooden sticks so Voldemort doesn't use the imperious curse on us!". That's how I treat such claims. I treat them the same as someone literally talking about magic from Harry potter. There isn't nothing that would make me believe that. But it requires actual evidence and not thought experiments.

the8472 3y ago
Voldemort is fictional and so are bumbling wizard apprentices. Toy-level, not-yet-harmful AIs on the other hand are real. And so are efforts to make them more powerful. So the proposition that more powerful AIs will exist in the future is far more likely than an evil super wizard coming into existence. And I don't think literal 5-word-magic-incantation mind control is essential for an AI to be dangerous. More subtle or elaborate manipulation will be sufficient. Employees already have been duped into financial transactions by faked video calls with what they assumed to be their CEOs[0], and this didn't require superhuman general intelligence, only one single superhuman capability (realtime video manipulation). [0] https://edition.cnn.com/2024/02/04/asia/deepfake-cfo-scam-hong-kong-intl-hnk/index.html https://edition.cnn.com/2024/02/04/asia/deepfake-cfo-scam-ho...
- stale2002 3y ago
  > Toy-level, not-yet-harmful AIs on the other hand are real. A computer that can cause harm is much different than the absurd claims that I am disagreeing with. The extraordinary claims that are equivalent to saying that the imperious curse exists would be the magic computers that create diamond nanobots and mind control humans. > that more powerful AIs will exist in the future Bad argument. Non safe Boxes exist in real life. People are trying to make more and better boxes. Therefore it is rational to be worried about Pandora's box being created and ending the world. That is the equivalent argument to what you just made. And it is absurd when talking about world ending box technology, even though Yes dangerous boxes exist, just as much as it is absurd to claim that world ending AI could exist.
  - the8472 3y ago
    Instead of gesturing at flawed analogies, let's return to the actual issue at hand. Do you think that agents more intelligent than humans are impossible or at least extremely unlikely to come into existence in the future? Or that such super-human intelligent agents are unlikely to have goals that are dangerous to humans? Or that they would be incapable of pursuing such goals? Also, it seems obvious that the standard of evidence that "AI could cause extinction" can't be observing an extinction level event, because at that point it would be too late. Considering that preventive measures would take time and safety margin, which level of evidence would be sufficient to motivate serious countermeasures?