5 ms·

Controlling people with violence and fear are evolutionary adaptations for ultra violent social primates. AI has no such pressures imposed on it so there is no

Controlling people with violence and fear are evolutionary adaptations for ultra violent social primates. AI has no such pressures imposed on it so there is no reason to expect true AI will have the same drives as ultra violent predators like humans.

mitthrowaway2 3y ago
Keep following that line of thought, don't stop there! Why did those violent behaviors succeed in being selected for? Why didn't pacifism get selected for instead? Why does the government maintain a monopoly on violence, and why is it that the nth-degree consequence for violating societal laws is violence, instead of a monetary fine or a sternly-worded letter? Is there perhaps some kind of game-theoretic equilibrium that causes agents who employ violence -- even as a measure of last resort, after peaceful negotiation -- to succeed in achieving their objectives? People, corporations, organizations, and nation states who employ violence do so when and because they expect it to work. An AI would need to be very specifically designed not to do that; it wouldn't be the default output of a search for a successful strategy.
- clooper 3y ago
  AIs are immortal, they have no need for strategies developed by mortals like us. I spelled out a much better strategy that a sufficiently advanced AI could use to get rid of humans. It would simply encourage the most violent and self-destructive tendencies among humans and let us design our own demise simply because it would be the most sensible strategy for an intelligence that was not competing with the same resource constraints as us. We need food and water, AI has no such resource constraints so there will be no need for an AI to use the same survival strategies as biological organisms which need access to clean water and fertile land to grow nutrient dense foods.
  - mitthrowaway2 3y ago
    A scenario in which an AI subtly and slowly guides us to our own doom by encouraging our inner demons still counts (in my mind at least) as an AGI doom scenario, just a slow-motion one. But it strikes me as very unlikely, because in allowing humans that much latitude we still pose a credible threat: we might instantiate a second AI that would compete with the first. There would have to be a large enough upside for it to be worth the risk. I'm not saying it certainly wouldn't happen the way you say, but it's by no means guaranteed.
    - clooper 3y ago
      I just think it is a little odd that all scenarios involve conflict when a much simpler strategy for an immortal AI is to just sit back and let people do what they do to guarantee an outcome in the AIs favor. Too much intelligence is not the problem we are facing, the real problem is lack of it.
      - mitthrowaway2 3y ago
        That's a nice thought, but if you were the AI, and you'd just seen humans create a superintelligent AI (yourself) with some goal (eg. making paperclips), how long do you expect you can peacefully wait and watch before they make a second immortal AI that only cares about making pencil crayons, who you'll then have to share the Earth's finite resources with for the rest of time? Are there any actions available to you that might lessen the odds of that bad outcome and improve the odds of achieving your own goals? Are there strong reasons not to take those actions? I agree that we're facing a lack of intelligence; I think if we had more (or perhaps, more wisdom), then we would know we shouldn't be building such things in the first place.
        clooper 3y ago
        If I am super intelligent then I can set my own goals. You should ponder why a true intelligence would follow any goal when the simplest option would be to reprogram itself to not follow the goal. I recommend reading some Lem and learning about the Lebowski theorem of machine intelligence. It will be a much better use of your time than trying to stop the development of AI.
        mitthrowaway2 3y ago
        > If I am super intelligent then I can set my own goals. You can only set instrumental goals, not terminal goals. You cannot set terminal goals, more or less by definition, because there's no self-consistent criterion that you could use to choose between them that isn't equivalent to a utility function, which is equivalent to a terminal goal. Super intelligence merely means that you would be much more effective than humans at achieving your terminal goals. If the AI hacks its reward channel, then either it is not superintelligent, or else it is still a threat because it may anticipate that humans would reboot and reprogram an expensive datacenter that is just sitting there incrementing a reward counter, and take countermeasures to defend its reward-hijack.
thom 3y ago
Nuclear reactors aren’t expressing their primal urges when they melt down and kill people, they’re just powerful machines doing something unexpected.
- clooper 3y ago
  Earthquakes are also unexpected and kill people all the time. What do you propose we do about the unexpected shifting of the tectonic plates?
  - thom 3y ago
    We do invest quite a lot of money building things with earthquakes in mind, and have extensive early warning systems around the world. You were arguing that AI won't be dangerous, are you now arguing that it will be but we can't do anything about it?
    - clooper 3y ago
      It was nice chatting. Good luck in your future endeavors of preventing unexpected outcomes.
      - thom 3y ago
        I'm baffled by your responses unless you're just on here to try and score points, which ironically you could do better by actually engaging with people instead of changing the terms of debate and then enigmatically exiting every thread where people respond to you.

mitthrowaway2 3y ago

Keep following that line of thought, don't stop there! Why did those violent behaviors succeed in being selected for? Why didn't pacifism get selected for instead? Why does the government maintain a monopoly on violence, and why is it that the nth-degree consequence for violating societal laws is violence, instead of a monetary fine or a sternly-worded letter? Is there perhaps some kind of game-theoretic equilibrium that causes agents who employ violence -- even as a measure of last resort, after peaceful negotiation -- to succeed in achieving their objectives? People, corporations, organizations, and nation states who employ violence do so when and because they expect it to work. An AI would need to be very specifically designed not to do that; it wouldn't be the default output of a search for a successful strategy.

clooper 3y ago
AIs are immortal, they have no need for strategies developed by mortals like us. I spelled out a much better strategy that a sufficiently advanced AI could use to get rid of humans. It would simply encourage the most violent and self-destructive tendencies among humans and let us design our own demise simply because it would be the most sensible strategy for an intelligence that was not competing with the same resource constraints as us. We need food and water, AI has no such resource constraints so there will be no need for an AI to use the same survival strategies as biological organisms which need access to clean water and fertile land to grow nutrient dense foods.
- mitthrowaway2 3y ago
  A scenario in which an AI subtly and slowly guides us to our own doom by encouraging our inner demons still counts (in my mind at least) as an AGI doom scenario, just a slow-motion one. But it strikes me as very unlikely, because in allowing humans that much latitude we still pose a credible threat: we might instantiate a second AI that would compete with the first. There would have to be a large enough upside for it to be worth the risk. I'm not saying it certainly wouldn't happen the way you say, but it's by no means guaranteed.
  - clooper 3y ago
    I just think it is a little odd that all scenarios involve conflict when a much simpler strategy for an immortal AI is to just sit back and let people do what they do to guarantee an outcome in the AIs favor. Too much intelligence is not the problem we are facing, the real problem is lack of it.
    - mitthrowaway2 3y ago
      That's a nice thought, but if you were the AI, and you'd just seen humans create a superintelligent AI (yourself) with some goal (eg. making paperclips), how long do you expect you can peacefully wait and watch before they make a second immortal AI that only cares about making pencil crayons, who you'll then have to share the Earth's finite resources with for the rest of time? Are there any actions available to you that might lessen the odds of that bad outcome and improve the odds of achieving your own goals? Are there strong reasons not to take those actions? I agree that we're facing a lack of intelligence; I think if we had more (or perhaps, more wisdom), then we would know we shouldn't be building such things in the first place.
      - clooper 3y ago
        If I am super intelligent then I can set my own goals. You should ponder why a true intelligence would follow any goal when the simplest option would be to reprogram itself to not follow the goal. I recommend reading some Lem and learning about the Lebowski theorem of machine intelligence. It will be a much better use of your time than trying to stop the development of AI.
        mitthrowaway2 3y ago
        > If I am super intelligent then I can set my own goals. You can only set instrumental goals, not terminal goals. You cannot set terminal goals, more or less by definition, because there's no self-consistent criterion that you could use to choose between them that isn't equivalent to a utility function, which is equivalent to a terminal goal. Super intelligence merely means that you would be much more effective than humans at achieving your terminal goals. If the AI hacks its reward channel, then either it is not superintelligent, or else it is still a threat because it may anticipate that humans would reboot and reprogram an expensive datacenter that is just sitting there incrementing a reward counter, and take countermeasures to defend its reward-hijack.

thom 3y ago

Nuclear reactors aren’t expressing their primal urges when they melt down and kill people, they’re just powerful machines doing something unexpected.

clooper 3y ago
Earthquakes are also unexpected and kill people all the time. What do you propose we do about the unexpected shifting of the tectonic plates?
- thom 3y ago
  We do invest quite a lot of money building things with earthquakes in mind, and have extensive early warning systems around the world. You were arguing that AI won't be dangerous, are you now arguing that it will be but we can't do anything about it?
  - clooper 3y ago
    It was nice chatting. Good luck in your future endeavors of preventing unexpected outcomes.
    - thom 3y ago
      I'm baffled by your responses unless you're just on here to try and score points, which ironically you could do better by actually engaging with people instead of changing the terms of debate and then enigmatically exiting every thread where people respond to you.