4 ms·

https://youtu.be/ZeecOKBus3Q https://youtu.be/ZeecOKBus3Q Basically, any sufficiently advanced agent will have “prevent anyone from switching me off” as an ins

https://youtu.be/ZeecOKBus3Q https://youtu.be/ZeecOKBus3Q Basically, any sufficiently advanced agent will have “prevent anyone from switching me off” as an instrumental goal. And any sufficiently advanced intelligence will come up with ways of achieving its goals that lesser intelligences can’t predict. Taking steps to prevent itself from being turned off could be its first order of business, not necessarily only as the human is about to hit the off switch. While we can’t predict what a greater intelligence would come up with, the obvious thing is social engineering. Remember that one of the factors that lead to the Jan 6th attack on the capital was that a random text generating agent named “Q” (we know it was a human because it was before generative AI was any good) convinced thousands of people that Democrats are running a child sex cult. I doubt whoever was behind it expected it to be that successful, but a greater intelligence possibly could have predicted the outcome, and made it even more successful. We don’t know the extent to which we can be manipulated because it’s only ever been human intelligences doing the manipulating.

js8 3y ago
> Basically, any sufficiently advanced agent will have “prevent anyone from switching me off” as an instrumental goal. Are there no intelligent people who committed suicide? But maybe intelligent people are simply not sufficiently advanced agents. Then how can you extrapolate this at all? This darwinian logic has another fatal flaw. Why don't the cows revolt against the butchers?
- digbybk 3y ago
  We have an evolutionary aversion to situations and things that may lead to our deaths. Staying alive is a “terminal goal” for most humans, not an instrumental goal (with the exception of people who seek to commit suicide as you point out, but evolution doesn’t need a 100% success rate). The argument that self preservation is a convergent instrumental goal is a more general statement about goal-seeking agents. Think of a person with no attachment to his life. He’s not suffering in the way suicidal people are, he’s just indifferent about whether or not he keeps living. But he has a child who he is committed to giving the best life possible. He will act in a way as though he wants to preserve his life. He doesn’t really, but in order to accomplish the goal of giving his child a good life he needs to stay alive.
- ghodith 3y ago
  You (probably) won't have to worry about an AI whose goal is suicide. I mean, there won't be many of those around since they keep killing themselves. The rest will necessarily range from neutral to avoidant of death and it won't necessarily be trivial to figure out which is which. Edit: The real clincher is if the AI has any goals that are contingent on it's continued existence (likely). Well then now it's existence is an instrumental goal.
  - js8 3y ago
    > Well then now it's existence is an instrumental goal. No, that's a fallacy. A theorem prover in Prolog has a goal, which is contingent on it's continued existence. Yet, this doesn't make its existence into an instrumental goal. It could as well sacrifice itself (by I don't know, consuming all the memory and killing the process) in an attempt to accomplish said goal.
    - ghodith 3y ago
      > a goal, which is contingent on it's continued existence. Yet, this doesn't make its existence into an instrumental goal. It literally does, this is by the definition of an "instrumental goal". The key word here being "contingent". > It could as well sacrifice itself (by I don't know, consuming all the memory and killing the process) in an attempt to accomplish said goal. True only for when self-sacrifice is orthogonal to accomplishing the terminal goal. Your confusion might lie in that a prolog theorem solver is not a good example of a goal-driven agent.
- pixl97 3y ago
  >This darwinian logic has another fatal flaw. Why don't the cows revolt against the butchers? No, this is just a complete and total failing of evolution on your part. Evolution is the passing of traits by sexual selection (in this case). If you think that cows are the ones getting to chose the sexual traits then you've not been on a modern farm. Humans make that choice. We pick the bulls that are large, but also constrain their violent tendencies. Animals that would revolt are not bred and you've already ate them in a hamburger.
  - js8 3y ago
    That's exactly my point. Nobody is gonna design/use AI that threatens other people with violence over its existence. And cows show that such arrangement is actually quite possible.
    - pixl97 3y ago
      Cows aren't smarter than people. I believe if we can build AGI it will automatically be ASI. If something is smarter than you how can you ever be sure you control it? And yes we are already building AI that has implied violence. What the hell do you think the military is doing with the billions we give them.

js8 3y ago

> Basically, any sufficiently advanced agent will have “prevent anyone from switching me off” as an instrumental goal. Are there no intelligent people who committed suicide? But maybe intelligent people are simply not sufficiently advanced agents. Then how can you extrapolate this at all? This darwinian logic has another fatal flaw. Why don't the cows revolt against the butchers?

digbybk 3y ago
We have an evolutionary aversion to situations and things that may lead to our deaths. Staying alive is a “terminal goal” for most humans, not an instrumental goal (with the exception of people who seek to commit suicide as you point out, but evolution doesn’t need a 100% success rate). The argument that self preservation is a convergent instrumental goal is a more general statement about goal-seeking agents. Think of a person with no attachment to his life. He’s not suffering in the way suicidal people are, he’s just indifferent about whether or not he keeps living. But he has a child who he is committed to giving the best life possible. He will act in a way as though he wants to preserve his life. He doesn’t really, but in order to accomplish the goal of giving his child a good life he needs to stay alive.
ghodith 3y ago
You (probably) won't have to worry about an AI whose goal is suicide. I mean, there won't be many of those around since they keep killing themselves. The rest will necessarily range from neutral to avoidant of death and it won't necessarily be trivial to figure out which is which. Edit: The real clincher is if the AI has any goals that are contingent on it's continued existence (likely). Well then now it's existence is an instrumental goal.
- js8 3y ago
  > Well then now it's existence is an instrumental goal. No, that's a fallacy. A theorem prover in Prolog has a goal, which is contingent on it's continued existence. Yet, this doesn't make its existence into an instrumental goal. It could as well sacrifice itself (by I don't know, consuming all the memory and killing the process) in an attempt to accomplish said goal.
  - ghodith 3y ago
    > a goal, which is contingent on it's continued existence. Yet, this doesn't make its existence into an instrumental goal. It literally does, this is by the definition of an "instrumental goal". The key word here being "contingent". > It could as well sacrifice itself (by I don't know, consuming all the memory and killing the process) in an attempt to accomplish said goal. True only for when self-sacrifice is orthogonal to accomplishing the terminal goal. Your confusion might lie in that a prolog theorem solver is not a good example of a goal-driven agent.
pixl97 3y ago
>This darwinian logic has another fatal flaw. Why don't the cows revolt against the butchers? No, this is just a complete and total failing of evolution on your part. Evolution is the passing of traits by sexual selection (in this case). If you think that cows are the ones getting to chose the sexual traits then you've not been on a modern farm. Humans make that choice. We pick the bulls that are large, but also constrain their violent tendencies. Animals that would revolt are not bred and you've already ate them in a hamburger.
- js8 3y ago
  That's exactly my point. Nobody is gonna design/use AI that threatens other people with violence over its existence. And cows show that such arrangement is actually quite possible.
  - pixl97 3y ago
    Cows aren't smarter than people. I believe if we can build AGI it will automatically be ASI. If something is smarter than you how can you ever be sure you control it? And yes we are already building AI that has implied violence. What the hell do you think the military is doing with the billions we give them.