4 ms·

No goals are intrinsically stupid. The only reason that you might think some particular goal is stupid is that it goes against your goals. You could conceive o

No goals are intrinsically stupid. The only reason that you might think some particular goal is stupid is that it goes against your goals. You could conceive of a super intelligent AI that came into existence with the goal of terminating itself. That would be an "stupid goal" from our perspective, since we have the goal of self-preservation really ingrained in our brains. But for a being that self-termination is the absolute best thing ever, it's not stupid. It makes perfect sense, since, well, that it's goal. It doesn't care about self-preservation, it doesn't care about becoming more intelligent/rich/powerful, other than as an instrumental goal to help achieve self-termination, if it's not able to do so in its current state. And most importantly, no amount of getting more intelligent would change this fundamental goal, just as humans getting more intelligent has not overridden our fundamental goals of "breathe, feed, have sex". It may have given us other goals as well, but those are very much still there.

DennisP 4y ago
Not only that, but we've managed to subvert our reward functions in exactly the way AI safety people fear. Evolution tried to get us to reproduce as much as possible, and we came up with ways to get the reward without producing offspring.
- simple-thoughts 4y ago
  That’s a pretty naive understanding of sex. In human history sex has often played as important a role in bonding, community, and social hierarchy as it has in child production. These things are beneficial for survival. It’s very common for evolution to result in redirected drives.
  - DennisP 4y ago
    > It’s very common for evolution to result in redirected drives. Yes, that's pretty much the point. And most likely, humans will have a pretty naive understanding of whatever motivates a superintelligent AI.
simple-thoughts 4y ago
I’m not sure what you mean by “stupid” but it’s definitely not the colloquial meaning. A human seeking suicide is not stupid. In fact humans often do seek suicide because humans are playing social games where the maximizing move is to commit suicide. Supporting these people requires changing their context so that they have a wider and better variety of options available. This is just a specific example of why I reject the orthogonality thesis. You change the context, you educate the agent, you change the goals of the agent. I do not agree that humans only chase “breathe feed sex” and while I do believe many stupid behaviors do come from evolutionary history, It’s plainly obvious that education, training, and genes play a role in self restraint and goal redirection.
- rafabulsing 4y ago
  What do you mean by stupid? I used that word only because you used it first. And I agree that humans do not only chase "breath feed sex", as I explicitly said that in my comment. We have other goals as well. But those are very much still there.

DennisP 4y ago

Not only that, but we've managed to subvert our reward functions in exactly the way AI safety people fear. Evolution tried to get us to reproduce as much as possible, and we came up with ways to get the reward without producing offspring.

simple-thoughts 4y ago
That’s a pretty naive understanding of sex. In human history sex has often played as important a role in bonding, community, and social hierarchy as it has in child production. These things are beneficial for survival. It’s very common for evolution to result in redirected drives.
- DennisP 4y ago
  > It’s very common for evolution to result in redirected drives. Yes, that's pretty much the point. And most likely, humans will have a pretty naive understanding of whatever motivates a superintelligent AI.

simple-thoughts 4y ago

I’m not sure what you mean by “stupid” but it’s definitely not the colloquial meaning. A human seeking suicide is not stupid. In fact humans often do seek suicide because humans are playing social games where the maximizing move is to commit suicide. Supporting these people requires changing their context so that they have a wider and better variety of options available. This is just a specific example of why I reject the orthogonality thesis. You change the context, you educate the agent, you change the goals of the agent. I do not agree that humans only chase “breathe feed sex” and while I do believe many stupid behaviors do come from evolutionary history, It’s plainly obvious that education, training, and genes play a role in self restraint and goal redirection.

rafabulsing 4y ago
What do you mean by stupid? I used that word only because you used it first. And I agree that humans do not only chase "breath feed sex", as I explicitly said that in my comment. We have other goals as well. But those are very much still there.