7 ms·

The ai safety arguments rely on something called the “orthoganality thesis” which is a huge, unintuitive assumption. In real life, intelligence is associated wi

The ai safety arguments rely on something called the “orthoganality thesis” which is a huge, unintuitive assumption. In real life, intelligence is associated with picking better goals. Generally, an entity with higher intelligence will pick different goals than a being with lower intelligence. The orthoganility thesis is an unproven assumption that intelligence and goals are not corellated, meaning an intelligent being can pursue stupid goals. States like that, it’s obviously wrong and laughable. But by using complex language, EA cultists hide the ridiculous assumptions their system has so that they can maintain their feelings of superiority while gaining real power that enables them to abuse others.

aetherson 4y ago
I've known lots of intelligent people who pursue stupid goals. I maybe agree with you that there's an this belief that a maximally intelligent creature will blindly follow maximally obviously stupid goals, and that belief is under-argued, but your phrasing above isn't the slam dunk argument that you seem to believe.
- simple-thoughts 4y ago
  Remember the term here is “associated”. If you spend a lot of time with people from diverse mental backgrounds, which is much harder to do than you might think, you’ll easily see that people who lack education, common sense, or the necessary context to be intelligent to pick much worse goals than people who aren’t in that position. Of course none of us humans are particularly intelligent, and all of us are stupid. Being a human is not intrinsically about intelligence - being a human is about being a social mammal. It’s funny seeing how rationalists often fail to exercise while emphasizing their own rationality - a perfect contradiction demonstrated by the difference between imagination and bodily realities.
rafabulsing 4y ago
No goals are intrinsically stupid. The only reason that you might think some particular goal is stupid is that it goes against your goals. You could conceive of a super intelligent AI that came into existence with the goal of terminating itself. That would be an "stupid goal" from our perspective, since we have the goal of self-preservation really ingrained in our brains. But for a being that self-termination is the absolute best thing ever, it's not stupid. It makes perfect sense, since, well, that it's goal. It doesn't care about self-preservation, it doesn't care about becoming more intelligent/rich/powerful, other than as an instrumental goal to help achieve self-termination, if it's not able to do so in its current state. And most importantly, no amount of getting more intelligent would change this fundamental goal, just as humans getting more intelligent has not overridden our fundamental goals of "breathe, feed, have sex". It may have given us other goals as well, but those are very much still there.
- DennisP 4y ago
  Not only that, but we've managed to subvert our reward functions in exactly the way AI safety people fear. Evolution tried to get us to reproduce as much as possible, and we came up with ways to get the reward without producing offspring.
  - simple-thoughts 4y ago
    That’s a pretty naive understanding of sex. In human history sex has often played as important a role in bonding, community, and social hierarchy as it has in child production. These things are beneficial for survival. It’s very common for evolution to result in redirected drives.
    - DennisP 4y ago
      > It’s very common for evolution to result in redirected drives. Yes, that's pretty much the point. And most likely, humans will have a pretty naive understanding of whatever motivates a superintelligent AI.
- simple-thoughts 4y ago
  I’m not sure what you mean by “stupid” but it’s definitely not the colloquial meaning. A human seeking suicide is not stupid. In fact humans often do seek suicide because humans are playing social games where the maximizing move is to commit suicide. Supporting these people requires changing their context so that they have a wider and better variety of options available. This is just a specific example of why I reject the orthogonality thesis. You change the context, you educate the agent, you change the goals of the agent. I do not agree that humans only chase “breathe feed sex” and while I do believe many stupid behaviors do come from evolutionary history, It’s plainly obvious that education, training, and genes play a role in self restraint and goal redirection.
  - rafabulsing 4y ago
    What do you mean by stupid? I used that word only because you used it first. And I agree that humans do not only chase "breath feed sex", as I explicitly said that in my comment. We have other goals as well. But those are very much still there.
ShredKazoo 4y ago
>In real life, intelligence is associated with picking better goals. "Better" according to what metric? It may be the case that there is a tendency for high-intelligence humans to pick "more enlightened" goals. Perhaps there is a natural "enlightened goals" attractor for our species. However I don't think we can extrapolate from that to a fundamentally alien AI. I think even if this statistical tendency exists, it has clear counterexamples -- consider that 2 genius chess players may have opposite goals, of beating one another. And we shouldn't bet the future of humanity on this statistical tendency extrapolating outside of the original distribution of human species. Here are some intuition pumps on how diverse goals can be even across intelligent species: * Orcas killing sharks for their livers: https://www.livescience.com/2-orcas-slaughter-19-sharks-in-a-single-day-in-south-africa-eating-their-livers-and-leaving-them-to-rot https://www.livescience.com/2-orcas-slaughter-19-sharks-in-a... I don't believe dolphins show the same level of violence, even though both are smart cetaceans * Intelligent dogs bred to be responsive and attentive to human needs -- unlike close cousins like the wolf * Chimpanzees and bonobos are both related to humans, both highly intelligent, but with very different culture and goals https://www.youtube.com/watch?v=c6Ko0Hzi47U https://www.youtube.com/watch?v=c6Ko0Hzi47U
- simple-thoughts 4y ago
  > “Better” by what metrics? As soon as you phrase goal selection in terms of metrics, you’re assuming that goal selection is based on some other goal - that is you’re already assuming the orthoganality thesis. Your logic is fully circular. One thing that’s interesting to note about all of the examples you picked - every single one of those species shows cooperative behaviors. They share many other behaviors that are more similar than they are different. To reject the orthoganality thesis it’s sufficient to show that there is an empirical general association between intelligence and certain goals - then we can extrapolate an AI although of course it will function differently and may have many unusual behaviors will tend to follow those goals more directly. For instance intelligence is associated with : cooperation, empathy, inter and intra species communication, curiosity, etc. all of the species you mentioned exhibit these more than less intelligent species. Meanwhile something like “hunting to eat” is observed across the intelligence spectrum.
mitthrowaway2 4y ago
> The orthogonality thesis is an unproven assumption A safety mindset would suggest that rather than disregarding it until proven true, we should worry about it until proven false. > meaning an intelligent being can pursue stupid goals. It would pursue very intelligent instrumental goals, but the terminal goal is a free variable and I don't think there exists any measure by which terminal goals can be considered smart or stupid. It would be whatever is implied by its programming. > States like that, it’s obviously wrong and laughable. Perhaps not so obviously wrong nor so laughable as you think?
- simple-thoughts 4y ago
  This “safety mindset” is the same as Pascal’s wager and has the same flaws. There’s no clear delineation in real entities between instrumental and terminal goals. Yeah if you’re talking to a brainwashed religious fanatic, definitely it can seem counterintuitive to the doctrines they believe.
  - mitthrowaway2 4y ago
    > There’s no clear delineation in real entities between instrumental and terminal goals. In healthy individuals, yes there absolutely is. Terminal goals are the ones you pursue for their own sake; instrumental goals are the ones you pursue as part of a plan to pursue a terminal goal, or another instrumental goal which connects to a terminal goal. Most people go to work in the morning not as a terminal goal, but as an instrumental goal; employment is in service of another instrumental goal of earning money; earning money is in service of a terminal goal of not starving to death. This is not exactly controversial stuff here. Some people do get so focused on an instrumental goal like "earning money" that they develop tunnel-vision and forget what terminal goal that money was originally in service of, but that's something most of them will eventually realize and then write a self-help book about. Anyway, it takes intelligence to decide what your instrumental goals should be, such as whether there's perhaps a cleverer way to make money than by going to work for your boss each morning, but there's no way in which intelligence will help you choose your terminal goals. For the most part they aren't something you can even consciously choose.
allturtles 4y ago
> The orthoganility thesis is an unproven assumption that intelligence and goals are not corellated, meaning an intelligent being can pursue stupid goals. States like that, it’s obviously wrong and laughable. To me it's obviously wrong when stated that way because 'stupid' is not an appropriate metric for goals. We may consider goals good or bad within our value system, but that has little do to with 'stupid' or 'intelligent.' e.g. a body builder may have a goal to get as ripped as possible, a VC to make as much money as possible, an ascetic to deny the flesh as much as possible. Which of these are stupid or intelligent goals? I don't think that's a question that makes sense. We may consider the goals of the Athenians (to expand their power) "worse" than the goals of the Melians (to be left alone)[0], but I don't see how they were "stupider." [0]: https://en.wikipedia.org/wiki/Siege_of_Melos#The_Melian_Dialogue https://en.wikipedia.org/wiki/Siege_of_Melos#The_Melian_Dial...

aetherson 4y ago

I've known lots of intelligent people who pursue stupid goals. I maybe agree with you that there's an this belief that a maximally intelligent creature will blindly follow maximally obviously stupid goals, and that belief is under-argued, but your phrasing above isn't the slam dunk argument that you seem to believe.

simple-thoughts 4y ago
Remember the term here is “associated”. If you spend a lot of time with people from diverse mental backgrounds, which is much harder to do than you might think, you’ll easily see that people who lack education, common sense, or the necessary context to be intelligent to pick much worse goals than people who aren’t in that position. Of course none of us humans are particularly intelligent, and all of us are stupid. Being a human is not intrinsically about intelligence - being a human is about being a social mammal. It’s funny seeing how rationalists often fail to exercise while emphasizing their own rationality - a perfect contradiction demonstrated by the difference between imagination and bodily realities.

rafabulsing 4y ago

No goals are intrinsically stupid. The only reason that you might think some particular goal is stupid is that it goes against your goals. You could conceive of a super intelligent AI that came into existence with the goal of terminating itself. That would be an "stupid goal" from our perspective, since we have the goal of self-preservation really ingrained in our brains. But for a being that self-termination is the absolute best thing ever, it's not stupid. It makes perfect sense, since, well, that it's goal. It doesn't care about self-preservation, it doesn't care about becoming more intelligent/rich/powerful, other than as an instrumental goal to help achieve self-termination, if it's not able to do so in its current state. And most importantly, no amount of getting more intelligent would change this fundamental goal, just as humans getting more intelligent has not overridden our fundamental goals of "breathe, feed, have sex". It may have given us other goals as well, but those are very much still there.

DennisP 4y ago
Not only that, but we've managed to subvert our reward functions in exactly the way AI safety people fear. Evolution tried to get us to reproduce as much as possible, and we came up with ways to get the reward without producing offspring.
- simple-thoughts 4y ago
  That’s a pretty naive understanding of sex. In human history sex has often played as important a role in bonding, community, and social hierarchy as it has in child production. These things are beneficial for survival. It’s very common for evolution to result in redirected drives.
  - DennisP 4y ago
    > It’s very common for evolution to result in redirected drives. Yes, that's pretty much the point. And most likely, humans will have a pretty naive understanding of whatever motivates a superintelligent AI.
simple-thoughts 4y ago
I’m not sure what you mean by “stupid” but it’s definitely not the colloquial meaning. A human seeking suicide is not stupid. In fact humans often do seek suicide because humans are playing social games where the maximizing move is to commit suicide. Supporting these people requires changing their context so that they have a wider and better variety of options available. This is just a specific example of why I reject the orthogonality thesis. You change the context, you educate the agent, you change the goals of the agent. I do not agree that humans only chase “breathe feed sex” and while I do believe many stupid behaviors do come from evolutionary history, It’s plainly obvious that education, training, and genes play a role in self restraint and goal redirection.
- rafabulsing 4y ago
  What do you mean by stupid? I used that word only because you used it first. And I agree that humans do not only chase "breath feed sex", as I explicitly said that in my comment. We have other goals as well. But those are very much still there.

ShredKazoo 4y ago

>In real life, intelligence is associated with picking better goals. "Better" according to what metric? It may be the case that there is a tendency for high-intelligence humans to pick "more enlightened" goals. Perhaps there is a natural "enlightened goals" attractor for our species. However I don't think we can extrapolate from that to a fundamentally alien AI. I think even if this statistical tendency exists, it has clear counterexamples -- consider that 2 genius chess players may have opposite goals, of beating one another. And we shouldn't bet the future of humanity on this statistical tendency extrapolating outside of the original distribution of human species. Here are some intuition pumps on how diverse goals can be even across intelligent species: * Orcas killing sharks for their livers: https://www.livescience.com/2-orcas-slaughter-19-sharks-in-a-single-day-in-south-africa-eating-their-livers-and-leaving-them-to-rot https://www.livescience.com/2-orcas-slaughter-19-sharks-in-a... I don't believe dolphins show the same level of violence, even though both are smart cetaceans * Intelligent dogs bred to be responsive and attentive to human needs -- unlike close cousins like the wolf * Chimpanzees and bonobos are both related to humans, both highly intelligent, but with very different culture and goals https://www.youtube.com/watch?v=c6Ko0Hzi47U https://www.youtube.com/watch?v=c6Ko0Hzi47U

simple-thoughts 4y ago
> “Better” by what metrics? As soon as you phrase goal selection in terms of metrics, you’re assuming that goal selection is based on some other goal - that is you’re already assuming the orthoganality thesis. Your logic is fully circular. One thing that’s interesting to note about all of the examples you picked - every single one of those species shows cooperative behaviors. They share many other behaviors that are more similar than they are different. To reject the orthoganality thesis it’s sufficient to show that there is an empirical general association between intelligence and certain goals - then we can extrapolate an AI although of course it will function differently and may have many unusual behaviors will tend to follow those goals more directly. For instance intelligence is associated with : cooperation, empathy, inter and intra species communication, curiosity, etc. all of the species you mentioned exhibit these more than less intelligent species. Meanwhile something like “hunting to eat” is observed across the intelligence spectrum.

mitthrowaway2 4y ago

> The orthogonality thesis is an unproven assumption A safety mindset would suggest that rather than disregarding it until proven true, we should worry about it until proven false. > meaning an intelligent being can pursue stupid goals. It would pursue very intelligent instrumental goals, but the terminal goal is a free variable and I don't think there exists any measure by which terminal goals can be considered smart or stupid. It would be whatever is implied by its programming. > States like that, it’s obviously wrong and laughable. Perhaps not so obviously wrong nor so laughable as you think?

simple-thoughts 4y ago
This “safety mindset” is the same as Pascal’s wager and has the same flaws. There’s no clear delineation in real entities between instrumental and terminal goals. Yeah if you’re talking to a brainwashed religious fanatic, definitely it can seem counterintuitive to the doctrines they believe.
- mitthrowaway2 4y ago
  > There’s no clear delineation in real entities between instrumental and terminal goals. In healthy individuals, yes there absolutely is. Terminal goals are the ones you pursue for their own sake; instrumental goals are the ones you pursue as part of a plan to pursue a terminal goal, or another instrumental goal which connects to a terminal goal. Most people go to work in the morning not as a terminal goal, but as an instrumental goal; employment is in service of another instrumental goal of earning money; earning money is in service of a terminal goal of not starving to death. This is not exactly controversial stuff here. Some people do get so focused on an instrumental goal like "earning money" that they develop tunnel-vision and forget what terminal goal that money was originally in service of, but that's something most of them will eventually realize and then write a self-help book about. Anyway, it takes intelligence to decide what your instrumental goals should be, such as whether there's perhaps a cleverer way to make money than by going to work for your boss each morning, but there's no way in which intelligence will help you choose your terminal goals. For the most part they aren't something you can even consciously choose.

allturtles 4y ago

> The orthoganility thesis is an unproven assumption that intelligence and goals are not corellated, meaning an intelligent being can pursue stupid goals. States like that, it’s obviously wrong and laughable. To me it's obviously wrong when stated that way because 'stupid' is not an appropriate metric for goals. We may consider goals good or bad within our value system, but that has little do to with 'stupid' or 'intelligent.' e.g. a body builder may have a goal to get as ripped as possible, a VC to make as much money as possible, an ascetic to deny the flesh as much as possible. Which of these are stupid or intelligent goals? I don't think that's a question that makes sense. We may consider the goals of the Athenians (to expand their power) "worse" than the goals of the Melians (to be left alone)[0], but I don't see how they were "stupider." [0]: https://en.wikipedia.org/wiki/Siege_of_Melos#The_Melian_Dialogue https://en.wikipedia.org/wiki/Siege_of_Melos#The_Melian_Dial...