3 ms·

I would consider being able to think about and change ones goals/purpose would be a reasonably important in regards to deciding if a machine was intelligent. N

I would consider being able to think about and change ones goals/purpose would be a reasonably important in regards to deciding if a machine was intelligent. Note that I think Bostrom is referencing the section on the paperclip thought experiment in his book, which goes into a bit more detail than this article and to me characterises the paper clip optimiser as an example of above human level intelligence.

Houshalter 11y ago
I don't think people change their own top level goals. At least not intentionally. You wouldn't take a pill that made you a sociopath, or rip out the part of your brain that causes you to feel happy, etc. Our values are essentially given to us by evolution. Pleasure, pain, empathy for others, novelty seeking, sense of beauty, all our social instincts, joy and sadness, etc. There is no reason to believe that an AI would have any of these things unless we made a massive effort to replicate them exactly. It would naturally have very different motivations, different goals and different values. For example an AI without boredom would just find some optimal experience and perform it over and over again until the end of time. An AI without empathy wouldn't care about harming humans or other beings. In other words, the Orthogonality thesis, that there is no universal correlation between intelligence and values. http://wiki.lesswrong.com/wiki/Orthogonality_thesis http://wiki.lesswrong.com/wiki/Orthogonality_thesis
- thomasfoster96 11y ago
  But isn't the paper clip scenario more a goal then an ideal? Most people change their goals throughout life - not many kids find themselves at age 30 still wanton to be a fairy or superman. I do see what you mean by 'values' (or top level 'goals'), but I would consider that very different to a 'goal' like making lots of paper clips.
  - Houshalter 11y ago
    Why do you do anything? Because you have wants/desires/values. And those don't change. Why does a kid want to be superman? Because it sounds interesting or cool or fun. You don't lose your sense of interestingness or funness or coolness throughout your life, you just decide other things are more interesting or more fun. That is what I mean by "top level" goal.

Houshalter 11y ago

I don't think people change their own top level goals. At least not intentionally. You wouldn't take a pill that made you a sociopath, or rip out the part of your brain that causes you to feel happy, etc. Our values are essentially given to us by evolution. Pleasure, pain, empathy for others, novelty seeking, sense of beauty, all our social instincts, joy and sadness, etc. There is no reason to believe that an AI would have any of these things unless we made a massive effort to replicate them exactly. It would naturally have very different motivations, different goals and different values. For example an AI without boredom would just find some optimal experience and perform it over and over again until the end of time. An AI without empathy wouldn't care about harming humans or other beings. In other words, the Orthogonality thesis, that there is no universal correlation between intelligence and values. http://wiki.lesswrong.com/wiki/Orthogonality_thesis http://wiki.lesswrong.com/wiki/Orthogonality_thesis

thomasfoster96 11y ago
But isn't the paper clip scenario more a goal then an ideal? Most people change their goals throughout life - not many kids find themselves at age 30 still wanton to be a fairy or superman. I do see what you mean by 'values' (or top level 'goals'), but I would consider that very different to a 'goal' like making lots of paper clips.
- Houshalter 11y ago
  Why do you do anything? Because you have wants/desires/values. And those don't change. Why does a kid want to be superman? Because it sounds interesting or cool or fun. You don't lose your sense of interestingness or funness or coolness throughout your life, you just decide other things are more interesting or more fun. That is what I mean by "top level" goal.