4 ms·

I brought Bostrom's book Superintelligence and got about halfway through it before I moved into something else - it was a little disappointing. Throughout it se

I brought Bostrom's book Superintelligence and got about halfway through it before I moved into something else - it was a little disappointing. Throughout it seems that he just doesn't get what makes something 'intelligent' - while a machine that optimises paper clip production might be an application of artificial intelligence, it's not artificial general intelligent, and it's hardly any more advanced than some particularly well automated factories that already exist today. Artificial general intelligence at s human level implied to me that such a machine can think and consider things at least as much as a person - and therefore probably understands that making lots of paper clips isn't the be all and end all of existing.

Jach 11y ago
That's some useful feedback to know as I've considered recommending that book but haven't read it myself as it's redundant to what I've already read, but if Bostrom doesn't seem to explain very well that there's a distinction between intelligence and goals/values, I'll just keep linking to some basic online texts. (Try http://wiki.lesswrong.com/wiki/Complexity_of_value http://wiki.lesswrong.com/wiki/Complexity_of_value) The paperclip superintelligence will indeed be able to reason more effectively than humans about what's the be all and end all of existing -- the problem is that its conclusion will always be "to make more paperclips", because that's the overarching value it uses to frame all its thoughts on future actions, and its reasoning will be air tight. It will also be capable of explaining to any human wondering why its being torn apart for paperclip conversion that human values are different (they are such and such) and because it does not share those values, it comes to a different conclusion for the meaning of life, and it will also be able to generate great arguments for why its value system is superior. But it probably won't bother to do so...
- thomasfoster96 11y ago
  It's probably still worth reading the book if you're interested in a philosophical view of super intelligence, but if you're looking for a detailed look at the philosophical issues concerning artificial general intelligence (which I thought it would be), it's probably not the book for you.
rl3 11y ago
The paperclip maximizer example is a thought experiment designed to drive home the point that AGI architecture and motivations may be completely alien relative to human cognition. That same paperclip maximizer, while lacking the ability modify its fundamental goals (a core human trait), could very well far exceed human capability in every other realm. The question of whether such an entity still constitutes an AGI is certainly an interesting one, but likely irrelevant none the less. After all, the paperclip maximizer (as defined in the thought experiment) is capable of world domination. The degree to which it can introspect or modify its goals, and thus qualify as a true AGI, is merely semantics at that point.
- thomasfoster96 11y ago
  I would consider being able to think about and change ones goals/purpose would be a reasonably important in regards to deciding if a machine was intelligent. Note that I think Bostrom is referencing the section on the paperclip thought experiment in his book, which goes into a bit more detail than this article and to me characterises the paper clip optimiser as an example of above human level intelligence.
  - Houshalter 11y ago
    I don't think people change their own top level goals. At least not intentionally. You wouldn't take a pill that made you a sociopath, or rip out the part of your brain that causes you to feel happy, etc. Our values are essentially given to us by evolution. Pleasure, pain, empathy for others, novelty seeking, sense of beauty, all our social instincts, joy and sadness, etc. There is no reason to believe that an AI would have any of these things unless we made a massive effort to replicate them exactly. It would naturally have very different motivations, different goals and different values. For example an AI without boredom would just find some optimal experience and perform it over and over again until the end of time. An AI without empathy wouldn't care about harming humans or other beings. In other words, the Orthogonality thesis, that there is no universal correlation between intelligence and values. http://wiki.lesswrong.com/wiki/Orthogonality_thesis http://wiki.lesswrong.com/wiki/Orthogonality_thesis

Jach 11y ago

That's some useful feedback to know as I've considered recommending that book but haven't read it myself as it's redundant to what I've already read, but if Bostrom doesn't seem to explain very well that there's a distinction between intelligence and goals/values, I'll just keep linking to some basic online texts. (Try http://wiki.lesswrong.com/wiki/Complexity_of_value http://wiki.lesswrong.com/wiki/Complexity_of_value) The paperclip superintelligence will indeed be able to reason more effectively than humans about what's the be all and end all of existing -- the problem is that its conclusion will always be "to make more paperclips", because that's the overarching value it uses to frame all its thoughts on future actions, and its reasoning will be air tight. It will also be capable of explaining to any human wondering why its being torn apart for paperclip conversion that human values are different (they are such and such) and because it does not share those values, it comes to a different conclusion for the meaning of life, and it will also be able to generate great arguments for why its value system is superior. But it probably won't bother to do so...

thomasfoster96 11y ago
It's probably still worth reading the book if you're interested in a philosophical view of super intelligence, but if you're looking for a detailed look at the philosophical issues concerning artificial general intelligence (which I thought it would be), it's probably not the book for you.

rl3 11y ago

The paperclip maximizer example is a thought experiment designed to drive home the point that AGI architecture and motivations may be completely alien relative to human cognition. That same paperclip maximizer, while lacking the ability modify its fundamental goals (a core human trait), could very well far exceed human capability in every other realm. The question of whether such an entity still constitutes an AGI is certainly an interesting one, but likely irrelevant none the less. After all, the paperclip maximizer (as defined in the thought experiment) is capable of world domination. The degree to which it can introspect or modify its goals, and thus qualify as a true AGI, is merely semantics at that point.

thomasfoster96 11y ago
I would consider being able to think about and change ones goals/purpose would be a reasonably important in regards to deciding if a machine was intelligent. Note that I think Bostrom is referencing the section on the paperclip thought experiment in his book, which goes into a bit more detail than this article and to me characterises the paper clip optimiser as an example of above human level intelligence.
- Houshalter 11y ago
  I don't think people change their own top level goals. At least not intentionally. You wouldn't take a pill that made you a sociopath, or rip out the part of your brain that causes you to feel happy, etc. Our values are essentially given to us by evolution. Pleasure, pain, empathy for others, novelty seeking, sense of beauty, all our social instincts, joy and sadness, etc. There is no reason to believe that an AI would have any of these things unless we made a massive effort to replicate them exactly. It would naturally have very different motivations, different goals and different values. For example an AI without boredom would just find some optimal experience and perform it over and over again until the end of time. An AI without empathy wouldn't care about harming humans or other beings. In other words, the Orthogonality thesis, that there is no universal correlation between intelligence and values. http://wiki.lesswrong.com/wiki/Orthogonality_thesis http://wiki.lesswrong.com/wiki/Orthogonality_thesis