5 ms·

I read "Superintelligence" by Nick Bostrom, essentially on the recommendation of Elon Musk (he tweeted about it). It talks about the dangers of strong AI and p

by chubot 12y ago

I read "Superintelligence" by Nick Bostrom, essentially on the recommendation of Elon Musk (he tweeted about it). It talks about the dangers of strong AI and possible paths to it, and how humans can mitigate its effects. The only reason I read past the beginning is because in the preface he says: "This book is likely to be seriously mistaken in a number of ways". So at least he's intellectually honest. I believe he's building 300 pages of argument and analysis on a flawed premise. As far as I can tell, the entire discussion rests on what he calls "instrumental goals" vs. "final goals". (This article I found on Google has similar content: http://www.nickbostrom.com/superintelligentwill.pdf http://www.nickbostrom.com/superintelligentwill.pdf ) His example is the "paper clip maximizer": http://wiki.lesswrong.com/wiki/Paperclip_maximizer http://wiki.lesswrong.com/wiki/Paperclip_maximizer In this situation, the final goal is: Produce the maximum number of paper clips. The instrumental goal is: Acquire all resources in the world so that you can direct them toward paper clip production, which involve destroying all humans, etc. Personally, I don't believe this threat is worth thinking about this point. The supposed path to implementing such a technology isn't credible, and it seems orders of magnitude less likely than, say, us having to evacuate the entire planet. In other words, I believe that we will be able to build very useful special purpose AIs that accomplish our goals. I can see a future full of benign "plant-like" intelligences, existing indefinitely. They are machines that take in incredible amounts of information, and spit out ingenious answers that no human could have come up with. From that, it doesn't follow there there is any motivation to take over the world. We should think about the many, many challenges ahead with special purpose AI instead, and our increasing dependence on computing. All these special purpose AIs will collecting everybody's personal data, shape their behavior, etc. For example, you can easily imagine a company like Facebook or Google deciding to sway an election. There are a lot more important problems to be thinking about now.

ikeboy 12y ago
Why do you feel we will not build "Strong" AI? We know AI as smart as humans can theoretically exist, because humans exist, so intelligence is possible. Is there a reason to think that humans are near the upper limit of what's possible? If not, why can't we build something significantly smarter than us?
- DougMerritt 12y ago
  Just to play Devil's Advocate, not because I believe this: We can, and do, create machines that are faster than humans, and it is reasonable to assume that eventually computers can be created that have capacity and connectivity larger than humans. But that doesn't prove that the result will be as intelligent as humans, let alone more intelligent, because intelligence is clearly more than just speed and capacity. Edit: for instance, faster and bigger does not change the O(n) complexity analysis of algorithms, thus faster and bigger does relatively little to improve the ability to solve exponential problems. Exactly what intelligence is, is still not known, and it is quite possible that we will need an algorithmic breakthrough to create something equivalent to human minds. And even if that happens, making that thing faster and bigger than human minds may turn out not to result in superintelligence -- that would just be another similar assumption as the first. So it might need yet another algorithmic breakthrough to create superintelligence, and it is possible that neither humans nor human-equivalent AIs will be able to make that breakthrough. Again, I don't believe the above, but I think it is purely a matter of optimism versus pessimism, not about things that are proven.
  - ikeboy 12y ago
    >Exactly what intelligence is, is still not known, and it is quite possible that we will need an algorithmic breakthrough to create something equivalent to human minds. Let's assume this is true. >And even if that happens, making that thing faster and bigger than human minds may turn out not to result in superintelligence -- that would just be another similar assumption as the first. So it might need yet another algorithmic breakthrough to create superintelligence, and it is possible that neither humans nor human-equivalent AIs will be able to make that breakthrough. This seems unlikely to be true. Imagine we have a program that is as smart as the human brain. It's running on hardware a million times as fast, though. So at the very least, it can think in one day what a human can think in a million days. Even if it's restricted to human-level intelligence, it should still be far more effective than a human. This is even without considering self-modifications.
    - DougMerritt 12y ago

by chubot 12y ago

ikeboy 12y ago

Why do you feel we will not build "Strong" AI? We know AI as smart as humans can theoretically exist, because humans exist, so intelligence is possible. Is there a reason to think that humans are near the upper limit of what's possible? If not, why can't we build something significantly smarter than us?

DougMerritt 12y ago
Just to play Devil's Advocate, not because I believe this: We can, and do, create machines that are faster than humans, and it is reasonable to assume that eventually computers can be created that have capacity and connectivity larger than humans. But that doesn't prove that the result will be as intelligent as humans, let alone more intelligent, because intelligence is clearly more than just speed and capacity. Edit: for instance, faster and bigger does not change the O(n) complexity analysis of algorithms, thus faster and bigger does relatively little to improve the ability to solve exponential problems. Exactly what intelligence is, is still not known, and it is quite possible that we will need an algorithmic breakthrough to create something equivalent to human minds. And even if that happens, making that thing faster and bigger than human minds may turn out not to result in superintelligence -- that would just be another similar assumption as the first. So it might need yet another algorithmic breakthrough to create superintelligence, and it is possible that neither humans nor human-equivalent AIs will be able to make that breakthrough. Again, I don't believe the above, but I think it is purely a matter of optimism versus pessimism, not about things that are proven.
- ikeboy 12y ago
  >Exactly what intelligence is, is still not known, and it is quite possible that we will need an algorithmic breakthrough to create something equivalent to human minds. Let's assume this is true. >And even if that happens, making that thing faster and bigger than human minds may turn out not to result in superintelligence -- that would just be another similar assumption as the first. So it might need yet another algorithmic breakthrough to create superintelligence, and it is possible that neither humans nor human-equivalent AIs will be able to make that breakthrough. This seems unlikely to be true. Imagine we have a program that is as smart as the human brain. It's running on hardware a million times as fast, though. So at the very least, it can think in one day what a human can think in a million days. Even if it's restricted to human-level intelligence, it should still be far more effective than a human. This is even without considering self-modifications.
  - DougMerritt 12y ago