3 ms·

Take the human general intelligence learning architecture and scale it up to a billion operations a second. The trick with AGI is that it can learn faster, and

Take the human general intelligence learning architecture and scale it up to a billion operations a second. The trick with AGI is that it can learn faster, and humans (even with our limitations) did take over the world.

goatlover 8y ago
So one AGI is like a billion human beings, so a little bit less than China or India. And like countries, there will be other AGIs. An AGI isn't born in a vacuum, it comes into existence in a world advanced enough for such a tech to be developed. As such, it's reasonable to suppose there will limitations for AGIs when it decides that world conquest is a worthy goal.
- fossuser 8y ago
  It's not like one billion human beings. Humans are limited by a lot of things like motivation and communication which is likely distinct from the general problem solving architecture itself. Humans also have goals and an evolved morality that guides their thinking. Humans share a similar type of intelligence and understanding of goals, culture, right and wrong. You don't get this for free with AGI. The point of the human example is that there already exists a general problem solving architecture you can scale up (it's not impossible and there are brains everywhere in nature). If you increase operations per second many magnitudes you can have all human learning over thousands of years compressed into a couple of hours. You see this loosely in narrow spaces now with things like Alpha Go Zero. There may not be other AGIs if there's an intelligence explosion and the first one improves itself very quickly. If its utility function is not aligned with human goals then it could end up turning all matter into paperclips - not because it's evil, but because it's hard to set goals tied to human morality (when humans don't even agree in every case) and it happens to be configured such that making paperclips maxes out its reward function by accident. It may be possible to make the unsafe version before making the safe one, since it seems harder to make an AGI that's aligned with human goals than just an AGI in general.

goatlover 8y ago

So one AGI is like a billion human beings, so a little bit less than China or India. And like countries, there will be other AGIs. An AGI isn't born in a vacuum, it comes into existence in a world advanced enough for such a tech to be developed. As such, it's reasonable to suppose there will limitations for AGIs when it decides that world conquest is a worthy goal.

fossuser 8y ago
It's not like one billion human beings. Humans are limited by a lot of things like motivation and communication which is likely distinct from the general problem solving architecture itself. Humans also have goals and an evolved morality that guides their thinking. Humans share a similar type of intelligence and understanding of goals, culture, right and wrong. You don't get this for free with AGI. The point of the human example is that there already exists a general problem solving architecture you can scale up (it's not impossible and there are brains everywhere in nature). If you increase operations per second many magnitudes you can have all human learning over thousands of years compressed into a couple of hours. You see this loosely in narrow spaces now with things like Alpha Go Zero. There may not be other AGIs if there's an intelligence explosion and the first one improves itself very quickly. If its utility function is not aligned with human goals then it could end up turning all matter into paperclips - not because it's evil, but because it's hard to set goals tied to human morality (when humans don't even agree in every case) and it happens to be configured such that making paperclips maxes out its reward function by accident. It may be possible to make the unsafe version before making the safe one, since it seems harder to make an AGI that's aligned with human goals than just an AGI in general.