It's not like one billion human beings. Humans are limited by a lot of things like motivation and communication which is likely distinct from the general probl

It's not like one billion human beings. Humans are limited by a lot of things like motivation and communication which is likely distinct from the general problem solving architecture itself. Humans also have goals and an evolved morality that guides their thinking. Humans share a similar type of intelligence and understanding of goals, culture, right and wrong. You don't get this for free with AGI. The point of the human example is that there already exists a general problem solving architecture you can scale up (it's not impossible and there are brains everywhere in nature). If you increase operations per second many magnitudes you can have all human learning over thousands of years compressed into a couple of hours. You see this loosely in narrow spaces now with things like Alpha Go Zero. There may not be other AGIs if there's an intelligence explosion and the first one improves itself very quickly. If its utility function is not aligned with human goals then it could end up turning all matter into paperclips - not because it's evil, but because it's hard to set goals tied to human morality (when humans don't even agree in every case) and it happens to be configured such that making paperclips maxes out its reward function by accident. It may be possible to make the unsafe version before making the safe one, since it seems harder to make an AGI that's aligned with human goals than just an AGI in general.