Perhaps, but only to an extent. It is not as if, if only average people were clever enough, we could all become billionaires, and the GDP could be 10,000x high

Perhaps, but only to an extent. It is not as if, if only average people were clever enough, we could all become billionaires, and the GDP could be 10,000x higher than it currently is. There are, in fact, scarce resources – including land and minerals and other natural resources, but also labor. Most billionaires become that way by being in charge of a large company, one with a lot of workers. Well, you can't be the ruler without people to rule over. And if not for you, those people would still exist and would be making some other contribution to society.