4 ms·

Why is it not a stretch? This is not a rhetorical question (in the Socratic sense) - what I'm asking for is precision. If you literally mean infinite memory, y

Why is it not a stretch? This is not a rhetorical question (in the Socratic sense) - what I'm asking for is precision. If you literally mean infinite memory, your claim is trivially correct regardless of whether we postulate a team of 100 Elon Musks or a team of any 100 replicated adult humans in the world. Even a modest human intelligence with access to infinite memory can eventually do anything possible in a computational sense. So let's pare that back a bit, because infinities break thought experiments. Do you mean something like 100 Elon Musks, each with a datacenter's worth of GPUs? Is that sufficiently many to conquer the Earth and do what they wish? If so, why? How would they do it? If you don't have a realistic example for how they'd do it, why are you confident they can?

rayalez 7y ago
All I'm saying is that much dumber people have been known to get much closer to taking over the world. Obviously "infinite memory" and "taking over the world" are exaggerations. By "infinite memory" I mean "much more than any human have ever had, enough to memorize the internet". And by "taking over the world" I mean "be able to amass enough power to become a serious threat". Humans, with human intelligence have already been way too close to taking over the world or causing apocalypse or whatever. If we're talking about non-aligned AGI, we're talking, at the very least, about an ElonMusk+ powered sociopath. Isn't that already a cause for concern? Or, from a different perspective, imagine a business that employs 100 Elon Musks. The article claims that AGIs already exists and they're "useless". Do they? Are they?
- oreo83 7y ago
  "All I'm saying is that much dumber people have been known to get much closer to taking over the world." Like who?
- ddxxdd 7y ago
  This entire comment thread is populated by people who did not read and comtemplate the posted article. James Clerk Maxwell was a genius, likely more intelligent than Musk, who did not take over the world. If James Clerk Maxwell was born 10,000 years earlier, he would likely be a respected shaman or an intelligent warrior, but he would not have derived the laws of Electromagnetism without 10,000 years of scientific development. Likewise, Elon Musk is nothing without the experimentation, empiricism, and the sum total of human knowledge accumulated to this day. Paraphrasing the linked article, a superintelligent AGI will do nothing because physical resources and crystallized empirical knowledge is the bottleneck to human progress. Not intelligence. Now I would like to see AGI fanatics address that core thesis.
  - rayalez 7y ago
    Unless AGI can do 1000s years of human thinking in a few weeks. Or, you know, read wikipedia.
  - harperlee 7y ago
    I’m not an AGI fanatic but I will try to address it. Due to several reasons, based on ethics, the possibility of violent reaction, etc., there is an upper bound on how much power (read work / time), or more generally performance ( results / resources) we can squeeze out of us and other fellow humans. This upper bound limits the crystalization process that you mention. Technology enables us to push that upper bound, and it has been applied in a lot of areas - including the improvement of thought assistance (programming, excel, etc.). But the actual thought is an untapped area that resists automation; and, following Amdahl’s law, the more you optimize a part, the more the non-optimized part weights as the reason we don’t go quicker. You can see this in the growing demand of intellectually intensive jobs in the last couple of centuries, and the huge salaries that elite workers in finance or programming command. If we reach thought automation that is at least on par with humans, we have just opened the only area that was resisting optimization, and we can suddenly improve entire dimensions that were out of bounds. And, as a free bonus, we can now more easily convert capital to labor (just deploying more machines and “cloning workers”, instead of raising children). This could create a runaway effect, or at least have an immense multiplicative effect. Until now, your central point still stands: > Elon Musk is nothing without the experimentation, empiricism, and the sum total of human knowledge accumulated to this day. That /is/ true. But! If AGI is reached, you a) can buy your own Elon, b) can copy it, improve it, experiment with it and generally manage it in ways we currently can’t; and c) you start to accumulate knowledge capital at increasing velocity that is yours and doesn’t decide by itself to die, quit and go elsewhere, or play politics with its power instead of diligently working. So from where we stand, it seems reasonable to expect a “winner takes all” scenario. In sum: a huge part of the initial value doesn’t come (nor needs) from god-level power, but from “enslaving” thinkers the way we “enslave” production lines, cars and computers. None of this requires something a thousandfold more intelligent than the average human. Not at first at least... ;)

rayalez 7y ago

All I'm saying is that much dumber people have been known to get much closer to taking over the world. Obviously "infinite memory" and "taking over the world" are exaggerations. By "infinite memory" I mean "much more than any human have ever had, enough to memorize the internet". And by "taking over the world" I mean "be able to amass enough power to become a serious threat". Humans, with human intelligence have already been way too close to taking over the world or causing apocalypse or whatever. If we're talking about non-aligned AGI, we're talking, at the very least, about an ElonMusk+ powered sociopath. Isn't that already a cause for concern? Or, from a different perspective, imagine a business that employs 100 Elon Musks. The article claims that AGIs already exists and they're "useless". Do they? Are they?

oreo83 7y ago
"All I'm saying is that much dumber people have been known to get much closer to taking over the world." Like who?
ddxxdd 7y ago
This entire comment thread is populated by people who did not read and comtemplate the posted article. James Clerk Maxwell was a genius, likely more intelligent than Musk, who did not take over the world. If James Clerk Maxwell was born 10,000 years earlier, he would likely be a respected shaman or an intelligent warrior, but he would not have derived the laws of Electromagnetism without 10,000 years of scientific development. Likewise, Elon Musk is nothing without the experimentation, empiricism, and the sum total of human knowledge accumulated to this day. Paraphrasing the linked article, a superintelligent AGI will do nothing because physical resources and crystallized empirical knowledge is the bottleneck to human progress. Not intelligence. Now I would like to see AGI fanatics address that core thesis.
- rayalez 7y ago
  Unless AGI can do 1000s years of human thinking in a few weeks. Or, you know, read wikipedia.
- harperlee 7y ago
  I’m not an AGI fanatic but I will try to address it. Due to several reasons, based on ethics, the possibility of violent reaction, etc., there is an upper bound on how much power (read work / time), or more generally performance ( results / resources) we can squeeze out of us and other fellow humans. This upper bound limits the crystalization process that you mention. Technology enables us to push that upper bound, and it has been applied in a lot of areas - including the improvement of thought assistance (programming, excel, etc.). But the actual thought is an untapped area that resists automation; and, following Amdahl’s law, the more you optimize a part, the more the non-optimized part weights as the reason we don’t go quicker. You can see this in the growing demand of intellectually intensive jobs in the last couple of centuries, and the huge salaries that elite workers in finance or programming command. If we reach thought automation that is at least on par with humans, we have just opened the only area that was resisting optimization, and we can suddenly improve entire dimensions that were out of bounds. And, as a free bonus, we can now more easily convert capital to labor (just deploying more machines and “cloning workers”, instead of raising children). This could create a runaway effect, or at least have an immense multiplicative effect. Until now, your central point still stands: > Elon Musk is nothing without the experimentation, empiricism, and the sum total of human knowledge accumulated to this day. That /is/ true. But! If AGI is reached, you a) can buy your own Elon, b) can copy it, improve it, experiment with it and generally manage it in ways we currently can’t; and c) you start to accumulate knowledge capital at increasing velocity that is yours and doesn’t decide by itself to die, quit and go elsewhere, or play politics with its power instead of diligently working. So from where we stand, it seems reasonable to expect a “winner takes all” scenario. In sum: a huge part of the initial value doesn’t come (nor needs) from god-level power, but from “enslaving” thinkers the way we “enslave” production lines, cars and computers. None of this requires something a thousandfold more intelligent than the average human. Not at first at least... ;)