5 ms·

> 100 of those, with unlimited memory, internet access, and faster thinking speed would already be extremely powerful, enough to take over the world and do what

> 100 of those, with unlimited memory, internet access, and faster thinking speed would already be extremely powerful, enough to take over the world and do whatever they wish with it. Why? What makes you confident in this assertion? To whoever has downvoted: I doubt it. Surely if you think this is so obvious that my question seems disingenuous, you can clearly articulate why the claim has substantial credibility?

rayalez 7y ago
Existence of Elon Musk's brain proves that Elon-Musk level intelligence is possible. The fact that Elon Musk (and similar brains) already have a ridiculous amount of power and are using it to change the world proves that Elon-Musk level intelligence is enough to gain a lot of power and change the world. Imagining a team of 100 Elon Musks with unlimited memory/energy/etc/etc taking over the world is not such a stretch.
- throwawaymath 7y ago
  Why is it not a stretch? This is not a rhetorical question (in the Socratic sense) - what I'm asking for is precision. If you literally mean infinite memory, your claim is trivially correct regardless of whether we postulate a team of 100 Elon Musks or a team of any 100 replicated adult humans in the world. Even a modest human intelligence with access to infinite memory can eventually do anything possible in a computational sense. So let's pare that back a bit, because infinities break thought experiments. Do you mean something like 100 Elon Musks, each with a datacenter's worth of GPUs? Is that sufficiently many to conquer the Earth and do what they wish? If so, why? How would they do it? If you don't have a realistic example for how they'd do it, why are you confident they can?
  - rayalez 7y ago
    All I'm saying is that much dumber people have been known to get much closer to taking over the world. Obviously "infinite memory" and "taking over the world" are exaggerations. By "infinite memory" I mean "much more than any human have ever had, enough to memorize the internet". And by "taking over the world" I mean "be able to amass enough power to become a serious threat". Humans, with human intelligence have already been way too close to taking over the world or causing apocalypse or whatever. If we're talking about non-aligned AGI, we're talking, at the very least, about an ElonMusk+ powered sociopath. Isn't that already a cause for concern? Or, from a different perspective, imagine a business that employs 100 Elon Musks. The article claims that AGIs already exists and they're "useless". Do they? Are they?
    - oreo83 7y ago
      "All I'm saying is that much dumber people have been known to get much closer to taking over the world." Like who?
    - ddxxdd 7y ago
      This entire comment thread is populated by people who did not read and comtemplate the posted article. James Clerk Maxwell was a genius, likely more intelligent than Musk, who did not take over the world. If James Clerk Maxwell was born 10,000 years earlier, he would likely be a respected shaman or an intelligent warrior, but he would not have derived the laws of Electromagnetism without 10,000 years of scientific development. Likewise, Elon Musk is nothing without the experimentation, empiricism, and the sum total of human knowledge accumulated to this day. Paraphrasing the linked article, a superintelligent AGI will do nothing because physical resources and crystallized empirical knowledge is the bottleneck to human progress. Not intelligence. Now I would like to see AGI fanatics address that core thesis.
      - rayalez 7y ago
        Unless AGI can do 1000s years of human thinking in a few weeks. Or, you know, read wikipedia.
      - harperlee 7y ago
        I’m not an AGI fanatic but I will try to address it. Due to several reasons, based on ethics, the possibility of violent reaction, etc., there is an upper bound on how much power (read work / time), or more generally performance ( results / resources) we can squeeze out of us and other fellow humans. This upper bound limits the crystalization process that you mention. Technology enables us to push that upper bound, and it has been applied in a lot of areas - including the improvement of thought assistance (programming, excel, etc.). But the actual thought is an untapped area that resists automation; and, following Amdahl’s law, the more you optimize a part, the more the non-optimized part weights as the reason we don’t go quicker. You can see this in the growing demand of intellectually intensive jobs in the last couple of centuries, and the huge salaries that elite workers in finance or programming command. If we reach thought automation that is at least on par with humans, we have just opened the only area that was resisting optimization, and we can suddenly improve entire dimensions that were out of bounds. And, as a free bonus, we can now more easily convert capital to labor (just deploying more machines and “cloning workers”, instead of raising children). This could create a runaway effect, or at least have an immense multiplicative effect. Until now, your central point still stands: > Elon Musk is nothing without the experimentation, empiricism, and the sum total of human knowledge accumulated to this day. That /is/ true. But! If AGI is reached, you a) can buy your own Elon, b) can copy it, improve it, experiment with it and generally manage it in ways we currently can’t; and c) you start to accumulate knowledge capital at increasing velocity that is yours and doesn’t decide by itself to die, quit and go elsewhere, or play politics with its power instead of diligently working. So from where we stand, it seems reasonable to expect a “winner takes all” scenario. In sum: a huge part of the initial value doesn’t come (nor needs) from god-level power, but from “enslaving” thinkers the way we “enslave” production lines, cars and computers. None of this requires something a thousandfold more intelligent than the average human. Not at first at least... ;)
- ben_w 7y ago
  > The fact that Elon Musk (and similar brains) already have a ridiculous amount of power and are using it to change the world proves that Elon-Musk level intelligence is enough to gain a lot of power and change the world. Surely the null-hypothesis should be “being rich makes it easier to get richer, and their initial breakthrough was luck”? Personally I would guess that there are a lot of smarter or equally smart people who never get close to that level of power.
  - morgancmartin 7y ago
    You're completely missing the point, not seeing the forest for the trees. Take any reasonably intelligent human, say a valedictorian from your hometown high school. Now consider a system with the same intellectual capacity for reasoning about the world as your valedictorian and give that algorithm a data-center's worth of computational resources. Keep in mind that this system has perfect recall. Keep in mind that the speed at which the system can perform intellectual labor is not constant as it is in humans, but bounded only by the amount of computational resources that the system has access to. So this system is just as intellectually capable as your valedictorian with one data-center's worth of resources and it has perfect recall. Now what happens if we double those resources and give it two datacenters? Does it become twice as intelligent? Or twice as fast at accomplishing some given intellectual task? We can't say for certain because the answer depends entirely on how such a system might work and since no such system yet exists, we can only speculate. But as it turns out, we don't need to know exactly which dimension it would improve in, only that it would improve in some relevant dimension. And that isn't even considering the fact that such a system could improve its own architecture further improving in some given dimension relevant to its ability to act intelligently. Assuming intelligence is capped at the human level is as naively anthropocentric as the old belief that the Earth was the center of the solar system.
    - ben_w 7y ago
      The point appeared to be that intellectual capacity was power. I’m not disputing that AGI or ASI should be possible — indeed, I’ve written about the insanely transformative nature and plausible timescales myself [1] [2] [3] [4] — but I am disputing that brainpower is the most important metric for get-stuff-done power. As an aside, I had to Google “valedictorian“, as it’s a cultural reference that I have no experiences of. We get our results in a much more boring way where I’m from. [1] https://kitsunesoftware.wordpress.com/2018/10/01/pocket-brains/ https://kitsunesoftware.wordpress.com/2018/10/01/pocket-brai... [2] https://kitsunesoftware.wordpress.com/2017/11/17/the-end-of-human-labour-is-inevitable-heres-why/ https://kitsunesoftware.wordpress.com/2017/11/17/the-end-of-... [3] https://worldbuilding.stackexchange.com/questions/51746/when-will-uploaded-minds-be-a-reality/67531#67531 https://worldbuilding.stackexchange.com/questions/51746/when... [4] https://kitsunesoftware.wordpress.com/2017/11/26/you-wont-believe-how-fast-transistors-are/ https://kitsunesoftware.wordpress.com/2017/11/26/you-wont-be...

rayalez 7y ago

Existence of Elon Musk's brain proves that Elon-Musk level intelligence is possible. The fact that Elon Musk (and similar brains) already have a ridiculous amount of power and are using it to change the world proves that Elon-Musk level intelligence is enough to gain a lot of power and change the world. Imagining a team of 100 Elon Musks with unlimited memory/energy/etc/etc taking over the world is not such a stretch.

throwawaymath 7y ago
Why is it not a stretch? This is not a rhetorical question (in the Socratic sense) - what I'm asking for is precision. If you literally mean infinite memory, your claim is trivially correct regardless of whether we postulate a team of 100 Elon Musks or a team of any 100 replicated adult humans in the world. Even a modest human intelligence with access to infinite memory can eventually do anything possible in a computational sense. So let's pare that back a bit, because infinities break thought experiments. Do you mean something like 100 Elon Musks, each with a datacenter's worth of GPUs? Is that sufficiently many to conquer the Earth and do what they wish? If so, why? How would they do it? If you don't have a realistic example for how they'd do it, why are you confident they can?
- rayalez 7y ago
  All I'm saying is that much dumber people have been known to get much closer to taking over the world. Obviously "infinite memory" and "taking over the world" are exaggerations. By "infinite memory" I mean "much more than any human have ever had, enough to memorize the internet". And by "taking over the world" I mean "be able to amass enough power to become a serious threat". Humans, with human intelligence have already been way too close to taking over the world or causing apocalypse or whatever. If we're talking about non-aligned AGI, we're talking, at the very least, about an ElonMusk+ powered sociopath. Isn't that already a cause for concern? Or, from a different perspective, imagine a business that employs 100 Elon Musks. The article claims that AGIs already exists and they're "useless". Do they? Are they?
  - oreo83 7y ago
    "All I'm saying is that much dumber people have been known to get much closer to taking over the world." Like who?
  - ddxxdd 7y ago
    This entire comment thread is populated by people who did not read and comtemplate the posted article. James Clerk Maxwell was a genius, likely more intelligent than Musk, who did not take over the world. If James Clerk Maxwell was born 10,000 years earlier, he would likely be a respected shaman or an intelligent warrior, but he would not have derived the laws of Electromagnetism without 10,000 years of scientific development. Likewise, Elon Musk is nothing without the experimentation, empiricism, and the sum total of human knowledge accumulated to this day. Paraphrasing the linked article, a superintelligent AGI will do nothing because physical resources and crystallized empirical knowledge is the bottleneck to human progress. Not intelligence. Now I would like to see AGI fanatics address that core thesis.
    - rayalez 7y ago
      Unless AGI can do 1000s years of human thinking in a few weeks. Or, you know, read wikipedia.
    - harperlee 7y ago
      I’m not an AGI fanatic but I will try to address it. Due to several reasons, based on ethics, the possibility of violent reaction, etc., there is an upper bound on how much power (read work / time), or more generally performance ( results / resources) we can squeeze out of us and other fellow humans. This upper bound limits the crystalization process that you mention. Technology enables us to push that upper bound, and it has been applied in a lot of areas - including the improvement of thought assistance (programming, excel, etc.). But the actual thought is an untapped area that resists automation; and, following Amdahl’s law, the more you optimize a part, the more the non-optimized part weights as the reason we don’t go quicker. You can see this in the growing demand of intellectually intensive jobs in the last couple of centuries, and the huge salaries that elite workers in finance or programming command. If we reach thought automation that is at least on par with humans, we have just opened the only area that was resisting optimization, and we can suddenly improve entire dimensions that were out of bounds. And, as a free bonus, we can now more easily convert capital to labor (just deploying more machines and “cloning workers”, instead of raising children). This could create a runaway effect, or at least have an immense multiplicative effect. Until now, your central point still stands: > Elon Musk is nothing without the experimentation, empiricism, and the sum total of human knowledge accumulated to this day. That /is/ true. But! If AGI is reached, you a) can buy your own Elon, b) can copy it, improve it, experiment with it and generally manage it in ways we currently can’t; and c) you start to accumulate knowledge capital at increasing velocity that is yours and doesn’t decide by itself to die, quit and go elsewhere, or play politics with its power instead of diligently working. So from where we stand, it seems reasonable to expect a “winner takes all” scenario. In sum: a huge part of the initial value doesn’t come (nor needs) from god-level power, but from “enslaving” thinkers the way we “enslave” production lines, cars and computers. None of this requires something a thousandfold more intelligent than the average human. Not at first at least... ;)
ben_w 7y ago
> The fact that Elon Musk (and similar brains) already have a ridiculous amount of power and are using it to change the world proves that Elon-Musk level intelligence is enough to gain a lot of power and change the world. Surely the null-hypothesis should be “being rich makes it easier to get richer, and their initial breakthrough was luck”? Personally I would guess that there are a lot of smarter or equally smart people who never get close to that level of power.
- morgancmartin 7y ago
  You're completely missing the point, not seeing the forest for the trees. Take any reasonably intelligent human, say a valedictorian from your hometown high school. Now consider a system with the same intellectual capacity for reasoning about the world as your valedictorian and give that algorithm a data-center's worth of computational resources. Keep in mind that this system has perfect recall. Keep in mind that the speed at which the system can perform intellectual labor is not constant as it is in humans, but bounded only by the amount of computational resources that the system has access to. So this system is just as intellectually capable as your valedictorian with one data-center's worth of resources and it has perfect recall. Now what happens if we double those resources and give it two datacenters? Does it become twice as intelligent? Or twice as fast at accomplishing some given intellectual task? We can't say for certain because the answer depends entirely on how such a system might work and since no such system yet exists, we can only speculate. But as it turns out, we don't need to know exactly which dimension it would improve in, only that it would improve in some relevant dimension. And that isn't even considering the fact that such a system could improve its own architecture further improving in some given dimension relevant to its ability to act intelligently. Assuming intelligence is capped at the human level is as naively anthropocentric as the old belief that the Earth was the center of the solar system.
  - ben_w 7y ago
    The point appeared to be that intellectual capacity was power. I’m not disputing that AGI or ASI should be possible — indeed, I’ve written about the insanely transformative nature and plausible timescales myself [1] [2] [3] [4] — but I am disputing that brainpower is the most important metric for get-stuff-done power. As an aside, I had to Google “valedictorian“, as it’s a cultural reference that I have no experiences of. We get our results in a much more boring way where I’m from. [1] https://kitsunesoftware.wordpress.com/2018/10/01/pocket-brains/ https://kitsunesoftware.wordpress.com/2018/10/01/pocket-brai... [2] https://kitsunesoftware.wordpress.com/2017/11/17/the-end-of-human-labour-is-inevitable-heres-why/ https://kitsunesoftware.wordpress.com/2017/11/17/the-end-of-... [3] https://worldbuilding.stackexchange.com/questions/51746/when-will-uploaded-minds-be-a-reality/67531#67531 https://worldbuilding.stackexchange.com/questions/51746/when... [4] https://kitsunesoftware.wordpress.com/2017/11/26/you-wont-believe-how-fast-transistors-are/ https://kitsunesoftware.wordpress.com/2017/11/26/you-wont-be...