6 ms·

Currently humans are super intelligent compared to machine intelligence, so if the super intelligence can give rise to something more intelligent than it, could

by xt00 6y ago

Currently humans are super intelligent compared to machine intelligence, so if the super intelligence can give rise to something more intelligent than it, could the super intelligence give rise to something more intelligent than it? The answer must be yes, then the question is if containment is the problem and the conclusion is that it cannot be contained, then what we should be making right now is a super intelligence whose sole job in life is to contain superintelligences. Which sounds problematic because containment could result in physical destruction to create the containment. Hmm... superintelligence feels an awful lot like the worst case definition of pandora's box..

WJW 6y ago
> The answer must be yes This does not logically follow. It is entirely possible that going even further would require greater resources than even the superintelligence can bring to bear.
- TheOtherHobbes 6y ago
  For human-recognisable values of "resources." I realise that all of these arguments come down to "AI of the gaps". But in true Dunning-Kruger style, we don't know what we don't know - not just about AGI, but intelligence of any kind, including our own.
  - alisonkisk 6y ago
    That's not what Dunning-Kruger means.
  - dragonwriter 6y ago
    > But in true Dunning-Kruger style, we don't know what we don't know That's not D-K, but Rumsfeldian Uncertainty.
- xt00 6y ago
  Definitely feels like a challenging philosophical question -- if you have a superintelligence how much more room is there above it in intelligence and what does that additional intelligence buy you?
  - contravariant 6y ago
    The challenges don't even start there. The real problem is how to even determine superintelligence. If it's just a system that can do stuff a human can't then we've long since passed it. If it's a system that can achieve stuff humans can't, through whatever means, then it's necessarily beyond human intelligence to build it, at least intentionally.
geocar 6y ago
> could the super intelligence give rise to something more intelligent than it? The answer must be yes, I don't see any reason to believe that intelligence forms an infinite ladder, I mean, it's fun to think about, but surely Zeno catches the tortoise eventually!
- UnFleshedOne 6y ago
  There is also little reason to believe we are anywhere near the top of the finite ladder either.
hansvm 6y ago
Even if the hypotheses hold, a sequence x, f(x), f(f(x)), ... doesn't have to diverge even with f(x)>x for all x. A chain of super intelligences could feasibly have a small upper bound not much greater than that for a single person. That idea even lines up with some crude experimental data -- vast additional resources barely make a dent in the frontier for chess, image recognition, a breadth of graph problems, etc....
- michaelmrose 6y ago
  The problem of how to win at chess is bounded in complexity in a way that maximizing computation isn't. To provide an analogy there is a maximum speed anything can move at but it would be laughable if a medieval scholar had wondered if the falcon was near the maximum speed possible. It seems similarly ridiculous to imagine that the human brain just happens to be close to the maximum computational power because it was what we suppose to be the most powerful instrument produced thus far on our planet. Wouldn't it be equal parts unlikely and strange if the most effective and compact process that can be produced in any conditions just happened to be the one that random evolution happened on in a random walk through what is practical with earths organic chemistry. Even if we imagine the incredibly unlikely idea that you can't make a more effective mind you would think you could run the same process faster than real time or more instances of it for increasingly small resources on a better computer.
  - hansvm 6y ago
    To be perfectly clear, my point wasn't that people have near-supremal intelligence (the thing you seem to be arguing against), but that given the hypotheses in the parent comment you couldn't rule out such an eventuality. While we're on the topic though, no it doesn't have to be unlikely or strange for people to be near-supremal, and throwing more hardware at the problem doesn't necessarily make it go away: On some level that idea comes down to defining what we mean by "intelligence." Suppose we can only measure intelligence by proxy by measuring performance on specific tasks. Then the composition of those tasks plays into any final intelligence score. If they're all fully parallelizable then you're totally right that people almost certainly aren't anywhere near the "top," and moreover a "top" probably wouldn't even exist. In standard parlance though, intelligence is something more than the ability to add a lot of numbers quickly, and most of our current measures of intelligence have exponentially (or worse) diminishing returns as more hardware is added. In that latter case where intelligence is measured by performance on algorithms which are fundamentally hard, people wouldn't be near the top because of some rare process which accidentally made us that way. People would be near the top because _anything_ displaying a modicum of thought would be near the top because of the vastly diminishing returns of additional hardware. As a crude ballpark, if you mustered every atom on earth into its own processor running at 4GHz then you could solve a fully parallelizable exponential problem roughly 4x bigger than what one of Google's newest TPU's can manage.
qwantim1 6y ago
The most obvious solution to problems arising from assisting the creation of a superintelligence is not to assist the creation of a superintelligence. Unfortunately, we didn’t learn from the proliferation of nuclear weapons: “If we don’t make nuclear weapons, others will make them. If we make them then they will make them. If we make them and they make them, we’ll have to make more. If they make them and we make them, they’ll make more...” Note: This fallacy also helps sell guns and martial arts lessons.

by xt00 6y ago

WJW 6y ago

> The answer must be yes This does not logically follow. It is entirely possible that going even further would require greater resources than even the superintelligence can bring to bear.

TheOtherHobbes 6y ago
For human-recognisable values of "resources." I realise that all of these arguments come down to "AI of the gaps". But in true Dunning-Kruger style, we don't know what we don't know - not just about AGI, but intelligence of any kind, including our own.
- alisonkisk 6y ago
  That's not what Dunning-Kruger means.
- dragonwriter 6y ago
  > But in true Dunning-Kruger style, we don't know what we don't know That's not D-K, but Rumsfeldian Uncertainty.
xt00 6y ago
Definitely feels like a challenging philosophical question -- if you have a superintelligence how much more room is there above it in intelligence and what does that additional intelligence buy you?
- contravariant 6y ago
  The challenges don't even start there. The real problem is how to even determine superintelligence. If it's just a system that can do stuff a human can't then we've long since passed it. If it's a system that can achieve stuff humans can't, through whatever means, then it's necessarily beyond human intelligence to build it, at least intentionally.

geocar 6y ago

> could the super intelligence give rise to something more intelligent than it? The answer must be yes, I don't see any reason to believe that intelligence forms an infinite ladder, I mean, it's fun to think about, but surely Zeno catches the tortoise eventually!

UnFleshedOne 6y ago
There is also little reason to believe we are anywhere near the top of the finite ladder either.

hansvm 6y ago

Even if the hypotheses hold, a sequence x, f(x), f(f(x)), ... doesn't have to diverge even with f(x)>x for all x. A chain of super intelligences could feasibly have a small upper bound not much greater than that for a single person. That idea even lines up with some crude experimental data -- vast additional resources barely make a dent in the frontier for chess, image recognition, a breadth of graph problems, etc....

michaelmrose 6y ago
The problem of how to win at chess is bounded in complexity in a way that maximizing computation isn't. To provide an analogy there is a maximum speed anything can move at but it would be laughable if a medieval scholar had wondered if the falcon was near the maximum speed possible. It seems similarly ridiculous to imagine that the human brain just happens to be close to the maximum computational power because it was what we suppose to be the most powerful instrument produced thus far on our planet. Wouldn't it be equal parts unlikely and strange if the most effective and compact process that can be produced in any conditions just happened to be the one that random evolution happened on in a random walk through what is practical with earths organic chemistry. Even if we imagine the incredibly unlikely idea that you can't make a more effective mind you would think you could run the same process faster than real time or more instances of it for increasingly small resources on a better computer.
- hansvm 6y ago
  To be perfectly clear, my point wasn't that people have near-supremal intelligence (the thing you seem to be arguing against), but that given the hypotheses in the parent comment you couldn't rule out such an eventuality. While we're on the topic though, no it doesn't have to be unlikely or strange for people to be near-supremal, and throwing more hardware at the problem doesn't necessarily make it go away: On some level that idea comes down to defining what we mean by "intelligence." Suppose we can only measure intelligence by proxy by measuring performance on specific tasks. Then the composition of those tasks plays into any final intelligence score. If they're all fully parallelizable then you're totally right that people almost certainly aren't anywhere near the "top," and moreover a "top" probably wouldn't even exist. In standard parlance though, intelligence is something more than the ability to add a lot of numbers quickly, and most of our current measures of intelligence have exponentially (or worse) diminishing returns as more hardware is added. In that latter case where intelligence is measured by performance on algorithms which are fundamentally hard, people wouldn't be near the top because of some rare process which accidentally made us that way. People would be near the top because _anything_ displaying a modicum of thought would be near the top because of the vastly diminishing returns of additional hardware. As a crude ballpark, if you mustered every atom on earth into its own processor running at 4GHz then you could solve a fully parallelizable exponential problem roughly 4x bigger than what one of Google's newest TPU's can manage.

qwantim1 6y ago

The most obvious solution to problems arising from assisting the creation of a superintelligence is not to assist the creation of a superintelligence. Unfortunately, we didn’t learn from the proliferation of nuclear weapons: “If we don’t make nuclear weapons, others will make them. If we make them then they will make them. If we make them and they make them, we’ll have to make more. If they make them and we make them, they’ll make more...” Note: This fallacy also helps sell guns and martial arts lessons.