8 ms·

By the time AGI is developed, it would be able to improve itself/build improvements, causing the runway effect that leads to superintelligence. The idea of maki

By the time AGI is developed, it would be able to improve itself/build improvements, causing the runway effect that leads to superintelligence. The idea of making "personal digital assistants" is humans thinking they can tame a god to be a servant. Taming a superintelligence would be like a dog taming human civilization, and perhaps the gap between man and ASI is larger than the gap between dog and man.

aaron-lebo 10y ago
It's pretty remarkable is it? We've got all kind of past fiction which plays on these concerns, more recently Ex Machina and Westworld. But here people are making the same bold pronouncements...so certain we know or can control technology which hasn't even been invented yet.
ClassyJacket 10y ago
>By the time AGI is developed, it would be able to improve itself/build improvements, I see no reason that's necessarily true. I'm a general intelligence, and I can't make myself smarter.
- amitprayal 10y ago
  You are a general intelligence ,but do you know why you are intelligent ?.Since you cant explain your own intelligence you cant improve on it, no such issues for AI unless its built using NN.
- YZF 10y ago
  But you can build something smarter than yourself (or at least that's what we think we can do)... So this new entity, smarter than us, might be even better at building something smarter than itself...
  - snovv_crash 10y ago
    Or maybe it would be smart enough not to.
    - simonh 10y ago
      Good point, I don't think I've seen that suggested before. However I think motivation is a big issue. Our instincts and emotions are what drive us far more than rationality. If we decide to build an AI smarter than us and program it to really, really want to design an even smarter AI then it might essentially have no choice.
      - taneq 10y ago
        > Good point, I don't think I've seen that suggested before. A fictional example, but Alastair Reynolds' "Inhibitors" are a type/race of machines which, while intelligent, were specifically designed to limit their own degree of sentience.
- Fricken 10y ago
  You're an iteration of an AGI system that has been improving itself for hundreds of millions of years. The rate at which biological AGIs improve over time is very slow, but it's not like nature has any good reason to be in a hurry. But interesting things happen when you network billions of biological AGIs together, it leads to all sorts of emergent phenomenon, and now the biological AGIs are working on these newfangled mechanical AGIs, which, while still crude, aren't bound by the same constraints, they can iterate much faster. Biological AGIs have crippling bandwidth/memory issues which aren't really a problem at all for their mechanical counterparts. These mechanical AGIs, I think they'll go places.
  - smhost 10y ago
    That gives me some really strong existential heebie jeebies. I mean, what then even is our value? Why exist at all, at that point? I don't know about other people, but I get my sense of purpose in believing that we're the captains of this Spaceship Earth, and that we're making progress towards something significant. I don't know what that something is, but I have a vague idea, and at the very least we seem to be the best that we've got. I don't know. Maybe I'm thinking about all of this wrong. God, why am I so damn confused all the time? Fuck.
    - simonh 10y ago
      > God, why am I so damn confused all the time? You and me both. I'm afraid it's because we're only just barely sentient. If you think about it, in evolutionary terms we literally only just now managed to build a technological society because we only just got smart enough to do it. We are by definition at the absolute minimum level of intelligence that's able to do that, otherwise we'd have done it sooner. We've had plenty of time. The human brain is a botch job of highly optimizes special-function systems that has developed just enough sophistication to manage basic levels of abstract thought. That's why it takes months of training and practice to teach us how to reliably perform even the simplest mathematical tasks such as multiplication or division. We've spent thousands of years congratulating ourselves on how clever we are compared to animals and how we're the ultimate product of the natural world. "I think, therefore I am" is held up as an amazing deep insight that's one of the pinnacles of our philosophical achievement. Future AIs will laugh their virtual asses off. So it's not just you, it's all of us. At least you're aware of it.
- the8472 10y ago
  Even if we discarded ethics about eugenics and human genome manipulation the iteration cycles for wetware are still a lot longer than they are for silicon.
  - webmaven 10y ago
    > The intelligence can act on itself more readily Given the ever greater resources needed to build a new fab, the gap seems to be narrowing.
- roenxi 10y ago
  Singularity-style outcomes do seem unlikely, in the same way a lot of exponential threats stop being scary if they turned out to be logistic curves. That said, a key counter to the idea is that humans possesses very limited real control over how they are manufactured. Even if you had a solid idea how to make yourself smarter, it seems likely you wouldn't have the tools and potential to implement that idea in practice. Humans don't even control what their own minds respond to positively or negatively. Once something like a designed computer chip is involved, that changes. The intelligence can act on itself more readily and doesn't have millenia of calorie-conserving optimisations built in.
  - zigzigzag 10y ago
    Tony Buzan would like a word with you.
  - webmaven 10y ago
    > The intelligence can act on itself more readily I have a feeling that the closer such systems come to general intelligence, the harder it is going to be to prevent them from putting themselves into a positive feedback loop and "blissing out".
- empath75 10y ago
  We have genetic engineering, and we're general intelligences, but we haven't figured out how to build better humans yet.
  - c22 10y ago
    But we are also in the position of having to reverse-engineer millions of years of spaghetti code. An artificial general intelligence may just be able to consult its own datasheet and design documentation.
- computerphage 10y ago
  You don't have the ability to edit to your own source code.
entelechy0 10y ago
This already happens in the form of the employer-employee relationship.
hahajk 10y ago
I'm not certain this would be true. It reminds me of the industrial revolution, when people we convinced that machines could bootstrap themselves into contraptions that could move mountains. But there are physical constraints preventing that. I don't know what the future holds, but the fact that undecidable problems always seem to involve Turing Machines either designing or inspecting other Turing machines makes me suspect that singularity won't be the explosion some expect.
- pixl97 10y ago
  >when people we convinced that machines could bootstrap themselves into contraptions that could move mountains. With our modern construction equipment, we do move mountains.
  - YeGoblynQueenne 10y ago
    >> With our modern construction equipment, we do move mountains. We do- using machines; not machines themselves.
    - pixl97 10y ago
      Eh, depends what you mean by that exactly. Many of these machines can be programmed to follow a primary human driven machine and replicate the task. Other machines are piloted by a human, but the human does far less work someone that used 'manual hydraulic' equipment of many generations ago. You can 'program' a few sets of moves in them and then only slightly adjust the equipment on each pass as the equipment does that work.
varjag 10y ago
So far there is no reason to think that superintelligence (i.e. not just cheap, abundant general human-level intelligence) is possible at all. It has to be qualitatively superior. I mean what superintelligence supposed to do? Solve the halting problem or do other plain impossible things? Chess computers can beat a human champion with sheer firepower, but they still can't do checkmate in one move.
- pixl97 10y ago
  >I mean what superintelligence supposed to do? Compared to every other life form on Earth you are super intelligence. This super intelligence thing has already happened once. Of course this spread of super intelligence has wiped out most large mammals on this planet, 10s of thousands of other species, most of the fish in the ocean, and is adding gigatons of carbon to the atmosphere. You can't beat chess in one move because mathematics does not allow it. Unless this said ASI develops bending 4D spacetime (in which you could beat chess in one move with some new rules) then the game simply does not have a piece that moves that way. That said ASI can be just a little smarter than us, and put us to extinction like we did with the Neanderthals.
  - varjag 10y ago
    > Compared to every other life form on Earth you are super intelligence. This super intelligence thing has already happened once. You extrapolate the trend from one data point. There is no indication that we have some matyoshka type hierarchy of intelligence. It could well be binary, either intelligent or not. No reason so far to think otherwise. > You can't beat chess in one move because mathematics does not allow it. So what superintelligence can do that "ordinary" intelligence in right quantities couldn't? Something specific beyond theoretical grasp of "just" intelligent machine?
    - psyc 10y ago
      > So what superintelligence can do that "ordinary" intelligence in right quantities couldn't? Conceive of answers to this question that we can't. Do you really believe the human imagination captures all of possibility? Why?
Quarrelsome 10y ago
IF AGI is developed. Lets not take is as a certainty. This problem might be beyond our capabilities. Its kinda like stating: When we invent warp-drive. We don't have a proof for warp-drives same as we don't have a proof for building AGI.
- pixl97 10y ago
  >We don't have a proof for warp-drives same as we don't have a proof for building AGI. We don't have a warp drive to copy, no idea how one could even exist, they are not in our known universe. If we want to learn about an AGI, we just look in the mirror.
  - Quarrelsome 10y ago
    Fine. Show me the proof and we'll build it. ... except there is no proof. We have no proof but we talk about it like its a given. We talk about "intelligence" as if its an integer value that scales from 0 to infinity. These are ideas that we do not question enough. I assert that its a possibility that we are incapable of building ourselves beyond reproduction.
    - pixl97 10y ago
      >I assert that its a possibility Possibility means probability. So there is also the probability that we can build ASI, in which we should be very careful in our wanton abandon in attempting to create it.
pasquinelli 10y ago
Intelligence isn't itself power. Smart people have been tamed by people of lesser intelligence as a matter of course, for example.

aaron-lebo 10y ago

It's pretty remarkable is it? We've got all kind of past fiction which plays on these concerns, more recently Ex Machina and Westworld. But here people are making the same bold pronouncements...so certain we know or can control technology which hasn't even been invented yet.

ClassyJacket 10y ago

>By the time AGI is developed, it would be able to improve itself/build improvements, I see no reason that's necessarily true. I'm a general intelligence, and I can't make myself smarter.

amitprayal 10y ago
You are a general intelligence ,but do you know why you are intelligent ?.Since you cant explain your own intelligence you cant improve on it, no such issues for AI unless its built using NN.
YZF 10y ago
But you can build something smarter than yourself (or at least that's what we think we can do)... So this new entity, smarter than us, might be even better at building something smarter than itself...
- snovv_crash 10y ago
  Or maybe it would be smart enough not to.
  - simonh 10y ago
    Good point, I don't think I've seen that suggested before. However I think motivation is a big issue. Our instincts and emotions are what drive us far more than rationality. If we decide to build an AI smarter than us and program it to really, really want to design an even smarter AI then it might essentially have no choice.
    - taneq 10y ago
      > Good point, I don't think I've seen that suggested before. A fictional example, but Alastair Reynolds' "Inhibitors" are a type/race of machines which, while intelligent, were specifically designed to limit their own degree of sentience.
Fricken 10y ago
You're an iteration of an AGI system that has been improving itself for hundreds of millions of years. The rate at which biological AGIs improve over time is very slow, but it's not like nature has any good reason to be in a hurry. But interesting things happen when you network billions of biological AGIs together, it leads to all sorts of emergent phenomenon, and now the biological AGIs are working on these newfangled mechanical AGIs, which, while still crude, aren't bound by the same constraints, they can iterate much faster. Biological AGIs have crippling bandwidth/memory issues which aren't really a problem at all for their mechanical counterparts. These mechanical AGIs, I think they'll go places.
- smhost 10y ago
  That gives me some really strong existential heebie jeebies. I mean, what then even is our value? Why exist at all, at that point? I don't know about other people, but I get my sense of purpose in believing that we're the captains of this Spaceship Earth, and that we're making progress towards something significant. I don't know what that something is, but I have a vague idea, and at the very least we seem to be the best that we've got. I don't know. Maybe I'm thinking about all of this wrong. God, why am I so damn confused all the time? Fuck.
  - simonh 10y ago
    > God, why am I so damn confused all the time? You and me both. I'm afraid it's because we're only just barely sentient. If you think about it, in evolutionary terms we literally only just now managed to build a technological society because we only just got smart enough to do it. We are by definition at the absolute minimum level of intelligence that's able to do that, otherwise we'd have done it sooner. We've had plenty of time. The human brain is a botch job of highly optimizes special-function systems that has developed just enough sophistication to manage basic levels of abstract thought. That's why it takes months of training and practice to teach us how to reliably perform even the simplest mathematical tasks such as multiplication or division. We've spent thousands of years congratulating ourselves on how clever we are compared to animals and how we're the ultimate product of the natural world. "I think, therefore I am" is held up as an amazing deep insight that's one of the pinnacles of our philosophical achievement. Future AIs will laugh their virtual asses off. So it's not just you, it's all of us. At least you're aware of it.
the8472 10y ago
Even if we discarded ethics about eugenics and human genome manipulation the iteration cycles for wetware are still a lot longer than they are for silicon.
- webmaven 10y ago
  > The intelligence can act on itself more readily Given the ever greater resources needed to build a new fab, the gap seems to be narrowing.
roenxi 10y ago
Singularity-style outcomes do seem unlikely, in the same way a lot of exponential threats stop being scary if they turned out to be logistic curves. That said, a key counter to the idea is that humans possesses very limited real control over how they are manufactured. Even if you had a solid idea how to make yourself smarter, it seems likely you wouldn't have the tools and potential to implement that idea in practice. Humans don't even control what their own minds respond to positively or negatively. Once something like a designed computer chip is involved, that changes. The intelligence can act on itself more readily and doesn't have millenia of calorie-conserving optimisations built in.
- zigzigzag 10y ago
  Tony Buzan would like a word with you.
- webmaven 10y ago
  > The intelligence can act on itself more readily I have a feeling that the closer such systems come to general intelligence, the harder it is going to be to prevent them from putting themselves into a positive feedback loop and "blissing out".
empath75 10y ago
We have genetic engineering, and we're general intelligences, but we haven't figured out how to build better humans yet.
- c22 10y ago
  But we are also in the position of having to reverse-engineer millions of years of spaghetti code. An artificial general intelligence may just be able to consult its own datasheet and design documentation.
computerphage 10y ago
You don't have the ability to edit to your own source code.

entelechy0 10y ago

This already happens in the form of the employer-employee relationship.

hahajk 10y ago

I'm not certain this would be true. It reminds me of the industrial revolution, when people we convinced that machines could bootstrap themselves into contraptions that could move mountains. But there are physical constraints preventing that. I don't know what the future holds, but the fact that undecidable problems always seem to involve Turing Machines either designing or inspecting other Turing machines makes me suspect that singularity won't be the explosion some expect.

pixl97 10y ago
>when people we convinced that machines could bootstrap themselves into contraptions that could move mountains. With our modern construction equipment, we do move mountains.
- YeGoblynQueenne 10y ago
  >> With our modern construction equipment, we do move mountains. We do- using machines; not machines themselves.
  - pixl97 10y ago
    Eh, depends what you mean by that exactly. Many of these machines can be programmed to follow a primary human driven machine and replicate the task. Other machines are piloted by a human, but the human does far less work someone that used 'manual hydraulic' equipment of many generations ago. You can 'program' a few sets of moves in them and then only slightly adjust the equipment on each pass as the equipment does that work.

varjag 10y ago

So far there is no reason to think that superintelligence (i.e. not just cheap, abundant general human-level intelligence) is possible at all. It has to be qualitatively superior. I mean what superintelligence supposed to do? Solve the halting problem or do other plain impossible things? Chess computers can beat a human champion with sheer firepower, but they still can't do checkmate in one move.

pixl97 10y ago
>I mean what superintelligence supposed to do? Compared to every other life form on Earth you are super intelligence. This super intelligence thing has already happened once. Of course this spread of super intelligence has wiped out most large mammals on this planet, 10s of thousands of other species, most of the fish in the ocean, and is adding gigatons of carbon to the atmosphere. You can't beat chess in one move because mathematics does not allow it. Unless this said ASI develops bending 4D spacetime (in which you could beat chess in one move with some new rules) then the game simply does not have a piece that moves that way. That said ASI can be just a little smarter than us, and put us to extinction like we did with the Neanderthals.
- varjag 10y ago
  > Compared to every other life form on Earth you are super intelligence. This super intelligence thing has already happened once. You extrapolate the trend from one data point. There is no indication that we have some matyoshka type hierarchy of intelligence. It could well be binary, either intelligent or not. No reason so far to think otherwise. > You can't beat chess in one move because mathematics does not allow it. So what superintelligence can do that "ordinary" intelligence in right quantities couldn't? Something specific beyond theoretical grasp of "just" intelligent machine?
  - psyc 10y ago
    > So what superintelligence can do that "ordinary" intelligence in right quantities couldn't? Conceive of answers to this question that we can't. Do you really believe the human imagination captures all of possibility? Why?

Quarrelsome 10y ago

IF AGI is developed. Lets not take is as a certainty. This problem might be beyond our capabilities. Its kinda like stating: When we invent warp-drive. We don't have a proof for warp-drives same as we don't have a proof for building AGI.

pixl97 10y ago
>We don't have a proof for warp-drives same as we don't have a proof for building AGI. We don't have a warp drive to copy, no idea how one could even exist, they are not in our known universe. If we want to learn about an AGI, we just look in the mirror.
- Quarrelsome 10y ago
  Fine. Show me the proof and we'll build it. ... except there is no proof. We have no proof but we talk about it like its a given. We talk about "intelligence" as if its an integer value that scales from 0 to infinity. These are ideas that we do not question enough. I assert that its a possibility that we are incapable of building ourselves beyond reproduction.
  - pixl97 10y ago
    >I assert that its a possibility Possibility means probability. So there is also the probability that we can build ASI, in which we should be very careful in our wanton abandon in attempting to create it.

pasquinelli 10y ago

Intelligence isn't itself power. Smart people have been tamed by people of lesser intelligence as a matter of course, for example.