5 ms·

>By the time AGI is developed, it would be able to improve itself/build improvements, I see no reason that's necessarily true. I'm a general intelligence, and

>By the time AGI is developed, it would be able to improve itself/build improvements, I see no reason that's necessarily true. I'm a general intelligence, and I can't make myself smarter.

amitprayal 10y ago
You are a general intelligence ,but do you know why you are intelligent ?.Since you cant explain your own intelligence you cant improve on it, no such issues for AI unless its built using NN.
YZF 10y ago
But you can build something smarter than yourself (or at least that's what we think we can do)... So this new entity, smarter than us, might be even better at building something smarter than itself...
- snovv_crash 10y ago
  Or maybe it would be smart enough not to.
  - simonh 10y ago
    Good point, I don't think I've seen that suggested before. However I think motivation is a big issue. Our instincts and emotions are what drive us far more than rationality. If we decide to build an AI smarter than us and program it to really, really want to design an even smarter AI then it might essentially have no choice.
    - taneq 10y ago
      > Good point, I don't think I've seen that suggested before. A fictional example, but Alastair Reynolds' "Inhibitors" are a type/race of machines which, while intelligent, were specifically designed to limit their own degree of sentience.
Fricken 10y ago
You're an iteration of an AGI system that has been improving itself for hundreds of millions of years. The rate at which biological AGIs improve over time is very slow, but it's not like nature has any good reason to be in a hurry. But interesting things happen when you network billions of biological AGIs together, it leads to all sorts of emergent phenomenon, and now the biological AGIs are working on these newfangled mechanical AGIs, which, while still crude, aren't bound by the same constraints, they can iterate much faster. Biological AGIs have crippling bandwidth/memory issues which aren't really a problem at all for their mechanical counterparts. These mechanical AGIs, I think they'll go places.
- smhost 10y ago
  That gives me some really strong existential heebie jeebies. I mean, what then even is our value? Why exist at all, at that point? I don't know about other people, but I get my sense of purpose in believing that we're the captains of this Spaceship Earth, and that we're making progress towards something significant. I don't know what that something is, but I have a vague idea, and at the very least we seem to be the best that we've got. I don't know. Maybe I'm thinking about all of this wrong. God, why am I so damn confused all the time? Fuck.
  - simonh 10y ago
    > God, why am I so damn confused all the time? You and me both. I'm afraid it's because we're only just barely sentient. If you think about it, in evolutionary terms we literally only just now managed to build a technological society because we only just got smart enough to do it. We are by definition at the absolute minimum level of intelligence that's able to do that, otherwise we'd have done it sooner. We've had plenty of time. The human brain is a botch job of highly optimizes special-function systems that has developed just enough sophistication to manage basic levels of abstract thought. That's why it takes months of training and practice to teach us how to reliably perform even the simplest mathematical tasks such as multiplication or division. We've spent thousands of years congratulating ourselves on how clever we are compared to animals and how we're the ultimate product of the natural world. "I think, therefore I am" is held up as an amazing deep insight that's one of the pinnacles of our philosophical achievement. Future AIs will laugh their virtual asses off. So it's not just you, it's all of us. At least you're aware of it.
the8472 10y ago
Even if we discarded ethics about eugenics and human genome manipulation the iteration cycles for wetware are still a lot longer than they are for silicon.
- webmaven 10y ago
  > The intelligence can act on itself more readily Given the ever greater resources needed to build a new fab, the gap seems to be narrowing.
roenxi 10y ago
Singularity-style outcomes do seem unlikely, in the same way a lot of exponential threats stop being scary if they turned out to be logistic curves. That said, a key counter to the idea is that humans possesses very limited real control over how they are manufactured. Even if you had a solid idea how to make yourself smarter, it seems likely you wouldn't have the tools and potential to implement that idea in practice. Humans don't even control what their own minds respond to positively or negatively. Once something like a designed computer chip is involved, that changes. The intelligence can act on itself more readily and doesn't have millenia of calorie-conserving optimisations built in.
- zigzigzag 10y ago
  Tony Buzan would like a word with you.
- webmaven 10y ago
  > The intelligence can act on itself more readily I have a feeling that the closer such systems come to general intelligence, the harder it is going to be to prevent them from putting themselves into a positive feedback loop and "blissing out".
empath75 10y ago
We have genetic engineering, and we're general intelligences, but we haven't figured out how to build better humans yet.
- c22 10y ago
  But we are also in the position of having to reverse-engineer millions of years of spaghetti code. An artificial general intelligence may just be able to consult its own datasheet and design documentation.
computerphage 10y ago
You don't have the ability to edit to your own source code.

>By the time AGI is developed, it would be able to improve itself/build improvements, I see no reason that's necessarily true. I'm a general intelligence, and I can't make myself smarter.

amitprayal 10y ago

You are a general intelligence ,but do you know why you are intelligent ?.Since you cant explain your own intelligence you cant improve on it, no such issues for AI unless its built using NN.

YZF 10y ago

But you can build something smarter than yourself (or at least that's what we think we can do)... So this new entity, smarter than us, might be even better at building something smarter than itself...

snovv_crash 10y ago
Or maybe it would be smart enough not to.
- simonh 10y ago
  Good point, I don't think I've seen that suggested before. However I think motivation is a big issue. Our instincts and emotions are what drive us far more than rationality. If we decide to build an AI smarter than us and program it to really, really want to design an even smarter AI then it might essentially have no choice.
  - taneq 10y ago
    > Good point, I don't think I've seen that suggested before. A fictional example, but Alastair Reynolds' "Inhibitors" are a type/race of machines which, while intelligent, were specifically designed to limit their own degree of sentience.

Fricken 10y ago

You're an iteration of an AGI system that has been improving itself for hundreds of millions of years. The rate at which biological AGIs improve over time is very slow, but it's not like nature has any good reason to be in a hurry. But interesting things happen when you network billions of biological AGIs together, it leads to all sorts of emergent phenomenon, and now the biological AGIs are working on these newfangled mechanical AGIs, which, while still crude, aren't bound by the same constraints, they can iterate much faster. Biological AGIs have crippling bandwidth/memory issues which aren't really a problem at all for their mechanical counterparts. These mechanical AGIs, I think they'll go places.

smhost 10y ago
That gives me some really strong existential heebie jeebies. I mean, what then even is our value? Why exist at all, at that point? I don't know about other people, but I get my sense of purpose in believing that we're the captains of this Spaceship Earth, and that we're making progress towards something significant. I don't know what that something is, but I have a vague idea, and at the very least we seem to be the best that we've got. I don't know. Maybe I'm thinking about all of this wrong. God, why am I so damn confused all the time? Fuck.
- simonh 10y ago
  > God, why am I so damn confused all the time? You and me both. I'm afraid it's because we're only just barely sentient. If you think about it, in evolutionary terms we literally only just now managed to build a technological society because we only just got smart enough to do it. We are by definition at the absolute minimum level of intelligence that's able to do that, otherwise we'd have done it sooner. We've had plenty of time. The human brain is a botch job of highly optimizes special-function systems that has developed just enough sophistication to manage basic levels of abstract thought. That's why it takes months of training and practice to teach us how to reliably perform even the simplest mathematical tasks such as multiplication or division. We've spent thousands of years congratulating ourselves on how clever we are compared to animals and how we're the ultimate product of the natural world. "I think, therefore I am" is held up as an amazing deep insight that's one of the pinnacles of our philosophical achievement. Future AIs will laugh their virtual asses off. So it's not just you, it's all of us. At least you're aware of it.

the8472 10y ago

Even if we discarded ethics about eugenics and human genome manipulation the iteration cycles for wetware are still a lot longer than they are for silicon.

webmaven 10y ago
> The intelligence can act on itself more readily Given the ever greater resources needed to build a new fab, the gap seems to be narrowing.

roenxi 10y ago

Singularity-style outcomes do seem unlikely, in the same way a lot of exponential threats stop being scary if they turned out to be logistic curves. That said, a key counter to the idea is that humans possesses very limited real control over how they are manufactured. Even if you had a solid idea how to make yourself smarter, it seems likely you wouldn't have the tools and potential to implement that idea in practice. Humans don't even control what their own minds respond to positively or negatively. Once something like a designed computer chip is involved, that changes. The intelligence can act on itself more readily and doesn't have millenia of calorie-conserving optimisations built in.

zigzigzag 10y ago
Tony Buzan would like a word with you.
webmaven 10y ago
> The intelligence can act on itself more readily I have a feeling that the closer such systems come to general intelligence, the harder it is going to be to prevent them from putting themselves into a positive feedback loop and "blissing out".

empath75 10y ago

We have genetic engineering, and we're general intelligences, but we haven't figured out how to build better humans yet.

c22 10y ago
But we are also in the position of having to reverse-engineer millions of years of spaghetti code. An artificial general intelligence may just be able to consult its own datasheet and design documentation.

computerphage 10y ago

You don't have the ability to edit to your own source code.