5 ms·

I always get the feeling AI is going to be like nuclear capability. Great reason to create it, but then once its made everyone wants to get rid of it.

JoshTriplett 10y ago
There's a huge difference, though. Creating a nuclear weapon encourages others to do the same. An AGI, if done right, would never allow the creation of a second one with conflicting values; there should be no second AGI.
- otoburb 10y ago
  Might never want to allow, but may not necessarily have a choice in the matter. Nobody really knows how singularity AGI fooms will play out.
  - daveguy 10y ago
    Probably the same way flying cars played out.
    - pixl97 10y ago
      Huh? We have flying cars. They are called planes, and they take a lot of maintenance to keep from falling out of the sky. But let's change the question up a bit... There are billions and billions of flying intelligences on this planet. Birds, insects, even mammals. Nature has already created that. We've created things that are even better at flying fast and carrying more weight. So simply looking at 'flying cars' and saying they didn't happen so AI can't happen is at the least, very ignorant. If nature can create something randomly, we can create something directed in a shorter period of time (well, we don't really have another 4 billion years to try). AGI is an eventuality.
      - argonaut 10y ago
        Saying AGI is an eventuality is as speculative as saying an alien invasion is an eventuality. We have evidence of intelligent species (ourselves) invading. We also have evidence spaceflight is possible.
        pixl97 10y ago
        GI already exists, AGI is only speculative if you believe that humans go extinct before they get complex. We can perform scientific experiments on GI at this time. An alien invasion is speculative at this point because there is no alien species on to which we can base any scientific refutation. It is not impossible, it is simply impossible to define any probably of occurrence.
  - JoshTriplett 10y ago
    > Might never want to allow, but may not necessarily have a choice in the matter. "if done right" glosses over quite a bit, yes.
- onion2k 10y ago
  Whether or not an AGI should have the capability to stop the creation of a second AGI is exactly the problem the article is considering.
- argonaut 10y ago
  You are mistaking an AGI for an artificial superintelligence (I might also add that the very concept of superintelligence is pure speculation - basic AGI at least can be grounded in replication of human brains). The first AGI will be closer to a low-IQ human than a Machiavellian super-optimizer.
  - xyience 10y ago
    I don't think there's much reason to be confident that the first AGI will be like a low-IQ human, at least for very long, even if it starts off as an emulated human brain. Machines have the huge advantage of faster materials (neurons are slow), perfect memory, perfect calculation, ability to scale horizontally, backups to restore in the event of bad self-modification experiments, and no need for things like sleep and food to take up its time from learning, improving itself, and acting upon the world.
    - argonaut 10y ago
      You've omitted the huge, overwhelmingly outweighing disadvantages of intelligence in machines, which is that they don't work. Given that research progress is incremental, there really isn't reason to believe we will jump from narrow AI to super-smart AGI, instead of narrow AI to dumb AGI to smart AGI to super-smart AGI.
      - xyience 10y ago
        There are historical examples of discontinuities, though I don't think the FOOM debate will be settled soon. It may be quite a while before we even get to "dumb AGI", but the hardest part of that is the G part. Right now "they don't work" is indistinguishable from "they don't exist", but if we get that G, I don't see how you could claim either. From there, even if we suppose it's another huge leap to get to true super-intelligence instead of a FOOM, the time to get to merely smart AGI, and indeed smarter-than-human AGI, would be short, if only for the basic advantages of a silicon machine substrate. If all we had were human brains running on silicon even, that would be enough to quickly reach superhuman general intelligence, even if not true super- (or perhaps ultra- as I.G. Good originally put it) intelligence that we expect for a Singularity event.

JoshTriplett 10y ago

There's a huge difference, though. Creating a nuclear weapon encourages others to do the same. An AGI, if done right, would never allow the creation of a second one with conflicting values; there should be no second AGI.

otoburb 10y ago
Might never want to allow, but may not necessarily have a choice in the matter. Nobody really knows how singularity AGI fooms will play out.
- daveguy 10y ago
  Probably the same way flying cars played out.
  - pixl97 10y ago
    Huh? We have flying cars. They are called planes, and they take a lot of maintenance to keep from falling out of the sky. But let's change the question up a bit... There are billions and billions of flying intelligences on this planet. Birds, insects, even mammals. Nature has already created that. We've created things that are even better at flying fast and carrying more weight. So simply looking at 'flying cars' and saying they didn't happen so AI can't happen is at the least, very ignorant. If nature can create something randomly, we can create something directed in a shorter period of time (well, we don't really have another 4 billion years to try). AGI is an eventuality.
    - argonaut 10y ago
      Saying AGI is an eventuality is as speculative as saying an alien invasion is an eventuality. We have evidence of intelligent species (ourselves) invading. We also have evidence spaceflight is possible.
      - pixl97 10y ago
        GI already exists, AGI is only speculative if you believe that humans go extinct before they get complex. We can perform scientific experiments on GI at this time. An alien invasion is speculative at this point because there is no alien species on to which we can base any scientific refutation. It is not impossible, it is simply impossible to define any probably of occurrence.
- JoshTriplett 10y ago
  > Might never want to allow, but may not necessarily have a choice in the matter. "if done right" glosses over quite a bit, yes.
onion2k 10y ago
Whether or not an AGI should have the capability to stop the creation of a second AGI is exactly the problem the article is considering.
argonaut 10y ago
You are mistaking an AGI for an artificial superintelligence (I might also add that the very concept of superintelligence is pure speculation - basic AGI at least can be grounded in replication of human brains). The first AGI will be closer to a low-IQ human than a Machiavellian super-optimizer.
- xyience 10y ago
  I don't think there's much reason to be confident that the first AGI will be like a low-IQ human, at least for very long, even if it starts off as an emulated human brain. Machines have the huge advantage of faster materials (neurons are slow), perfect memory, perfect calculation, ability to scale horizontally, backups to restore in the event of bad self-modification experiments, and no need for things like sleep and food to take up its time from learning, improving itself, and acting upon the world.
  - argonaut 10y ago
    You've omitted the huge, overwhelmingly outweighing disadvantages of intelligence in machines, which is that they don't work. Given that research progress is incremental, there really isn't reason to believe we will jump from narrow AI to super-smart AGI, instead of narrow AI to dumb AGI to smart AGI to super-smart AGI.
    - xyience 10y ago
      There are historical examples of discontinuities, though I don't think the FOOM debate will be settled soon. It may be quite a while before we even get to "dumb AGI", but the hardest part of that is the G part. Right now "they don't work" is indistinguishable from "they don't exist", but if we get that G, I don't see how you could claim either. From there, even if we suppose it's another huge leap to get to true super-intelligence instead of a FOOM, the time to get to merely smart AGI, and indeed smarter-than-human AGI, would be short, if only for the basic advantages of a silicon machine substrate. If all we had were human brains running on silicon even, that would be enough to quickly reach superhuman general intelligence, even if not true super- (or perhaps ultra- as I.G. Good originally put it) intelligence that we expect for a Singularity event.