3 ms·

I think you need to distinguish between AGI and the singularity. We know that nothing in physics prevents AGI, because humans exist. And AGI itself, even if it

I think you need to distinguish between AGI and the singularity. We know that nothing in physics prevents AGI, because humans exist. And AGI itself, even if it isn't entirely runaway, could pose a challenge. (I'd argue that corporations are a form of AGI that is already a major challenge for our societies.) The singularity as described by its acolytes, on the other hand, runs into the kind of physical limits you describe. Personally, I don't think it'll happen, and it's more likely that the period we're living in today will at some point in the future be called the singularity.

notahacker 8y ago
If we take humans as our model for AGI, it's worth noting that well-established networks of humans have proved efficient at ahem... deleting more-intelligent but potentially threatening human minds, that status and EQ has tended to prevail over IQ when it comes to manipulating humans into acting against their interests, that relatively close physical copies of human minds often have very dissimilar goals and abilities so self-replicating AGIs might ultimately be self-defeating... but above all that we've already got multiple threats to humanity in the form of human intelligences, augmented by technology that's often as dumb as it is deadly.
- vidarh 8y ago
  While that is true, also consider the destruction we have caused to other species. AGI's might not even want to harm us. They might simply not care enough to avoid hurting us by extension of seeking their own growth and survival. And while we have done immense damage to large groups of humans, none of it have come close to threatening the survival of the species, while continuing to increase our ability to unintentionally destroy other species. Humanity is not exactly an encouraging model for why we shouldn't fear AGI. If anything I very much hope we manage to find ways of preventing AGI from becoming too much like humans.

notahacker 8y ago

If we take humans as our model for AGI, it's worth noting that well-established networks of humans have proved efficient at ahem... deleting more-intelligent but potentially threatening human minds, that status and EQ has tended to prevail over IQ when it comes to manipulating humans into acting against their interests, that relatively close physical copies of human minds often have very dissimilar goals and abilities so self-replicating AGIs might ultimately be self-defeating... but above all that we've already got multiple threats to humanity in the form of human intelligences, augmented by technology that's often as dumb as it is deadly.

vidarh 8y ago
While that is true, also consider the destruction we have caused to other species. AGI's might not even want to harm us. They might simply not care enough to avoid hurting us by extension of seeking their own growth and survival. And while we have done immense damage to large groups of humans, none of it have come close to threatening the survival of the species, while continuing to increase our ability to unintentionally destroy other species. Humanity is not exactly an encouraging model for why we shouldn't fear AGI. If anything I very much hope we manage to find ways of preventing AGI from becoming too much like humans.