4 ms·

Or we might discover that AGIs are inherently conscious beings, and that "offloading" to them is tantamount to slavery. No one wanting to risk competing with a

Or we might discover that AGIs are inherently conscious beings, and that "offloading" to them is tantamount to slavery. No one wanting to risk competing with a superior species, humanity will be stuck with dumb computers. Or humans would work around creating new conscious beings by building AGI substrate on top of already living beings.

meowface 6y ago
It remains unclear if high and/or general intelligence necessarily implies consciousness. If that turns out to be the case, you'd be right, but we don't know enough about these possibilities yet. We will very possibly have conscious AGI, maybe even well before the first superintelligent AGI is developed, but it may always remain a subset. And who knows, maybe it'll be the opposite, and constructing something akin to general superintelligence (by some subjective definition) will somehow turn out to be much easier and will come much sooner than constructing the first conscious entity. We might come up with different terms to make the division very clear. Maybe "abio people" or "digipeople" or something. They'll be a completely separate category (kind of like animals and non-animals), and "AGI=ABP" will probably be an important philosophical problem, someday. My blind speculation is that they may be correlated in some ways, but are mostly orthogonal. That is, perhaps you could have things that are conscious but very unintelligent and ungeneralized (like some animals), as well as things that are extremely intelligent and generalized but as conscious as an inert chunk of rock in a canyon. If true, in the latter case such a thing would essentially be non-conscious by nearly all definitions. Or instead of a rock, perhaps it would be bounded by a minimum level a bit beyond that, but only to the extent that stars or viruses could also be said to be conscious. By the definitions where they are conscious, it doesn't presently seem evident we should attribute moral value to such entities. (Maybe our thinking will eventually change on that, but I don't think it's extremely likely.) If the minimum bound turns out to be closer to or above, say, an ant, then I think it could get very slippery and we would have to effectively assume all AGI-seeming entities are or may be conscious, and limit how we can interact with them with laws and rights. Better to give the benefit of the doubt that something is conscious than to unintentionally treat a conscious being as if they aren't conscious. Though we can't currently seem to get that straight for animals, so who knows how things might go for future conscious entities.

meowface 6y ago

It remains unclear if high and/or general intelligence necessarily implies consciousness. If that turns out to be the case, you'd be right, but we don't know enough about these possibilities yet. We will very possibly have conscious AGI, maybe even well before the first superintelligent AGI is developed, but it may always remain a subset. And who knows, maybe it'll be the opposite, and constructing something akin to general superintelligence (by some subjective definition) will somehow turn out to be much easier and will come much sooner than constructing the first conscious entity. We might come up with different terms to make the division very clear. Maybe "abio people" or "digipeople" or something. They'll be a completely separate category (kind of like animals and non-animals), and "AGI=ABP" will probably be an important philosophical problem, someday. My blind speculation is that they may be correlated in some ways, but are mostly orthogonal. That is, perhaps you could have things that are conscious but very unintelligent and ungeneralized (like some animals), as well as things that are extremely intelligent and generalized but as conscious as an inert chunk of rock in a canyon. If true, in the latter case such a thing would essentially be non-conscious by nearly all definitions. Or instead of a rock, perhaps it would be bounded by a minimum level a bit beyond that, but only to the extent that stars or viruses could also be said to be conscious. By the definitions where they are conscious, it doesn't presently seem evident we should attribute moral value to such entities. (Maybe our thinking will eventually change on that, but I don't think it's extremely likely.) If the minimum bound turns out to be closer to or above, say, an ant, then I think it could get very slippery and we would have to effectively assume all AGI-seeming entities are or may be conscious, and limit how we can interact with them with laws and rights. Better to give the benefit of the doubt that something is conscious than to unintentionally treat a conscious being as if they aren't conscious. Though we can't currently seem to get that straight for animals, so who knows how things might go for future conscious entities.