3 ms·

I think predicting what things will be like in the next 200 years is near-impossible. Maybe even the next 100. For the next 1000, I think the unknown unknowns a

I think predicting what things will be like in the next 200 years is near-impossible. Maybe even the next 100. For the next 1000, I think the unknown unknowns are so numerous that people alive then would look back and just laugh at 99.9% of predictions, like we do over most of what people predicted in the year 1020. Humans might have merged so thoroughly with some form of technology that it won't make sense to talk about humanity as a concept. I think natural language/VR programming would be considered extremely primitive then; maybe that's something we'll see over the next 100/200 years, but I don't think the next 1000. If humans still exist and remain as mostly independent biological entities (perhaps due to a deliberate decision to minimize merging for ethical/philosophical reasons, even if it may be feasible), I think AGIs will be doing the vast majority of all programming. Maybe there'll be a bit of near-instant "compilation" of human thoughts into a wide variety of applications, enabled by neural interfaces, but I think they'll mostly be toys or just things we make for our own entertainment/amusement. (This might even happen in this century or the early 2100s; who knows.) I think it'll be like toddlers fingerpainting compared to what the AGIs will be concocting and running. Maybe at best we'll sometimes be like middle managers, giving an AGI network extremely high-level requirements and some visual/neural examples/diagrams which they'll work with and adjust to deliver something close to what our true intentions were. Or maybe it won't even make sense to think of them that way, and instead we'll just consider them like extra brain lobes we automatically offload difficult stuff to, with bandwidth equal to or greater than our current intra-brain bandwidth - even though they won't be physically inside of our brains in any way.

rhn_mk1 6y ago
Or we might discover that AGIs are inherently conscious beings, and that "offloading" to them is tantamount to slavery. No one wanting to risk competing with a superior species, humanity will be stuck with dumb computers. Or humans would work around creating new conscious beings by building AGI substrate on top of already living beings.
- meowface 6y ago
  It remains unclear if high and/or general intelligence necessarily implies consciousness. If that turns out to be the case, you'd be right, but we don't know enough about these possibilities yet. We will very possibly have conscious AGI, maybe even well before the first superintelligent AGI is developed, but it may always remain a subset. And who knows, maybe it'll be the opposite, and constructing something akin to general superintelligence (by some subjective definition) will somehow turn out to be much easier and will come much sooner than constructing the first conscious entity. We might come up with different terms to make the division very clear. Maybe "abio people" or "digipeople" or something. They'll be a completely separate category (kind of like animals and non-animals), and "AGI=ABP" will probably be an important philosophical problem, someday. My blind speculation is that they may be correlated in some ways, but are mostly orthogonal. That is, perhaps you could have things that are conscious but very unintelligent and ungeneralized (like some animals), as well as things that are extremely intelligent and generalized but as conscious as an inert chunk of rock in a canyon. If true, in the latter case such a thing would essentially be non-conscious by nearly all definitions. Or instead of a rock, perhaps it would be bounded by a minimum level a bit beyond that, but only to the extent that stars or viruses could also be said to be conscious. By the definitions where they are conscious, it doesn't presently seem evident we should attribute moral value to such entities. (Maybe our thinking will eventually change on that, but I don't think it's extremely likely.) If the minimum bound turns out to be closer to or above, say, an ant, then I think it could get very slippery and we would have to effectively assume all AGI-seeming entities are or may be conscious, and limit how we can interact with them with laws and rights. Better to give the benefit of the doubt that something is conscious than to unintentionally treat a conscious being as if they aren't conscious. Though we can't currently seem to get that straight for animals, so who knows how things might go for future conscious entities.

rhn_mk1 6y ago

Or we might discover that AGIs are inherently conscious beings, and that "offloading" to them is tantamount to slavery. No one wanting to risk competing with a superior species, humanity will be stuck with dumb computers. Or humans would work around creating new conscious beings by building AGI substrate on top of already living beings.

meowface 6y ago
It remains unclear if high and/or general intelligence necessarily implies consciousness. If that turns out to be the case, you'd be right, but we don't know enough about these possibilities yet. We will very possibly have conscious AGI, maybe even well before the first superintelligent AGI is developed, but it may always remain a subset. And who knows, maybe it'll be the opposite, and constructing something akin to general superintelligence (by some subjective definition) will somehow turn out to be much easier and will come much sooner than constructing the first conscious entity. We might come up with different terms to make the division very clear. Maybe "abio people" or "digipeople" or something. They'll be a completely separate category (kind of like animals and non-animals), and "AGI=ABP" will probably be an important philosophical problem, someday. My blind speculation is that they may be correlated in some ways, but are mostly orthogonal. That is, perhaps you could have things that are conscious but very unintelligent and ungeneralized (like some animals), as well as things that are extremely intelligent and generalized but as conscious as an inert chunk of rock in a canyon. If true, in the latter case such a thing would essentially be non-conscious by nearly all definitions. Or instead of a rock, perhaps it would be bounded by a minimum level a bit beyond that, but only to the extent that stars or viruses could also be said to be conscious. By the definitions where they are conscious, it doesn't presently seem evident we should attribute moral value to such entities. (Maybe our thinking will eventually change on that, but I don't think it's extremely likely.) If the minimum bound turns out to be closer to or above, say, an ant, then I think it could get very slippery and we would have to effectively assume all AGI-seeming entities are or may be conscious, and limit how we can interact with them with laws and rights. Better to give the benefit of the doubt that something is conscious than to unintentionally treat a conscious being as if they aren't conscious. Though we can't currently seem to get that straight for animals, so who knows how things might go for future conscious entities.