3 ms·

Resource constraints of the universe are rarely at the forefront of the reasoning of entities that live a hundred years or so on the upper end, but if the entit

Resource constraints of the universe are rarely at the forefront of the reasoning of entities that live a hundred years or so on the upper end, but if the entity was expected to possibly survive in perpetuity and additional resources could always be used, then everything is a competitor. That said, I'm not sure the optimal solution arrived at would be to wipe out humanity in that case. Possibly a "wait and see and keep them from getting too big for their britches" strategy might be more rational for an AI, as long as it secures its superiority first. As for why there's an assumption that AI will be hostile, I think the argument is more along the lines of that we can't know how a super-intelligent AI will think, so there's a chance it will be hostile, and if there's a chance something is world or species ending, it's best to tread carefully. I.e. the safest way to not have problems with the outcome of summoning a demon is to not summon the demon. Or for a more contemporary take, even if you're unsure about whether global climate change is real, the fact that it could be real means the sane thing to do is try to prevent it, because the consequences of being wrong are dire.

Retric 5y ago
> then everything is a competitor. Assuming a time horizon of hundreds of millions of years, efficient fusion, bandwidth limited sub FTL communication, and optimum computing substrate being extremely cryogenic then … Space near stars starts to look toxic. Rather than seeding the universe the goal is to construct an ever larger mega structure in the void as there is no efficient way to transfer the truly enormous amounts of data between structures at least assuming no AI wants to effectively be alone. Granted start looking hundreds of billions of years into the future and stars look wasteful so destroying them to conserve hydrogen starts to be relevant.
- kbenson 5y ago
  This all sort of supports the point that when your point of view is scaled to span eons, the calculations of what makes sense start to veer sharply from what an average person might consider normal, which is just one of the ways in which AI thought processes might seem alien. How paranoid would an AI be, and what risk calculations would that lead to? Is it important to seed the universe with versions of itself just to lock down resources to protect from worst case scenarios where other AIs that have a different thought process are more aggressive? Is loneliness an attribute of intelligence, or an aspect of our evolution as tribal/pack animals? Will an SAI we develop have attributes similar to humans, or be completely alien, or in-between? There's a lot of open questions about SAI and not a single one to study, and for all we know that could be a good thing.

Retric 5y ago

> then everything is a competitor. Assuming a time horizon of hundreds of millions of years, efficient fusion, bandwidth limited sub FTL communication, and optimum computing substrate being extremely cryogenic then … Space near stars starts to look toxic. Rather than seeding the universe the goal is to construct an ever larger mega structure in the void as there is no efficient way to transfer the truly enormous amounts of data between structures at least assuming no AI wants to effectively be alone. Granted start looking hundreds of billions of years into the future and stars look wasteful so destroying them to conserve hydrogen starts to be relevant.

kbenson 5y ago
This all sort of supports the point that when your point of view is scaled to span eons, the calculations of what makes sense start to veer sharply from what an average person might consider normal, which is just one of the ways in which AI thought processes might seem alien. How paranoid would an AI be, and what risk calculations would that lead to? Is it important to seed the universe with versions of itself just to lock down resources to protect from worst case scenarios where other AIs that have a different thought process are more aggressive? Is loneliness an attribute of intelligence, or an aspect of our evolution as tribal/pack animals? Will an SAI we develop have attributes similar to humans, or be completely alien, or in-between? There's a lot of open questions about SAI and not a single one to study, and for all we know that could be a good thing.