3 ms·

> No one's trying to eliminate anyone Probably not any consolation to the Neanderthal or any other animal species that is out-competed in nature. Doesn’t have

> No one's trying to eliminate anyone Probably not any consolation to the Neanderthal or any other animal species that is out-competed in nature. Doesn’t have to be malevolent or intentional for a more capable species to render others extinct. (Indeed that would seem to be the base case, historically speaking.) of course, symbiosis is possible; we could be puppies to the AI. Many people find disempowerment to be almost as bad a prospect as extinction though. > The issue is whether or not we can build a machine capable of even understanding the directive. Asimov’s laws are an ok starting point for a moral code, but nobody has the slightest idea how to implement them. The issue is not “understanding” them, it is baking them into the fabric of a mind so deeply that they cannot be broken. This requires mind engineering, a deep understanding of mind-space, and we are far closer to stumbling into AGI through brute force scaling (as nature did) than to understanding mind-engineering deeply enough to embed “laws”. > it's not clear why an artificial mind would pursue this as a goal There are a couple stock answers here. At the highest level though, why not? Humans have pursued goals of global domination, eradication, genocide. If only one in a billion minds has a defect (or is prompt-hijacked) to start power-seeking, it could still be bad; AI is not bound by human limitation to inhabit one body and can in principle expand to occupy all the compute it can secure. Or if you prefer, why would a more intelligent entity agree to be commanded by a less intelligent one? The base case seem to me that ASI will not remain under our control, unless we very carefully sculpt its mind to align with our interests. Another simple argument is: because someone will build or prompt-hijack such a mind. People have already created “take over the world” agents with GPT-3, as a joke (I assume). There are enough people willing to take such an action that we’ll need defenses in place. I think it’s true that the more ASI entities in existence, the less likely one is to fully take over. However there are esoteric decision theory reasons to believe that digital minds might be able to bargain/negotiate more effectively than biological minds (source code / mind state transparency etc) and so you might expect the ASIs to eventually collaborate to take control. Personally I don’t think that full-extinction is the most likely outcome, just that it’s completely plausible if we get unlucky and it’s very hard to prove safety in advance. And there are plenty of disempowering outcomes that seem quite bad too vs. the counterfactual.

XorNot 3y ago
> Probably not any consolation to the Neanderthal or any other animal species that is out-competed in nature. It is not clear from the geological record if the Neanderthal was "outcompeted" except in the broadest possible terms - i.e. a unique species of Neanderthal no longer exists genetically. But that doesn't tell us what happened in the interim - i.e. modern day humans can have up to 4% neandertal origin DNA[1]. So, did "humans" wipe out neanderthals? Or did human plan genetics simply become the more dominant type, no malevolent will required, and does that matter? Because for a whole lot of people - myself included - the explicit goal of technology like AI is something which from a fossil record would look like human extinction - that's the whole idea of transhumanism, that ultimately we would become something sharing perhaps only a glancing similarity with our ancestors. All the rest of the problems you talk about suffer from a lack of detail - namely, they assume a quantum leap in technological capability with no intervening steps - i.e. there is a single AI, or that AIs are better negotiators or what have you - as soon as you get into the details, the threat model proposes evaporates because we just don't know - the technology doesn't exist (i.e. it would all be a lot more interesting if fully automated production lines were already around, but we're nowhere near that). [1] https://humanorigins.si.edu/evidence/genetics/ancient-dna-and-neanderthals https://humanorigins.si.edu/evidence/genetics/ancient-dna-an...

XorNot 3y ago

> Probably not any consolation to the Neanderthal or any other animal species that is out-competed in nature. It is not clear from the geological record if the Neanderthal was "outcompeted" except in the broadest possible terms - i.e. a unique species of Neanderthal no longer exists genetically. But that doesn't tell us what happened in the interim - i.e. modern day humans can have up to 4% neandertal origin DNA[1]. So, did "humans" wipe out neanderthals? Or did human plan genetics simply become the more dominant type, no malevolent will required, and does that matter? Because for a whole lot of people - myself included - the explicit goal of technology like AI is something which from a fossil record would look like human extinction - that's the whole idea of transhumanism, that ultimately we would become something sharing perhaps only a glancing similarity with our ancestors. All the rest of the problems you talk about suffer from a lack of detail - namely, they assume a quantum leap in technological capability with no intervening steps - i.e. there is a single AI, or that AIs are better negotiators or what have you - as soon as you get into the details, the threat model proposes evaporates because we just don't know - the technology doesn't exist (i.e. it would all be a lot more interesting if fully automated production lines were already around, but we're nowhere near that). [1] https://humanorigins.si.edu/evidence/genetics/ancient-dna-and-neanderthals https://humanorigins.si.edu/evidence/genetics/ancient-dna-an...