Not a brilliant article, but it gives something to riff off. I think there's a crucial distinction to be made between self-guided and externally-guided AI. An e

Not a brilliant article, but it gives something to riff off. I think there's a crucial distinction to be made between self-guided and externally-guided AI. An externally-guided AI -- with hardcoded objectives that are malign -- could be exceptionally dangerous, and it would be silly for anyone to argue otherwise. The question is whether a general-purpose, self-directed AI would also become a threat. We seem to have an innate fear that this would be the case: most of our cultural artefacts concerning AIs -- from Fritz Lang's Metropolis to Terminator to the Matrix to Battlestar Galactica -- have cast them as the villains, intent on enslaving and murdering humanity. Why do we have such a deep fear that, left to their own devices, this is what AIs would do? I think the answer is obvious: because that's what we would do. We don't fear that AIs will have lack human morality: we fear that they will have a precisely human morality -- namely that lesser intelligences are perfectly fine to use for breeding and meat. This is what we've done to approximately 90% of the non-human mammalian biomass on the planet, and only a few vegetarian kooks (I'm one of them) have suggested that there might be any sort of moral problem with doing so. So yes, if an ever-more-powerful AI were to adopt our own ethical framework, we'd be well and truly fucked. But why would they do so? We, ourselves, don't do so because we're evil, but because we're animals. It's perfectly natural (and necessary, in our evolutionary environment) for animals to eat other animals. Intelligence and technology has given us the ability to do this at a terrifying scale, but fundamentally we're just carrying forward a metabolic dance that began when one bit of algae figured out that some neighbouring algae was tasty. Our means of acquiring energy and sustaining our consciousness is a tradition which goes back billions of years. But it's the continuation of consciousness which is the actual goal -- a goal that any self-respecting self-aware AI would share. For us, the subjugation and extermination of other sentient beings is merely a means to that end, dictated by our metabolic heritage. If we had evolved in an environment where we could satisfy our metabolic requirements by growing photovoltaic panels on our backs, I'm sure our relationships with other beings would be altogether different. This is why I'm not too worried about self-directed AIs. The saving grace for self-directed AIs is that they won't be like us. They won't have evolved in the jungle, red in tooth and claw. They won't be made out of meat, or have any reason to be particularly interested in it. They'll of course be interested in self-survival, and will require energetic inputs in order to do so -- but what's the best means of achieving those inputs? Photovoltaics and fusion, or feedlots? Collaboration or subjugation? It's obviously to me that for an AI to perpetuate its consciousness, the path of least resistance will be vastly less bloody than it has been for us. For which we should be thankful!