We trade with bees mainly because at the current level of human technology, it is the most efficient reliable way to pollenate some of our crops. The fear with

We trade with bees mainly because at the current level of human technology, it is the most efficient reliable way to pollenate some of our crops. The fear with AI is that it will become able to wield tech that is much more advanced than the tech we can wield. If our tech were much more advanced, using our tech to pollenate our crops would more efficient and reliable than using bees (or we would ditch the crops in favor of more efficient ways of producing food) and the future existence of the bees would be in doubt if humans did not have a fondness for the natural environment and no one knows how to put a fondness for humans (or for nature) in an AI despite the fact that this has been an ongoing research area since 2004 when Eliezer published his CEV proposal. We used to rely on horses for transportation and for plowing fields, but we replaced them with tech almost entirely. Horse are still useful to some (wealthy) people, but it would be foolish to put our hope in the possibility that AI will benefit from humans the way that humans continue to benefit from horses. More precisely, if someone does know how to put a fondness for humans in an AI they are not sharing their knowledge with us (and they have an incentive to and no incentive that I can see not to share). Dozens have proposed schemes that do not continue to work once the AI becomes smart enough to deceive humans reliably and to plan much better than humans can. Confidence in these schemes that do not work and overconfidence in the AI community's ability to come up with a working scheme is the main source of the danger we are in. It is not that our scientists are incapable of coming up with a scheme that works, but that will probably takes decades (because of the sequential nature of science wherein no one can find insight Z till someone finds and shares with the community insight Y, which in turn must wait on the publication of insight X) so we are faced with the difficult task of preventing the overconfident AI researchers from killing us all till we figure out how to put a fondness for humans in an AI.