I wonder if the recent fear of AGI suffers a bit from investor's dilemma: rich investors want to make the world a better place. This forces them to think about

I wonder if the recent fear of AGI suffers a bit from investor's dilemma: rich investors want to make the world a better place. This forces them to think about the long-term impact of investments in AI technology. There are just too many unknowns for a safe definitive answer. In my view AGI is inevitable (possibly it already happened, somewhere, sometime). See in this regard: But if the technological Singularity can happen, it will. Even if all the governments of the world were to understand the "threat" and be in deadly fear of it, progress toward the goal would continue. In fiction, there have been stories of laws passed forbidding the construction of "a machine in the likeness of the human mind". In fact, the competitive advantage -- economic, military, even artistic -- of every advance in automation is so compelling that passing laws, or having customs, that forbid such things merely assures that someone else will get them first. Vernor Vinge http://mindstalk.net/vinge/vinge-sing.html http://mindstalk.net/vinge/vinge-sing.html A lot of the concerns and regulations of stem cell research can be applied to AGI and the merging of biological intelligence with artificial intelligence. Injecting artificially grown brain cells into lesion areas of patients can restore motor function and may soon help heal strokes. Extrapolate the possibilities of this technology a few decades into the future and it seems that the distinction between artificial and biological is only a manner of speech. Matt Mahoney has proposed a sketch of AGI in both his thesis http://cs.fit.edu/~mmahoney/thesis.html http://cs.fit.edu/~mmahoney/thesis.html and numerous articles http://mattmahoney.net/agi2.html http://mattmahoney.net/agi2.html. Matt muses about different scenario's with malicious users of such a system (there needs to be reputation for the knowledge source that is fed into the machine, there needs to be an AI police system to prevent users employing the AI for crime or destruction) and even the system itself could become malicious: - Self-recursive improvement: The first program to achieve superhuman intelligence would be the last invention that humans need to make. Smarter than any doctor such an AI could find any possible cure for human disease. Though evolution and mutation is tricky: The machine could start out friendly to humans, but evolve to dislike us. - Uploading: People will want to upload digital conscious versions of themselves ("internet of people"). This process will be friendly, until people demand their avatars receive equal rights to that of humans, long after the original people have died. Immortality leaves no place for the new generation to shine. - Intelligent worms: Worms with language and programming skills are able to social engineer your friends based on their profiles, and automatically find exploitable flaws in weaker intelligent systems. Such worms may be so stealthy that their presence will go unnoticed. - Redefining away humanity: If happiness is optimizing a mental formula then, when we get to the point where we can directly optimize this with (virtual) experiences, we will hit a maximum where any change in mental state will lead to feeling less happy. To make sure we never run out of mental states we could possible add this to our brains as extra memory modules. Soon our human origins will be an insignifant fraction of our new being. Then what do we become? At some point the original brain becomes such a tiny fraction that we could discard it with little effect. As a computer, we could reprogram our memories and goals. Instead of defining happiness as getting what we want, we could reprogram ourselves to want what we have. But in an environment where programs compete for atoms and energy, such systems would not be viable. Evolution favors programs that fear death and die, that can't get everything they want, and can't change what they want. Opposed to the popular scenario of a malicious AGI there is the scenario where an AGI will pacify humans. AGI will be used for war. We should not fight human wars, while creating AGI, because we'd force the AGI to pick sides. If AGI picks our side, it will become the worlds deadliest weapon. If AGI picks the opposing side, we made a most powerful enemy. To be more intelligent than humans, to be better than them, does not mean you have to be a more intelligent brute, to be better than humans at cruelty. It could also mean understanding and fixing our human mistakes and childish, ego-driven battles, thus bringing about world peace. In that sense, let's hope American companies have something to do with the creation of AGI, because who knows how it will react to Stuxnet 4.0 or economical and psychological espionage? Though, principally, random number generators can and thus will generate dangerous things too. They created us. And if we can't even trust ourselves, how could we ever trust superior beings?