3 ms·
The Magical Rationalism of Elon Musk and the Prophets of AI
- philipkglass 10y agoWhat's Artificial General Intelligence and how is it different from ordinary machines with programmable logic? It's machine intelligence that will do what I mean when I give it a directive, like an intelligent human being. What's the danger of AGI? It will do what I say instead of what I mean! Like a not-so-intelligent human being, or basically every not-strong-AI computer program ever written. It'll have a theory of mind so advanced that it can sweet-talk its way out of any virtualized prison humans put it in, and simultaneously so deficient that it might interpret "please cure cancer" as "please exterminate multicellular life." The story of AGI-as-existential-threat seems approximately as full of mystery, trembling, fear, awe, paradox, and nonsense as old time religion. (The Paperclip Maximizer is the familiar story of a loop without a terminating condition. The one plausible way to build a superhuman intelligence that's actually malicious is to "upload" a real human brain, with all corresponding abilities and quirks, and run it faster in emulation. That's conceptually plausible; I don't believe in souls. But "uploading" a human is so far from engineering plausibility at this point that you might as well worry about the existential threat posed by time machines.)