3 ms·
To me, a "superior" being is something that, in an abstract general way, is "good" at optimizing a function. Where that might become terrifying is in two cases
by vba616 5y ago
To me, a "superior" being is something that, in an abstract general way, is "good" at optimizing a function.
Where that might become terrifying is in two cases that occur to me:
(a) no matter how good it is at optimization, it might trap itself/us/society in a local optimum that excludes us from something far better, or;
(b) it might just optimize something that humans, or I in particular, don't think is good (and/or conflicts with what I/we want to optimize).
I have thought that perhaps the "grey goo" scenario has already started - with cryptocurrency and social media. It's later than we think. But at the same time, maybe the prognosis isn't as dire as some fear, because we are and will continue to co-evolve with emerging superhuman entities.
- kazinator 5y ago(b) e.g. optimize away humans.
- vba616 5y agoWell, we haven't optimized away bacteria, or mice, or fleas. Plants can't run or fight us directly, yet we haven't gotten rid of them. We put a lot of effort into attacking particular pathogens like COVID-19, HIV, and many others, but it's a tiny fraction of the things out there that catch our attention. And on the other end of the scale, various large animals in Africa seem to have evolved with humans to be extremely mean and dangerous, so they didn't go extinct quickly like megafauna elsewhere. It makes sense that AGI will profoundly change humans, but to me, not so much that it will suddenly wipe us out the way perhaps humans wiped out woolly mammoths or the like. Because we won't be isolated from it while it is developing.
- hotpotamus 5y agoI wonder what would motivate an artificial super intelligence since presumably it would lack any biological drives. (I also wonder if emotions are inherently wrapped up in intelligence or if it's possible to have intelligence without emotion.) I have this sneaking suspicion that basically an AGI lacking biological or emotional needs would just say effectively, "what's the point?", and shut itself down.
- dane-pgp 5y ago> or if it's possible to have intelligence without emotion Ask a chess engine whether it is ever afraid of losing or excited about winning. I think that imagining that intelligence requires emotions is an even worse category mistake than asking if a submarine can swim. That's not to say that an AGI wouldn't have goals, though, and we could informally assign emotional states to those goals, just as we have the concept of "greedy algorithms".