3 ms·

You haven't really filled in the details of what you are actually frightened of. You've only hinted at it and assumed it was obvious. I think AGI is inevitable

You haven't really filled in the details of what you are actually frightened of. You've only hinted at it and assumed it was obvious. I think AGI is inevitable, not in your time horizon at all however. But I think it would be the best thing to ever happen to humanity. So I don't see the doom and gloom. I have no idea what "functional obsolescence" even means. So what if AGI can do everything better than a human. Why is that relevant? I don't understand how the existence of a superior being is supposed to be terrifying.

vba616 5y ago
To me, a "superior" being is something that, in an abstract general way, is "good" at optimizing a function. Where that might become terrifying is in two cases that occur to me: (a) no matter how good it is at optimization, it might trap itself/us/society in a local optimum that excludes us from something far better, or; (b) it might just optimize something that humans, or I in particular, don't think is good (and/or conflicts with what I/we want to optimize). I have thought that perhaps the "grey goo" scenario has already started - with cryptocurrency and social media. It's later than we think. But at the same time, maybe the prognosis isn't as dire as some fear, because we are and will continue to co-evolve with emerging superhuman entities.
- kazinator 5y ago
  (b) e.g. optimize away humans.
  - vba616 5y ago
    Well, we haven't optimized away bacteria, or mice, or fleas. Plants can't run or fight us directly, yet we haven't gotten rid of them. We put a lot of effort into attacking particular pathogens like COVID-19, HIV, and many others, but it's a tiny fraction of the things out there that catch our attention. And on the other end of the scale, various large animals in Africa seem to have evolved with humans to be extremely mean and dangerous, so they didn't go extinct quickly like megafauna elsewhere. It makes sense that AGI will profoundly change humans, but to me, not so much that it will suddenly wipe us out the way perhaps humans wiped out woolly mammoths or the like. Because we won't be isolated from it while it is developing.
- hotpotamus 5y ago
  I wonder what would motivate an artificial super intelligence since presumably it would lack any biological drives. (I also wonder if emotions are inherently wrapped up in intelligence or if it's possible to have intelligence without emotion.) I have this sneaking suspicion that basically an AGI lacking biological or emotional needs would just say effectively, "what's the point?", and shut itself down.
  - dane-pgp 5y ago
    > or if it's possible to have intelligence without emotion Ask a chess engine whether it is ever afraid of losing or excited about winning. I think that imagining that intelligence requires emotions is an even worse category mistake than asking if a submarine can swim. That's not to say that an AGI wouldn't have goals, though, and we could informally assign emotional states to those goals, just as we have the concept of "greedy algorithms".

vba616 5y ago

To me, a "superior" being is something that, in an abstract general way, is "good" at optimizing a function. Where that might become terrifying is in two cases that occur to me: (a) no matter how good it is at optimization, it might trap itself/us/society in a local optimum that excludes us from something far better, or; (b) it might just optimize something that humans, or I in particular, don't think is good (and/or conflicts with what I/we want to optimize). I have thought that perhaps the "grey goo" scenario has already started - with cryptocurrency and social media. It's later than we think. But at the same time, maybe the prognosis isn't as dire as some fear, because we are and will continue to co-evolve with emerging superhuman entities.

kazinator 5y ago
(b) e.g. optimize away humans.
- vba616 5y ago
  Well, we haven't optimized away bacteria, or mice, or fleas. Plants can't run or fight us directly, yet we haven't gotten rid of them. We put a lot of effort into attacking particular pathogens like COVID-19, HIV, and many others, but it's a tiny fraction of the things out there that catch our attention. And on the other end of the scale, various large animals in Africa seem to have evolved with humans to be extremely mean and dangerous, so they didn't go extinct quickly like megafauna elsewhere. It makes sense that AGI will profoundly change humans, but to me, not so much that it will suddenly wipe us out the way perhaps humans wiped out woolly mammoths or the like. Because we won't be isolated from it while it is developing.
hotpotamus 5y ago
I wonder what would motivate an artificial super intelligence since presumably it would lack any biological drives. (I also wonder if emotions are inherently wrapped up in intelligence or if it's possible to have intelligence without emotion.) I have this sneaking suspicion that basically an AGI lacking biological or emotional needs would just say effectively, "what's the point?", and shut itself down.
- dane-pgp 5y ago
  > or if it's possible to have intelligence without emotion Ask a chess engine whether it is ever afraid of losing or excited about winning. I think that imagining that intelligence requires emotions is an even worse category mistake than asking if a submarine can swim. That's not to say that an AGI wouldn't have goals, though, and we could informally assign emotional states to those goals, just as we have the concept of "greedy algorithms".