2 ms·
> The real problem is that we simply cannot hope to predict what an entity that is far superior to any human would do, essentially by definition. In some ways.
by jbay808 3y ago
> The real problem is that we simply cannot hope to predict what an entity that is far superior to any human would do, essentially by definition.
In some ways. I can't predict Stockfish's next chess move, or else I'd be at least as good at chess as Stockfish. But even though I can't predict the detailed trajectory, I can predict certain aggregates. Like that those moves are on the path towards checkmate.
- p-e-w 3y agoThat's because you know what Stockfish's goal is. That's not true for a hypothetical AGI.
- jbay808 3y agoThat's where the idea of "instrumental convergence" comes in. For almost any end goal, there are intermediate goals that are nearly universal, like: not wanting to be turned off. Because being forcibly shut down is not compatible with achieving most goals. It's hard to predict exactly how an AGI would avoid being shut down, but like the Stockfish example, if the AGI is more intelligent than us, it's likely it would find a method that succeeds. It turns out to be quite a delicate problem to create an AI that is compatible with being shut down by an overseer, and neither strives to avoid being shut down nor strives to shut itself down. Here is a paper by MIRI about this[1], as well as a paper by Deepmind presenting one possible solution[2]. [1] https://intelligence.org/files/Corrigibility.pdf https://intelligence.org/files/Corrigibility.pdf [2] https://www.deepmind.com/publications/safely-interruptible-agents https://www.deepmind.com/publications/safely-interruptible-a... Good explainer video by Robert Miles: https://www.youtube.com/watch?v=ZeecOKBus3Q https://www.youtube.com/watch?v=ZeecOKBus3Q