43 ms·
It depends somewhat on what you mean by AGI, but if you mean something close to human-level, then the answer is that your focus should be entirely on figuring o
by jimrandomh 4y ago
It depends somewhat on what you mean by AGI, but if you mean something close to human-level, then the answer is that your focus should be entirely on figuring out how to ensure humanity survives the invention of that AGI. Major subproblems being figuring out what human values are and how to align the AGI with them ("outer alignment"), figuring out how to make the AGI successfully pass its values on to more-powerful successor AGIs ("inner alignment"), how to detect and limit the formation of misaligned mesa-optimizers inside your AGI, and how to detect if it's taken a treacherous turn.
And if you somehow find yourself in possession of a functioning human-level AGI and you didn't know all those jargon terms already, then the answer is to halt, melt and catch fire, shut down your research project and go study until you're ready to take things seriously.