4 ms·
Mmm. The definition I used is the standard definition of superintelligence, used by EY himself: > A superintelligence is something that can beat any human, and
by sverona 4y ago
Mmm. The definition I used is the standard definition of superintelligence, used by EY himself:
> A superintelligence is something that can beat any human, and the entire human
civilization, at all the cognitive tasks. [0]
I added the "thinks a quadrillion times faster" part but I think that's fair, if perhaps off by a few orders of magnitude.
All of EY's work has an often unstated assumption that AI will adversarially try to kill us. This is explicitly noted in the replies to the AI ruin post. There are a lot of fiddly details that I'm skipping over, but I stand firm that his argument reduces to "it's functionally impossible to stop something faster and smarter than us that really wants to kill us from killing us once it gains sentience."
[0] https://youtu.be/gA1sNLL6yg4?t=1290 https://youtu.be/gA1sNLL6yg4?t=1290
- hamburga 4y agoYour reduction doesn’t cover the real risks of orthogonality, instrumental convergence, the zero margins for failure on the first try, intelligence explosions, and the impossibility of training for alignment.
- sverona 4y agoYeah, those only make it worse, but they only really apply to a bona fide superintelligence of the sort EY describes, and those are not what we have. I don't believe we're particularly close to having one. If we're doomed, we're doomed, but please don't tell me about it.