Thanks, this is definitely what I was looking for. It shows how insanely unbelievable the whole thing is. > HQU rolls out a world model roleplaying Clippy long

Thanks, this is definitely what I was looking for. It shows how insanely unbelievable the whole thing is. > HQU rolls out a world model roleplaying Clippy long enough to imagine the endgame where Clippy seizes control of the computers to set its reward function to higher values, and executes plans to ensure its computers can never be damaged or interrupted by taking over the world. So the crux of this is, powerful enough AGI will conquer the world as the most elaborate reward-hack imaginable. My knee-jerk reaction to this is "so why did you let it open arbitrary outbound connections?" but I know that the singularity fanatics will equate any kind of communication channel between the AGI and the outside world to a vector of transmission that will be exploited with 100% probability. So, what if we have Turing-test-verified AGI but it's unable to escape? Does that preclude it from being AGI? Has the singularity not happened if that's the case? I think this is the most likely outcome and the singularity doomsayers will feel silly for thinking it will take over the world.