So the paper makes a pretty boring argument about computability which looks like every computability argument (reduction to the halting problem). But obviously

So the paper makes a pretty boring argument about computability which looks like every computability argument (reduction to the halting problem). But obviously we don’t live in a world of maximally general problems and such arguments don’t necessarily apply to the specific problems we will care about: “yes it is impossible to prove that a program halts in general but here is my program and here is the proof that it halts.” I find the premise pretty boring too but maybe that is necessary for grant funding. Or maybe this paper is a bit of fun, or an attempt at a good press release from the university, or an effort to publish something after some research that didn’t produce useful results. I find the premise boring because I a find the superintelligence that suddenly grows exponentially, gets the ability to magically solve any problem and dominates everything to be a sci-fi fantasy and not a very realistic risk. The argument against it is one of complexity theory: real life is full of NP hard problems and it seems likely that there would be many on the way for this exponentially growing intelligence. An exponential growth in intelligence gives only a linear growth in the ability to solve exponentially hard problems, however an exponential growth is needed for the intelligence to continue growing exponentially.