> I think the premise of an intelligence explosion usually starts with human programmers creating an AI that is at least slightly better than they are themselve

> I think the premise of an intelligence explosion usually starts with human programmers creating an AI that is at least slightly better than they are themselves, at at least the task of AI programming. If the author subscribes to the refutation that I quoted, then I think it follows that they also wouldn't believe this premise is possible to begin with. The author is quite explicit about this: he is contradicting that the singularity must be real, not trying to claim that the singularity can't be real. If you agree with his refutation of the arguments for the singularity, you'll be left with the claim "the singularity may or may not happen, we don't know", not with the claim "it's impossible for the singularity to happen". > > This ability of humans to build on one another’s work is precisely why I don’t believe that running a human-equivalent A.I. program for a hundred years in isolation is a good way to produce major breakthroughs. > As the saying goes, if I'm going to spend ten hours cutting down trees, I'd spend nine of them sharpening my axe. Just as a society of humans can develop tools to improve their capabilities, so can a single human, as the author's own example of Isaac Newton shows. If the AI is going to be locked in isolation for a hundred years to write software, why doesn't the author think it will spend some of that time on tools to improve its own productivity? I think a better argument against the author's claim here is that thinking of 1 AI as 1 human is probably wrong. Since an AI can be trivially copied and run independently, it's more useful to think of 1 AI as 1 society.