This makes several basic errors which more or less invalidate the entire argument. First, the analogy with humans attempting to "increase the intelligence" of

This makes several basic errors which more or less invalidate the entire argument. First, the analogy with humans attempting to "increase the intelligence" of other existing humans is missing the point. We wouldn't increase human intelligence by operating on existing humans, we would increase it by making new humans who are smarter than the last generation. (Let's ignore the politics there, it's a bit of a distraction.) This may not be _practical_ but it is obviously _possible_. Similarly, an AI would not necessarily be attempting to improve "itself", to whatever degree it has any sense of self; it would most likely be designing a better successor (in the ML paradigm, designing & training). As 'jbay808 mentioned in another reply, this only requires that we make an AI that's at least a bit better at AI design than we are. The digression on c. elegans is cute but irrelevant. We've already created narrow AIs that match or crush humans on tasks that were regarded as unlikely to be solved by AI for decades (Go, protein folding, text-to-image generation, many other many-dimensional optimization problems such as server farm cooling schedules, etc). I don't see an argument presented that "design AI" is a skill (or set of skills) which we can't optimize for. I'm not sure what the next argument is supposed to be about except some special pleading about civilization, as embodied by a collection of distinct individual agents, being necessary for "breakthroughs", which are unachievable by the "average" programmer. This is asserted without any particular thought as to how a human-level AI might be scaled further than just 100x the speed of a human programmer, such as, uh, horizontally (which means you can have a civilization if you really want it; I don't think that's how it'll end up playing out, but let's put that aside). There's some time spent talking about how you can't bootstrap compiler optimizations to infinity, which, yes, obviously, and is also a trivially invalid analogy. Then for some reason he ends his argument by assuming that when we create human-level AIs we'll be receiving AIs with distributions of skills at random, such that we'll need to create thousands of them to get one that's above-average on a useful axis, which is frankly a bizarre claim. Even if you take for granted the underlying "civilization > breakthrough" thesis, this is already not how ML works today, and I don't see any reason to think it'll move in that direction in the future. Meh. I'd strongly recommend reading one of the three (excellent) books he cites early on: Nick Bostrom’s “Superintelligence: Paths, Dangers, Strategies,” Max Tegmark’s “Life 3.0: Being Human in the age of Artificial Intelligence,” and Stuart Russell’s “Human Compatible: Artificial Intelligence and the Problem of Control” for a more rigorous treatment of the subject.