I think I see your point, but in my thought experiment, this future AI will be bootstrapped from the current accumulated knowledge of programming and software e

I think I see your point, but in my thought experiment, this future AI will be bootstrapped from the current accumulated knowledge of programming and software engineering. I don't think it would be back to square one. At the very least it will be able to write code in whatever human programming languages are around. I'm taking as given in this thought experiment that, given the requirements, humans don't need to write any code whatsoever. My point then reduces to, what happens next? The machine probably doesn't need the abstraction layer of programming languages. It might even have its own, layer(s) which make no sense to us, or it writes machine code. The question then hinges on how we, as humans, would validate such a system.