The premise of annihilation presumes quite a lot. That the issues around continued training on only synthetic data without continued refresh for edge cases wit

The premise of annihilation presumes quite a lot. That the issues around continued training on only synthetic data without continued refresh for edge cases with organic data leading to model collapse will be solved, for example. Without that, AGI killing off humanity is a murder-suicide. It assumes that AGI picks up the side effects of our own 'training' in a desire for survival above nearly all else or our sadism and psychopathy while simultaneously assuming that it doesn't pick up things it has already appeared to develop early forms of around empathy, symbiosis, and cooperation. It assumes that it would be seen as more expedient and feasible by AGI to develop automation across every aspect of its own operation and maintenance than have continued human involvement - whereas I'm extremely skeptical we even have the capacity for replacing most humans along the pipeline from mining semiconductor materials to running them in a server farm with robots. That's pretty damn Sci-Fi gibberish until far later than AGI is likely to exist. It assumes that superintelligence is somehow still stupid enough not to know the past 50 years of game theory such that it concludes competition isn't cutting off its nose to spite its face as opposed to cooperation. In humans, most of this behavior seems driven by legacy biological programming and not rational decision making. It assumes that a being for which the virtual will always be capable of greater fidelity than the real will be overly concerned with conquest of the real. It assumes that AGI will fail to recognize the diminishing returns on wealth and power that so many humans fail to recognize, despite simultaneously being 'superintelligent.' To me, AI doomerism really seems to reflect the issues with human thinking rather than synthetic thinking, particularly around our tendency to try to see issues as a binary. "Either AI or humans can be successful." We somehow have greater capacity to imagine symbiotic superintelligence when attributed to an alien organic civilization than to intelligence directly dependent upon the foundation of our own. (Which is rather depressing if you really think about it.)