The chessbot analogy is compelling but it does smuggle in assumptions about its generalizability. I'm partial to how so far the history of AI progress has shown

The chessbot analogy is compelling but it does smuggle in assumptions about its generalizability. I'm partial to how so far the history of AI progress has shown that digital neural networks are capable of generalizing more and more, which would allow one to draw a stronger parallel between chessbot domination to AGI domination. Regardless, I have the presence of mind to admit that there's still a gap there in terms of how humans might be doing something 'extra' that doesn't make these systems analogous, and hence the whole argument breaks down. Despite that, there are numerous other ways to envisage harms that, if not leading to outright extinction, still lead to really crappy futures (for both humans and AI) that warrant caution and coordination in developing this technology, neither of which we've seen too much of so far. (The actual history has been "No body can be trusted with this powerful technology except me!" "No, me!" "No, ME!" etc) It makes one really want to hope that decentralized bottoms-up processes of 'intelligence diffusion' can help blunt negative effects, but I'm skeptical of that, taking the long view on what intelligence has done to various groups on our planet so far.