I hope I was thoughtful enough to be worthwhile here. I also have the feeling that maybe I was just frustrated because I wasn't your target audience. You're a

I hope I was thoughtful enough to be worthwhile here. I also have the feeling that maybe I was just frustrated because I wasn't your target audience. You're absolutely right that Musk's vague handwaving shouldn't convince us AI is deadly, and too many people parrot those lines without criticism. Seeing your responses, I have the feeling that you know this ground and simply weren't writing for people who are already familiar with the orthogonality thesis and the specifics of the topic. (If that's true, then my frustration was simply that I thought you were claiming to rebut AI risk, not specific AI risk claims. Sorry.) 1. Searle and Mary's Room and the like are definitely fascinating. I think I just didn't follow your transition from "this is interesting" to "this is a good guide to AI risk". (And yeah, I misjudged how important you considered it based on length and block quoting.) 2. I think I still go further. I do think that AI is an X-risk worry. I think that 'unpredictable' still has the potential (within, say, a century) to be a world-altering crisis. Fundamentally, nothing here persuaded me that the Bostrom/Yudkowsky fear of a bootstrapping intelligence is less likely than I already thought. And I think that limiting our analysis to the individual-level problems of AI error and bias and side-effect are threatens to blind us to that. But those are still problems worth discussing, and they may have reciprocal insights with the x-risk problem. You did close it off nicely. Consider it a pet peeve, because I hate "I don't know" as a way to justify a specific course of action. I'm not sure that's what you're doing, but I've been annoyed by that quote serving that purpose in the past.