3 ms·

I'm always happy to receive feedback. Well, that might be the wrong way to phrase it: I value thoughtful feedback (especially since it is a rather long piece).

I'm always happy to receive feedback. Well, that might be the wrong way to phrase it: I value thoughtful feedback (especially since it is a rather long piece). 1. I understand your frustration with the Chinese Room, but I thought it was easier than going through a Prolog linear simplex program, and then peeling back the hidden layer of a neural net, and saying "Are you guys worried about getting from here to there?" And Searle had done such a better job of intuition priming than reading through programs. Strictly speaking, you are correct in that the point was that a lack consciousness is externally indistinguishable from consciousness if the behavior is appropriate. I also think the amount of time spent on that section is what drives people to over emphasize it, which was a mistake. 2. "Declaring something unpredictable is a far cry from declaring it safe..." That is true. And I don't mean to declare it safe. I think there is a real worry, which you and others have brought up, but its not a different worry related to AI. We live with that worry today. When someone's credit report, in error, makes them labor for 15 extra years. That is a form of "AI" that is with us, and that we suffer from today. The Wittgenstein quote is overused, I know. I felt like a college kid, but it closed the whole thing off nicely.

Bartweiss 10y ago
I hope I was thoughtful enough to be worthwhile here. I also have the feeling that maybe I was just frustrated because I wasn't your target audience. You're absolutely right that Musk's vague handwaving shouldn't convince us AI is deadly, and too many people parrot those lines without criticism. Seeing your responses, I have the feeling that you know this ground and simply weren't writing for people who are already familiar with the orthogonality thesis and the specifics of the topic. (If that's true, then my frustration was simply that I thought you were claiming to rebut AI risk, not specific AI risk claims. Sorry.) 1. Searle and Mary's Room and the like are definitely fascinating. I think I just didn't follow your transition from "this is interesting" to "this is a good guide to AI risk". (And yeah, I misjudged how important you considered it based on length and block quoting.) 2. I think I still go further. I do think that AI is an X-risk worry. I think that 'unpredictable' still has the potential (within, say, a century) to be a world-altering crisis. Fundamentally, nothing here persuaded me that the Bostrom/Yudkowsky fear of a bootstrapping intelligence is less likely than I already thought. And I think that limiting our analysis to the individual-level problems of AI error and bias and side-effect are threatens to blind us to that. But those are still problems worth discussing, and they may have reciprocal insights with the x-risk problem. You did close it off nicely. Consider it a pet peeve, because I hate "I don't know" as a way to justify a specific course of action. I'm not sure that's what you're doing, but I've been annoyed by that quote serving that purpose in the past.

Bartweiss 10y ago

I hope I was thoughtful enough to be worthwhile here. I also have the feeling that maybe I was just frustrated because I wasn't your target audience. You're absolutely right that Musk's vague handwaving shouldn't convince us AI is deadly, and too many people parrot those lines without criticism. Seeing your responses, I have the feeling that you know this ground and simply weren't writing for people who are already familiar with the orthogonality thesis and the specifics of the topic. (If that's true, then my frustration was simply that I thought you were claiming to rebut AI risk, not specific AI risk claims. Sorry.) 1. Searle and Mary's Room and the like are definitely fascinating. I think I just didn't follow your transition from "this is interesting" to "this is a good guide to AI risk". (And yeah, I misjudged how important you considered it based on length and block quoting.) 2. I think I still go further. I do think that AI is an X-risk worry. I think that 'unpredictable' still has the potential (within, say, a century) to be a world-altering crisis. Fundamentally, nothing here persuaded me that the Bostrom/Yudkowsky fear of a bootstrapping intelligence is less likely than I already thought. And I think that limiting our analysis to the individual-level problems of AI error and bias and side-effect are threatens to blind us to that. But those are still problems worth discussing, and they may have reciprocal insights with the x-risk problem. You did close it off nicely. Consider it a pet peeve, because I hate "I don't know" as a way to justify a specific course of action. I'm not sure that's what you're doing, but I've been annoyed by that quote serving that purpose in the past.