4 ms·

I'm one of the harsher voices here, so I'll respond if you're interested. 1. The goal/desire section was reasonable as a narrow way to rebut people like Musk w

I'm one of the harsher voices here, so I'll respond if you're interested. 1. The goal/desire section was reasonable as a narrow way to rebut people like Musk who go for demon/Skynet analogies about human-hating AI. But, it frustrated me because it seemed to be assigning semantics real-world consequences. The subtleties of the Chinese Room are by definition unrelated to its (flawless) behavior, so they're rather out of place in discussing whether AI is a serious risk. It's also possible that you meant that section to say that AI won't come up with unexpected desires, or won't self-modify. I think this undervalues the possibility of an algorithm that has self-modification as an (unintended) goal, like Asimov's suicidal MultiVac. I think it also underestimates the ability of a good AI to deal in higher-order reasoning - subjugating humanity is a perfectly good intermediate goal for a paperclip maximizer. 2. You did address your last section to exactly this concern, but with all respect I felt like it was a cop-out. You do make one excellent point: too many AI thinkers assume that the unimaginable future will be conveniently close to their imagined one. Kurzweil in particular should be embarrassed to say "we can't predict life after the singularity, but I predict that we'll all be happy immortal cyborgs!" Beyond that, though, I didn't find a convincing reason not to fear AI. Declaring something unpredictable is a far cry from declaring it safe, and people like Musk and Bostrom are not pretending to predict the exact course of a post-singularity AI. Rather, they're highlighting possible risks and calling for pre-singularity guidance/restrictions. It's gauche to tell stories about post-singularity life, but not to worry about whether we all die in the transition. More broadly, I dislike Wittgenstein's sentiment as it's typically applied. Speaking from ignorance is a bad idea, but so is treating passivity as superior to action. Continuing on our current path is a choice as much as any other. Choosing not to fear AI is as much as choice as choosing to fear it.

inputcoffee 10y ago
I'm always happy to receive feedback. Well, that might be the wrong way to phrase it: I value thoughtful feedback (especially since it is a rather long piece). 1. I understand your frustration with the Chinese Room, but I thought it was easier than going through a Prolog linear simplex program, and then peeling back the hidden layer of a neural net, and saying "Are you guys worried about getting from here to there?" And Searle had done such a better job of intuition priming than reading through programs. Strictly speaking, you are correct in that the point was that a lack consciousness is externally indistinguishable from consciousness if the behavior is appropriate. I also think the amount of time spent on that section is what drives people to over emphasize it, which was a mistake. 2. "Declaring something unpredictable is a far cry from declaring it safe..." That is true. And I don't mean to declare it safe. I think there is a real worry, which you and others have brought up, but its not a different worry related to AI. We live with that worry today. When someone's credit report, in error, makes them labor for 15 extra years. That is a form of "AI" that is with us, and that we suffer from today. The Wittgenstein quote is overused, I know. I felt like a college kid, but it closed the whole thing off nicely.
- Bartweiss 10y ago
  I hope I was thoughtful enough to be worthwhile here. I also have the feeling that maybe I was just frustrated because I wasn't your target audience. You're absolutely right that Musk's vague handwaving shouldn't convince us AI is deadly, and too many people parrot those lines without criticism. Seeing your responses, I have the feeling that you know this ground and simply weren't writing for people who are already familiar with the orthogonality thesis and the specifics of the topic. (If that's true, then my frustration was simply that I thought you were claiming to rebut AI risk, not specific AI risk claims. Sorry.) 1. Searle and Mary's Room and the like are definitely fascinating. I think I just didn't follow your transition from "this is interesting" to "this is a good guide to AI risk". (And yeah, I misjudged how important you considered it based on length and block quoting.) 2. I think I still go further. I do think that AI is an X-risk worry. I think that 'unpredictable' still has the potential (within, say, a century) to be a world-altering crisis. Fundamentally, nothing here persuaded me that the Bostrom/Yudkowsky fear of a bootstrapping intelligence is less likely than I already thought. And I think that limiting our analysis to the individual-level problems of AI error and bias and side-effect are threatens to blind us to that. But those are still problems worth discussing, and they may have reciprocal insights with the x-risk problem. You did close it off nicely. Consider it a pet peeve, because I hate "I don't know" as a way to justify a specific course of action. I'm not sure that's what you're doing, but I've been annoyed by that quote serving that purpose in the past.

inputcoffee 10y ago

I'm always happy to receive feedback. Well, that might be the wrong way to phrase it: I value thoughtful feedback (especially since it is a rather long piece). 1. I understand your frustration with the Chinese Room, but I thought it was easier than going through a Prolog linear simplex program, and then peeling back the hidden layer of a neural net, and saying "Are you guys worried about getting from here to there?" And Searle had done such a better job of intuition priming than reading through programs. Strictly speaking, you are correct in that the point was that a lack consciousness is externally indistinguishable from consciousness if the behavior is appropriate. I also think the amount of time spent on that section is what drives people to over emphasize it, which was a mistake. 2. "Declaring something unpredictable is a far cry from declaring it safe..." That is true. And I don't mean to declare it safe. I think there is a real worry, which you and others have brought up, but its not a different worry related to AI. We live with that worry today. When someone's credit report, in error, makes them labor for 15 extra years. That is a form of "AI" that is with us, and that we suffer from today. The Wittgenstein quote is overused, I know. I felt like a college kid, but it closed the whole thing off nicely.

Bartweiss 10y ago
I hope I was thoughtful enough to be worthwhile here. I also have the feeling that maybe I was just frustrated because I wasn't your target audience. You're absolutely right that Musk's vague handwaving shouldn't convince us AI is deadly, and too many people parrot those lines without criticism. Seeing your responses, I have the feeling that you know this ground and simply weren't writing for people who are already familiar with the orthogonality thesis and the specifics of the topic. (If that's true, then my frustration was simply that I thought you were claiming to rebut AI risk, not specific AI risk claims. Sorry.) 1. Searle and Mary's Room and the like are definitely fascinating. I think I just didn't follow your transition from "this is interesting" to "this is a good guide to AI risk". (And yeah, I misjudged how important you considered it based on length and block quoting.) 2. I think I still go further. I do think that AI is an X-risk worry. I think that 'unpredictable' still has the potential (within, say, a century) to be a world-altering crisis. Fundamentally, nothing here persuaded me that the Bostrom/Yudkowsky fear of a bootstrapping intelligence is less likely than I already thought. And I think that limiting our analysis to the individual-level problems of AI error and bias and side-effect are threatens to blind us to that. But those are still problems worth discussing, and they may have reciprocal insights with the x-risk problem. You did close it off nicely. Consider it a pet peeve, because I hate "I don't know" as a way to justify a specific course of action. I'm not sure that's what you're doing, but I've been annoyed by that quote serving that purpose in the past.