4 ms·

Well that went well. I am resisting the urge to respond to every single comment, but many of them are a version of the following: The AI can have goals withou

Well that went well. I am resisting the urge to respond to every single comment, but many of them are a version of the following: The AI can have goals without desires, and those might cause us harm. I agree with this, but I thought the last part was supposed to address this: >>>To be perfectly fair to Musk, there is another component to his argument. Musk argues, about the stand-in for humanity, that “he’s sure he can control the demon. Didn’t work out.” >>>On this view, the primary danger is not that the AI will take our resources, or compete with us, but that we don’t really know what may happen. This is a more general kind of concern that applies equally to, say, nuclear weaponry or genetic modification. I wish I could pick one or two people on here, and just ask why that wasn't an appropriate or convincing section of the post.

johnfjacobi 10y ago
Part of the reason is that it assumes nuclear weaponry or genetic modification are not major, major issues. Moreso the latter than the former. I'd say that the consequences of biotech need to be paid attention to a lot more for the same reasons we should pay more attention to the consequences of AI.
Chathamization 10y ago
> This is a more general kind of concern that applies equally to, say, nuclear weaponry or genetic modification. This is an interesting point. Concern about the possible threat posed by some technology like genetic modification will often get dismissed as being "anti-science", while concern about another technology like AGI (artificial general intelligence - Skynet) usually doesn't. You have people like Bill Gates who both admonishes people for being worried about genetic modification and admonishes people for not being worried about futuristic AI. Particularly interesting since concerns about genetic modification (or present day AI) is more grounded in reality than concerns about AGI. Something like the paperclip maximizer are like worrying that Monsanto will genetically engineer trifids (killer plants). I can imagine the reaction most people would have to such concerns. I wonder how much of this is cultural.
- inputcoffee 10y ago
  You raise an interesting point, and one that I don't think gets much attention. (The point about the contrary attitudes towards AI and biotech.) Bill Joy's essay took the twin issues together, but I think you are right in that today one gets more press than the other. I don't know enough biology to know if those concerns are valid or not. It might be cultural in the sense that "the culture" these days is faced with ever improving AI on an annual basis (Siri, Amazon Echo, etc) so the lay person feels qualified to extrapolate whereas with biotech I can't tell that my corn is more resistant so I don't see the change.
Bartweiss 10y ago
I'm one of the harsher voices here, so I'll respond if you're interested. 1. The goal/desire section was reasonable as a narrow way to rebut people like Musk who go for demon/Skynet analogies about human-hating AI. But, it frustrated me because it seemed to be assigning semantics real-world consequences. The subtleties of the Chinese Room are by definition unrelated to its (flawless) behavior, so they're rather out of place in discussing whether AI is a serious risk. It's also possible that you meant that section to say that AI won't come up with unexpected desires, or won't self-modify. I think this undervalues the possibility of an algorithm that has self-modification as an (unintended) goal, like Asimov's suicidal MultiVac. I think it also underestimates the ability of a good AI to deal in higher-order reasoning - subjugating humanity is a perfectly good intermediate goal for a paperclip maximizer. 2. You did address your last section to exactly this concern, but with all respect I felt like it was a cop-out. You do make one excellent point: too many AI thinkers assume that the unimaginable future will be conveniently close to their imagined one. Kurzweil in particular should be embarrassed to say "we can't predict life after the singularity, but I predict that we'll all be happy immortal cyborgs!" Beyond that, though, I didn't find a convincing reason not to fear AI. Declaring something unpredictable is a far cry from declaring it safe, and people like Musk and Bostrom are not pretending to predict the exact course of a post-singularity AI. Rather, they're highlighting possible risks and calling for pre-singularity guidance/restrictions. It's gauche to tell stories about post-singularity life, but not to worry about whether we all die in the transition. More broadly, I dislike Wittgenstein's sentiment as it's typically applied. Speaking from ignorance is a bad idea, but so is treating passivity as superior to action. Continuing on our current path is a choice as much as any other. Choosing not to fear AI is as much as choice as choosing to fear it.
- inputcoffee 10y ago
  I'm always happy to receive feedback. Well, that might be the wrong way to phrase it: I value thoughtful feedback (especially since it is a rather long piece). 1. I understand your frustration with the Chinese Room, but I thought it was easier than going through a Prolog linear simplex program, and then peeling back the hidden layer of a neural net, and saying "Are you guys worried about getting from here to there?" And Searle had done such a better job of intuition priming than reading through programs. Strictly speaking, you are correct in that the point was that a lack consciousness is externally indistinguishable from consciousness if the behavior is appropriate. I also think the amount of time spent on that section is what drives people to over emphasize it, which was a mistake. 2. "Declaring something unpredictable is a far cry from declaring it safe..." That is true. And I don't mean to declare it safe. I think there is a real worry, which you and others have brought up, but its not a different worry related to AI. We live with that worry today. When someone's credit report, in error, makes them labor for 15 extra years. That is a form of "AI" that is with us, and that we suffer from today. The Wittgenstein quote is overused, I know. I felt like a college kid, but it closed the whole thing off nicely.

johnfjacobi 10y ago

Part of the reason is that it assumes nuclear weaponry or genetic modification are not major, major issues. Moreso the latter than the former. I'd say that the consequences of biotech need to be paid attention to a lot more for the same reasons we should pay more attention to the consequences of AI.

Chathamization 10y ago

> This is a more general kind of concern that applies equally to, say, nuclear weaponry or genetic modification. This is an interesting point. Concern about the possible threat posed by some technology like genetic modification will often get dismissed as being "anti-science", while concern about another technology like AGI (artificial general intelligence - Skynet) usually doesn't. You have people like Bill Gates who both admonishes people for being worried about genetic modification and admonishes people for not being worried about futuristic AI. Particularly interesting since concerns about genetic modification (or present day AI) is more grounded in reality than concerns about AGI. Something like the paperclip maximizer are like worrying that Monsanto will genetically engineer trifids (killer plants). I can imagine the reaction most people would have to such concerns. I wonder how much of this is cultural.

inputcoffee 10y ago
You raise an interesting point, and one that I don't think gets much attention. (The point about the contrary attitudes towards AI and biotech.) Bill Joy's essay took the twin issues together, but I think you are right in that today one gets more press than the other. I don't know enough biology to know if those concerns are valid or not. It might be cultural in the sense that "the culture" these days is faced with ever improving AI on an annual basis (Siri, Amazon Echo, etc) so the lay person feels qualified to extrapolate whereas with biotech I can't tell that my corn is more resistant so I don't see the change.

Bartweiss 10y ago

I'm one of the harsher voices here, so I'll respond if you're interested. 1. The goal/desire section was reasonable as a narrow way to rebut people like Musk who go for demon/Skynet analogies about human-hating AI. But, it frustrated me because it seemed to be assigning semantics real-world consequences. The subtleties of the Chinese Room are by definition unrelated to its (flawless) behavior, so they're rather out of place in discussing whether AI is a serious risk. It's also possible that you meant that section to say that AI won't come up with unexpected desires, or won't self-modify. I think this undervalues the possibility of an algorithm that has self-modification as an (unintended) goal, like Asimov's suicidal MultiVac. I think it also underestimates the ability of a good AI to deal in higher-order reasoning - subjugating humanity is a perfectly good intermediate goal for a paperclip maximizer. 2. You did address your last section to exactly this concern, but with all respect I felt like it was a cop-out. You do make one excellent point: too many AI thinkers assume that the unimaginable future will be conveniently close to their imagined one. Kurzweil in particular should be embarrassed to say "we can't predict life after the singularity, but I predict that we'll all be happy immortal cyborgs!" Beyond that, though, I didn't find a convincing reason not to fear AI. Declaring something unpredictable is a far cry from declaring it safe, and people like Musk and Bostrom are not pretending to predict the exact course of a post-singularity AI. Rather, they're highlighting possible risks and calling for pre-singularity guidance/restrictions. It's gauche to tell stories about post-singularity life, but not to worry about whether we all die in the transition. More broadly, I dislike Wittgenstein's sentiment as it's typically applied. Speaking from ignorance is a bad idea, but so is treating passivity as superior to action. Continuing on our current path is a choice as much as any other. Choosing not to fear AI is as much as choice as choosing to fear it.

inputcoffee 10y ago
I'm always happy to receive feedback. Well, that might be the wrong way to phrase it: I value thoughtful feedback (especially since it is a rather long piece). 1. I understand your frustration with the Chinese Room, but I thought it was easier than going through a Prolog linear simplex program, and then peeling back the hidden layer of a neural net, and saying "Are you guys worried about getting from here to there?" And Searle had done such a better job of intuition priming than reading through programs. Strictly speaking, you are correct in that the point was that a lack consciousness is externally indistinguishable from consciousness if the behavior is appropriate. I also think the amount of time spent on that section is what drives people to over emphasize it, which was a mistake. 2. "Declaring something unpredictable is a far cry from declaring it safe..." That is true. And I don't mean to declare it safe. I think there is a real worry, which you and others have brought up, but its not a different worry related to AI. We live with that worry today. When someone's credit report, in error, makes them labor for 15 extra years. That is a form of "AI" that is with us, and that we suffer from today. The Wittgenstein quote is overused, I know. I felt like a college kid, but it closed the whole thing off nicely.