5 ms·

> How can it not be obvious You have succinctly and completely summed up the AI risk argument more eloquently than anyone I've seen before. "How can it not be

> How can it not be obvious You have succinctly and completely summed up the AI risk argument more eloquently than anyone I've seen before. "How can it not be obvious?" Everything else is just intellectual fig leaves for the core argument that intuitively, without evidence, this proposition is obvious. The problem is, lots of "obvious" things have turned out to be very wrong. Sometimes relatively harmlessly, like the obviousness of the sun revolving around the earth, and sometimes catastrophically, like the obviousness of one race being inherently inferior. We should be very suspicious of policy that is based on propositions so obvious that it's borderline offensive to question them.

computerphage 3y ago
> borderline offensive to question them I would be happy to politely discuss any proposition regarding AI Risk. I don't think any claim should go unquestioned. I can also point you to much longer-form discussions. For example, this post, which has 670 comments, discussing various aspects of the argument: https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a-list-of-lethalities https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a...
- frankzinger 3y ago
  That link is about the risks of AGI, which doesn't exist, and there's no reason to believe that it ever will exist. (If I'm wrong about AGI then I'm open to being convinced, but that's a different conversation as the topic here is non-general AI, is it not?)
  - computerphage 3y ago
    I disagree that there's no reason to believe it will ever exist. For one thing, many smart people are trying to build the technology right now and they believe it to be possible. I see no compelling case that the intelligence scale simply tops-out with humans; that a more intelligent system is ruled out by the laws of physics. The topic here is human extinction caused by AI. I don't know of any serious argument for why a non-general intelligence (really a system less intelligent than a human) would pose an extinction risk to humanity. Plus, my background understanding of the people who signed this is that they're worried about AGI, not present-day systems, but that's an inference.
    - frankzinger 3y ago
      Maybe these AI Apocalypse articles published for general consumption would be justified if there were any signs whatsoever that we were on a path towards AGI but there are none, are there? Even the best we have today are still just machines. They are clearly not really intelligent. At best they simulate intelligence, but poorly (because they still make ridiculous mistakes). Just because there are no physical limits to intelligence doesn't mean it's possible for beings with finite intelligence to create infinite intelligence. It all just seems extremely premature to me.
- brookst 3y ago
  I appreciate your reasonableness. I follow LW to some degree, but even the best of it (like the post you link) feels very in-group confirmation centric. That post is long and I have not read it all, but it seems to be missing any consideration of AGI upside. It’s like talking about the risk of dying in a car crash with no consideration of the benefits of travel. If I ask you “do you want to get in a metal can that has a small but non-zero chance of killing you”, of course that sounds like a terrible idea. There is risk in AGI. There is risk in everything. How many people are killed by furniture each year? I’m not dismissing AGI risk, I’m saying that I have yet to see a considered discussion that includes important context like how many people will live longer/happier because AGI helps reduce famine/disease. Somehow it is always the wealthy, employed, at-risk-of-disruption people who are worried, not the poor or starving or oppressed. I’m just super not impressed by the AI risk crowd, at least the core one on LW / SSC / etc.
  - FinchNova12 3y ago
    While I agree that the rhetoric around AI Safety would be better if it tried to address some of the benefits (and not embody the full doomer vibe), I don't think many of the 'core thinkers' are unaware of the benefits in AGI. I don't fully agree with this paper's conclusions, but I think https://nickbostrom.com/astronomical/waste https://nickbostrom.com/astronomical/waste is one piece that embodies this style of thinking well!
    - brookst 3y ago
      Thanks for the link -- that is a good paper (in the sense of making its point, though I also don't entirely agree), and it hurts the AI risk position that that kind of thinking doesn't get airtime. It may be that those 'core thinkers' are aware, but if so it's counter-productive and of questionable integrity to sweep that side of the argument under the rug.
patrec 3y ago
> We should be very suspicious of policy that is based on propositions so obvious that it's borderline offensive to question them. Mostly if the "obviousness" just masks a social taboo, which I don't see being the case here. Do you? > The problem is, lots of "obvious" things have turned out to be very wrong. A much bigger problem is that lots more "counter-intuitive" things that people like to believe because they elevate them over the unwashed masses have turned and continue to turn out to be very wrong and that this does not prevent them from forming the basis for important policy decisions. I'm all for questioning even what appears intuitively obvious (especially if much rides on getting it right, as presumably it does here). But frankly, of the many bizarre reasons I have heard why we should not worry about AI the claim that it seems far too obvious that we should must be the single most perverse one yet. > Everything else is just intellectual fig leaves for the core argument that intuitively, without evidence, this proposition is obvious. Maybe your appraisal of what counts as evidence is defective? For example, there's been a pattern of people confidently predicting AIs won't be able to perform various particular feats of the human mind (either fundamentally or in the next few decades) only to be proven wrong over increasingly shorter time-spans. And with AIs often not just reaching but far surpassing human ability. I'm happy to provide examples. Can you explain to me why you think this is does not count, in any way, as evidence that AIs have the potential to reach a level of capability that renders them quite dangerous?
- revelio 3y ago
  > Mostly if the "obviousness" just masks a social taboo, which I don't see being the case here. Do you? The social taboo here is saying that a position taken by lots of highly educated people is nonsense because they're all locked in a dumb purity spiral that leads to motivated reasoning. This is actually one of societies biggest taboos! Look at what happens to people who make that argument publicly under their own name in other contexts; they tend to get fired and cancelled really fast. > there's been a pattern of people confidently predicting AIs won't be able to perform various particular feats of the human mind (either fundamentally or in the next few decades) only to be proven wrong over increasingly shorter time-spans That sword cuts both ways! There have been lots of predictions in the last decade that AI will contribute novel and hithertofore unknown solutions to things like climate change or curing cancer. Try getting GPT-4 to spit out a novel research-quality solution to anything, even a simple product design problem, and you'll find it can't. > the claim that it seems far too obvious that we should They're not arguing that. They're saying that AI risk proponents don't actually have good arguments, which is why they so regularly fall back on "it's so obvious we shouldn't need to explain why it's important". If your argument consists primarily of "everyone knows that" then this is a good indication you might be wrong.

computerphage 3y ago

> borderline offensive to question them I would be happy to politely discuss any proposition regarding AI Risk. I don't think any claim should go unquestioned. I can also point you to much longer-form discussions. For example, this post, which has 670 comments, discussing various aspects of the argument: https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a-list-of-lethalities https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a...

frankzinger 3y ago
That link is about the risks of AGI, which doesn't exist, and there's no reason to believe that it ever will exist. (If I'm wrong about AGI then I'm open to being convinced, but that's a different conversation as the topic here is non-general AI, is it not?)
- computerphage 3y ago
  I disagree that there's no reason to believe it will ever exist. For one thing, many smart people are trying to build the technology right now and they believe it to be possible. I see no compelling case that the intelligence scale simply tops-out with humans; that a more intelligent system is ruled out by the laws of physics. The topic here is human extinction caused by AI. I don't know of any serious argument for why a non-general intelligence (really a system less intelligent than a human) would pose an extinction risk to humanity. Plus, my background understanding of the people who signed this is that they're worried about AGI, not present-day systems, but that's an inference.
  - frankzinger 3y ago
    Maybe these AI Apocalypse articles published for general consumption would be justified if there were any signs whatsoever that we were on a path towards AGI but there are none, are there? Even the best we have today are still just machines. They are clearly not really intelligent. At best they simulate intelligence, but poorly (because they still make ridiculous mistakes). Just because there are no physical limits to intelligence doesn't mean it's possible for beings with finite intelligence to create infinite intelligence. It all just seems extremely premature to me.
brookst 3y ago
I appreciate your reasonableness. I follow LW to some degree, but even the best of it (like the post you link) feels very in-group confirmation centric. That post is long and I have not read it all, but it seems to be missing any consideration of AGI upside. It’s like talking about the risk of dying in a car crash with no consideration of the benefits of travel. If I ask you “do you want to get in a metal can that has a small but non-zero chance of killing you”, of course that sounds like a terrible idea. There is risk in AGI. There is risk in everything. How many people are killed by furniture each year? I’m not dismissing AGI risk, I’m saying that I have yet to see a considered discussion that includes important context like how many people will live longer/happier because AGI helps reduce famine/disease. Somehow it is always the wealthy, employed, at-risk-of-disruption people who are worried, not the poor or starving or oppressed. I’m just super not impressed by the AI risk crowd, at least the core one on LW / SSC / etc.
- FinchNova12 3y ago
  While I agree that the rhetoric around AI Safety would be better if it tried to address some of the benefits (and not embody the full doomer vibe), I don't think many of the 'core thinkers' are unaware of the benefits in AGI. I don't fully agree with this paper's conclusions, but I think https://nickbostrom.com/astronomical/waste https://nickbostrom.com/astronomical/waste is one piece that embodies this style of thinking well!
  - brookst 3y ago
    Thanks for the link -- that is a good paper (in the sense of making its point, though I also don't entirely agree), and it hurts the AI risk position that that kind of thinking doesn't get airtime. It may be that those 'core thinkers' are aware, but if so it's counter-productive and of questionable integrity to sweep that side of the argument under the rug.

patrec 3y ago

> We should be very suspicious of policy that is based on propositions so obvious that it's borderline offensive to question them. Mostly if the "obviousness" just masks a social taboo, which I don't see being the case here. Do you? > The problem is, lots of "obvious" things have turned out to be very wrong. A much bigger problem is that lots more "counter-intuitive" things that people like to believe because they elevate them over the unwashed masses have turned and continue to turn out to be very wrong and that this does not prevent them from forming the basis for important policy decisions. I'm all for questioning even what appears intuitively obvious (especially if much rides on getting it right, as presumably it does here). But frankly, of the many bizarre reasons I have heard why we should not worry about AI the claim that it seems far too obvious that we should must be the single most perverse one yet. > Everything else is just intellectual fig leaves for the core argument that intuitively, without evidence, this proposition is obvious. Maybe your appraisal of what counts as evidence is defective? For example, there's been a pattern of people confidently predicting AIs won't be able to perform various particular feats of the human mind (either fundamentally or in the next few decades) only to be proven wrong over increasingly shorter time-spans. And with AIs often not just reaching but far surpassing human ability. I'm happy to provide examples. Can you explain to me why you think this is does not count, in any way, as evidence that AIs have the potential to reach a level of capability that renders them quite dangerous?

revelio 3y ago
> Mostly if the "obviousness" just masks a social taboo, which I don't see being the case here. Do you? The social taboo here is saying that a position taken by lots of highly educated people is nonsense because they're all locked in a dumb purity spiral that leads to motivated reasoning. This is actually one of societies biggest taboos! Look at what happens to people who make that argument publicly under their own name in other contexts; they tend to get fired and cancelled really fast. > there's been a pattern of people confidently predicting AIs won't be able to perform various particular feats of the human mind (either fundamentally or in the next few decades) only to be proven wrong over increasingly shorter time-spans That sword cuts both ways! There have been lots of predictions in the last decade that AI will contribute novel and hithertofore unknown solutions to things like climate change or curing cancer. Try getting GPT-4 to spit out a novel research-quality solution to anything, even a simple product design problem, and you'll find it can't. > the claim that it seems far too obvious that we should They're not arguing that. They're saying that AI risk proponents don't actually have good arguments, which is why they so regularly fall back on "it's so obvious we shouldn't need to explain why it's important". If your argument consists primarily of "everyone knows that" then this is a good indication you might be wrong.