4 ms·

These are the sober and informed takes that we need more of. > the claims from Coxon and his ilk are the most extraordinary a technologist can make, and we mus

These are the sober and informed takes that we need more of. > the claims from Coxon and his ilk are the most extraordinary a technologist can make, and we must demand evidence commensurate with the claims. Yes, exactly. These claims do not have sufficient evidence. > ...you had nothing to fear then — and (at least with respect to extinction risk!) you have nothing to fear now. Wait, this is another extraordinary claim without evidence, right? Unless you're going to dispute the power of AI you do have to acknowledge the danger of AI, and that does include the very real possibility (however small) of existential risk.

bcantrill 21d ago
Saying that we don't need to fear human extinction in the next ten years (or even the next hundred?) does not feel like an extraordinary claim?
- thin_carapace 21d ago
  the earth has been 1 decision away from explosion for nearly a hundred years now. do you anticipate a change to this agenda very soon? personally I imagine the same trajectory continuing.
  - goatlover 21d ago
    As bad as it would be, it's unlikely that a full scale nuclear war kills all humans across the planet.
    - Jtsummers 21d ago
      And a full scale nuclear war would require a few more than just one decision. So far, we've never been one decision away from annihilation.
      - thin_carapace 21d ago
        certainly I behave like the article's author and yourself all the time, this isn't meant to be some sort of moral point. watching all these smart people talk so confidently regarding things nobody knows about, it reminds of the confidence ai shows in hallucination. would you say that the singular choice of vasili arkhipov did not prevent annihilation?
        Jtsummers 21d ago
        > would you say that the singular choice of vasili arkhipov did not prevent annihilation? He prevented a catastrophe, but not annihilation. We were not at risk of that in 1962 and even if he had decided to go with the others, more decisions would have been needed (not just his) to fully escalate to full scale nuclear war. I will reiterate: We have never been one decision away from full scale nuclear war. But in case you don't understand why, it's because no one person can actually launch all the missiles. And considering the two major arsenals (US and USSR), there has never been a time when two people could make the same decision (launch) and actually launch all the missiles. The orders still have to go out and acted on, many decisions have to be made in order to have full scale nuclear war and come close to annihilation.
        thin_carapace 21d ago
        I like the way you frame your opinion as a truth with an obligation to be understood. still I disagree that a decision can only be said to affect its direct successors.
- 0xDEAFBEAD 21d ago
  No one knows what is going to happen. But a lot of species have been going extinct lately. https://pbs.twimg.com/media/FDd58a4WQAAWaXh.jpg https://pbs.twimg.com/media/FDd58a4WQAAWaXh.jpg
  - nozzlegear 13d ago
    wtf is this chart lmao, did you just google a line-going-up chart to stick your shower thought onto? > My life is pretty typical, I have a good grasp of what is normal vs weird, and people imagining very different futures with digital people and Al and space travel are indulging themselves in silly sci-fi speculation. This but unironically
- mortenjorck 21d ago
  The students in the computer lab had no guarantee that they wouldn't, say, download a copy of Napster infected with the CIH virus later. The fact that they were not under imminent threat from some kind of Hollywood-style network worm did not mean they were immune from more realistic attack vectors. Likewise, one can quite reasonably say there is no credible existential, Hollywood-style threat from AI in the foreseeable future while recognizing far lower-stakes, yet important risks that need to be addressed.
- abletonlive 21d ago
  No it does not seem like an extraordinary claim, unless this is your first time hearing such claims. For the rest of us, we've heard this every decade and it turns out to be entirely untrue. Nuclear, Overpopulation, Peak Oil, Y2K Bug, Global Warming. No, we aren't going extinct in the next 10 years.
- jacobgold 21d ago
  If you think AI will be powerful enough to change the world in huge ways, then it does seem reasonable to assume some level of risk of destroying the world too. Stating that the probability is zero when we simply don't know what the probabilities are does seem like an extraordinary claim. Imagine how reassuring it would be to people if we had evidence that there's no existential risk?
  - JeremyNT 20d ago
    > If you think AI will be powerful enough to change the world in huge ways, then it does seem reasonable to assume some level of risk of destroying the world too. There's an opportunity cost in the doomsday prophesying, though. The media coverage of "these extremely capable robots might kill us (according to the guys who sell the robots)" comes at the cost of coverage of the real, present issues surrounding the tech oligarchs that we're already facing. Sure, there are some failure modes that lead to some really bad stuff, but human extinction seems exceedingly unlikely to be one of them, and "10%" is pulled straight out of Dario's derriere. Far more likely are economic disruptions that impact the tenuous balance between labor and capital and lead to unpredictable societal upheaval.
duhhhhh1212 21d ago
bruh what? Are we reading the same blog? Bryan says the responsibility is the person making the extraordinary claim. If tomorrow Jacob Coxon comes on CNN and says "Jacob Gold is an extinction risk to humanity". What are we supposed to do? Put you in a bunker and never let you see the light of day until someone proves otherwise? If someone doesn't accept an extraordinary claim without evidence, that doesn't mean they are making an extraordinary claim.
- jacobgold 21d ago
  There seem to be two extraordinary claims which lack evidence: a) 10% existential risk b) 0% existential risk
  - adithyareddy 21d ago
    (a) is an extraordinary claim, (b) is not an extraordinary claim.
    - jacobgold 21d ago
      There are very clear and plausible pathways from AI to human extinction level events, regardless of how small you think the probability is. So yes, it very obviously is an extraordinary claim to say there is 0% existential risk.
      - deleted 21d ago
        [deleted]
      - duhhhhh1212 21d ago
        Good god. You are burden shifting. a) big claim no evidence. b). says a has no evidence; therefore, I will continue to not let fear manipulate me into believing a claim without evidence. I have a bridge to sell to anyone who believes a big claim without evidence.
        21d ago

bcantrill 21d ago

Saying that we don't need to fear human extinction in the next ten years (or even the next hundred?) does not feel like an extraordinary claim?

thin_carapace 21d ago
the earth has been 1 decision away from explosion for nearly a hundred years now. do you anticipate a change to this agenda very soon? personally I imagine the same trajectory continuing.
- goatlover 21d ago
  As bad as it would be, it's unlikely that a full scale nuclear war kills all humans across the planet.
  - Jtsummers 21d ago
    And a full scale nuclear war would require a few more than just one decision. So far, we've never been one decision away from annihilation.
    - thin_carapace 21d ago
      certainly I behave like the article's author and yourself all the time, this isn't meant to be some sort of moral point. watching all these smart people talk so confidently regarding things nobody knows about, it reminds of the confidence ai shows in hallucination. would you say that the singular choice of vasili arkhipov did not prevent annihilation?
      - Jtsummers 21d ago
        > would you say that the singular choice of vasili arkhipov did not prevent annihilation? He prevented a catastrophe, but not annihilation. We were not at risk of that in 1962 and even if he had decided to go with the others, more decisions would have been needed (not just his) to fully escalate to full scale nuclear war. I will reiterate: We have never been one decision away from full scale nuclear war. But in case you don't understand why, it's because no one person can actually launch all the missiles. And considering the two major arsenals (US and USSR), there has never been a time when two people could make the same decision (launch) and actually launch all the missiles. The orders still have to go out and acted on, many decisions have to be made in order to have full scale nuclear war and come close to annihilation.
        thin_carapace 21d ago
        I like the way you frame your opinion as a truth with an obligation to be understood. still I disagree that a decision can only be said to affect its direct successors.
0xDEAFBEAD 21d ago
No one knows what is going to happen. But a lot of species have been going extinct lately. https://pbs.twimg.com/media/FDd58a4WQAAWaXh.jpg https://pbs.twimg.com/media/FDd58a4WQAAWaXh.jpg
- nozzlegear 13d ago
  wtf is this chart lmao, did you just google a line-going-up chart to stick your shower thought onto? > My life is pretty typical, I have a good grasp of what is normal vs weird, and people imagining very different futures with digital people and Al and space travel are indulging themselves in silly sci-fi speculation. This but unironically
mortenjorck 21d ago
The students in the computer lab had no guarantee that they wouldn't, say, download a copy of Napster infected with the CIH virus later. The fact that they were not under imminent threat from some kind of Hollywood-style network worm did not mean they were immune from more realistic attack vectors. Likewise, one can quite reasonably say there is no credible existential, Hollywood-style threat from AI in the foreseeable future while recognizing far lower-stakes, yet important risks that need to be addressed.
abletonlive 21d ago
No it does not seem like an extraordinary claim, unless this is your first time hearing such claims. For the rest of us, we've heard this every decade and it turns out to be entirely untrue. Nuclear, Overpopulation, Peak Oil, Y2K Bug, Global Warming. No, we aren't going extinct in the next 10 years.
jacobgold 21d ago
If you think AI will be powerful enough to change the world in huge ways, then it does seem reasonable to assume some level of risk of destroying the world too. Stating that the probability is zero when we simply don't know what the probabilities are does seem like an extraordinary claim. Imagine how reassuring it would be to people if we had evidence that there's no existential risk?
- JeremyNT 20d ago
  > If you think AI will be powerful enough to change the world in huge ways, then it does seem reasonable to assume some level of risk of destroying the world too. There's an opportunity cost in the doomsday prophesying, though. The media coverage of "these extremely capable robots might kill us (according to the guys who sell the robots)" comes at the cost of coverage of the real, present issues surrounding the tech oligarchs that we're already facing. Sure, there are some failure modes that lead to some really bad stuff, but human extinction seems exceedingly unlikely to be one of them, and "10%" is pulled straight out of Dario's derriere. Far more likely are economic disruptions that impact the tenuous balance between labor and capital and lead to unpredictable societal upheaval.

duhhhhh1212 21d ago

bruh what? Are we reading the same blog? Bryan says the responsibility is the person making the extraordinary claim. If tomorrow Jacob Coxon comes on CNN and says "Jacob Gold is an extinction risk to humanity". What are we supposed to do? Put you in a bunker and never let you see the light of day until someone proves otherwise? If someone doesn't accept an extraordinary claim without evidence, that doesn't mean they are making an extraordinary claim.

jacobgold 21d ago
There seem to be two extraordinary claims which lack evidence: a) 10% existential risk b) 0% existential risk
- adithyareddy 21d ago
  (a) is an extraordinary claim, (b) is not an extraordinary claim.
  - jacobgold 21d ago
    There are very clear and plausible pathways from AI to human extinction level events, regardless of how small you think the probability is. So yes, it very obviously is an extraordinary claim to say there is 0% existential risk.
    - deleted 21d ago
      [deleted]
    - duhhhhh1212 21d ago
      Good god. You are burden shifting. a) big claim no evidence. b). says a has no evidence; therefore, I will continue to not let fear manipulate me into believing a claim without evidence. I have a bridge to sell to anyone who believes a big claim without evidence.
      - 21d ago