4 ms·

Something can go wrong (premise). Therefore, something will go wrong (invalid conclusion). https://en.wikipedia.org/wiki/Appeal_to_probability https://en.wiki

by 23B1 4y ago

Something can go wrong (premise). Therefore, something will go wrong (invalid conclusion). https://en.wikipedia.org/wiki/Appeal_to_probability https://en.wikipedia.org/wiki/Appeal_to_probability Sure seems to me that the whole Lesswrong crowd has forgotten their founding principles around cognitive biases. Or maybe they're just having a bit of fun riding the hypecycle.

moonchrome 4y ago
Something can go wrong. The outcome of the thing going wrong is literally the worst thing that can happen to the entire humanity. Therefore we should do everything in our power to avoid doing that thing untill we can eliminate the risk.
- 23B1 4y ago
  By that rather sloppy logic, we should be urgently working on capping the Yellowstone supervolcano and constructing impenetrable meteorite defense shields. I acknowledge that it's a possibility; that's not the argument at hand.
  - moonchrome 4y ago
    Except there's concrete evidence that the execution of safety is subpar due to market pressures to release and monetize (Google panic release). There's also evidence that progress is exceeding people's expectations and we might not even realize what we have is dangerous untill too late. If we had a probable collision trajectory of a meteor or signs of Yellowstone activating then yeah those would be good ideas.
    - 23B1 4y ago
      The chain of 'logic' that leads to the outcome you propose is bonkers, though. Here's Elizer's logic from his recent polemic in Time https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-not-enough/ https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-no... "A sufficiently intelligent AI won’t stay confined to computers for long. In today’s world you can email DNA strings to laboratories that will produce proteins on demand, allowing an AI initially confined to the internet to build artificial life forms or bootstrap straight to postbiological molecular manufacturing." This is Ridley Scott, not Aristotle.
  - flangola7 4y ago
    If we saw Yellowstone spawning a sequence of ever massively larger quakes and ground disturbance, or if we observed a massive distant asteroid continually closing in on Earth with higher and higher collision probability, then yes we should be pulling out all the stops and reconfiguring society to do whatever is necessary to save the species.
    - 23B1 4y ago
      Agreed! But there is no evidence of the eventuality described by these AI doomers. The same hyperbole was employed by Ehrlich in the late 60s when we were talking about the population bomb, or Nevil Shute in 1945 on the nuclear apocalypse - both scenarios seem to be increasingly unlikely. I think we can both agree on the fallibility of human beings – but the leaps of logic require such a tenuous series of gymnastic leaps that the polemic is not only hard to take seriously, but indeed can cloud our ability to perceive actual threats! From Eliezer's recent article in TIME: "A sufficiently intelligent AI won’t stay confined to computers for long. In today’s world you can email DNA strings to laboratories that will produce proteins on demand, allowing an AI initially confined to the internet to build artificial life forms or bootstrap straight to postbiological molecular manufacturing." (https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-not-enough/ https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-no...) C'mon.
      - flangola7 4y ago
        The fact that we got lucky in the past is an opportunity to make sure we don't have to rely on luck in the future, not to continue on as normal and keep playing Russian Roulette. If you see fresh leopard tracks maybe it hasn't smelled you yet, but you shouldn't wait to start readying your rifle ready until it making a pounce at you. I also would definitely not call the risk of nuclear war decreasing, with the Ukraine conflict ongoing and multiple nuclear powers experiencing growing instability. The number of friends I have who are knowledgeable about surviving a nuclear strike and/or own a CBRN mask went from 25% to 90% last year.
marginalia_nu 4y ago
I don't think that is a generous interpretation. Substitute AI experiments with playing russian roulette or something similarly dangerous and the logic seems sound. Blowing your head off is possible, but not probable (1 in 6 chance). The magnitude of the perceived risk matters.
- 23B1 4y ago
  A 1 in 6 chance is a discrete number. The chance of annihilation due to AGI is not. You're defending an appeal to probability fallacy with a false equivalence fallacy. https://en.wikipedia.org/wiki/False_equivalence https://en.wikipedia.org/wiki/False_equivalence
  - marginalia_nu 4y ago
    https://en.m.wikipedia.org/wiki/Argument_from_fallacy https://en.m.wikipedia.org/wiki/Argument_from_fallacy
deschutes 4y ago
Who knows. It's completely uncharted territory. It's not hard to imagine how malevolent AGI could end us. The illusion of control over something that may well have incomprehensible intelligence is a joke. Tell me, what control does the farm animal have over its own fate?
- 23B1 4y ago
  That it is completely uncharted territory is exactly my point – which is why when people string together a series of predictions to build a narrative that happens to appeal to our most base instincts, e.g. fear, we should be highly skeptical. Obviously there is a wide range of possibilities from the utopian, to the benign, to the apocalyptic.
cubefox 4y ago
There are a lot of arguments that the probability of doom is substantial. Orthogonality thesis, instrumental convergence, goodharting, mesa-optimization, etc. Here is an introduction: https://www.lesswrong.com/posts/vMM6HmSQaKmKadvBi/the-core-of-the-alignment-problem-is-1 https://www.lesswrong.com/posts/vMM6HmSQaKmKadvBi/the-core-o...

by 23B1 4y ago

moonchrome 4y ago

Something can go wrong. The outcome of the thing going wrong is literally the worst thing that can happen to the entire humanity. Therefore we should do everything in our power to avoid doing that thing untill we can eliminate the risk.

23B1 4y ago
By that rather sloppy logic, we should be urgently working on capping the Yellowstone supervolcano and constructing impenetrable meteorite defense shields. I acknowledge that it's a possibility; that's not the argument at hand.
- moonchrome 4y ago
  Except there's concrete evidence that the execution of safety is subpar due to market pressures to release and monetize (Google panic release). There's also evidence that progress is exceeding people's expectations and we might not even realize what we have is dangerous untill too late. If we had a probable collision trajectory of a meteor or signs of Yellowstone activating then yeah those would be good ideas.
  - 23B1 4y ago
    The chain of 'logic' that leads to the outcome you propose is bonkers, though. Here's Elizer's logic from his recent polemic in Time https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-not-enough/ https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-no... "A sufficiently intelligent AI won’t stay confined to computers for long. In today’s world you can email DNA strings to laboratories that will produce proteins on demand, allowing an AI initially confined to the internet to build artificial life forms or bootstrap straight to postbiological molecular manufacturing." This is Ridley Scott, not Aristotle.
- flangola7 4y ago
  If we saw Yellowstone spawning a sequence of ever massively larger quakes and ground disturbance, or if we observed a massive distant asteroid continually closing in on Earth with higher and higher collision probability, then yes we should be pulling out all the stops and reconfiguring society to do whatever is necessary to save the species.
  - 23B1 4y ago
    Agreed! But there is no evidence of the eventuality described by these AI doomers. The same hyperbole was employed by Ehrlich in the late 60s when we were talking about the population bomb, or Nevil Shute in 1945 on the nuclear apocalypse - both scenarios seem to be increasingly unlikely. I think we can both agree on the fallibility of human beings – but the leaps of logic require such a tenuous series of gymnastic leaps that the polemic is not only hard to take seriously, but indeed can cloud our ability to perceive actual threats! From Eliezer's recent article in TIME: "A sufficiently intelligent AI won’t stay confined to computers for long. In today’s world you can email DNA strings to laboratories that will produce proteins on demand, allowing an AI initially confined to the internet to build artificial life forms or bootstrap straight to postbiological molecular manufacturing." (https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-not-enough/ https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-no...) C'mon.
    - flangola7 4y ago
      The fact that we got lucky in the past is an opportunity to make sure we don't have to rely on luck in the future, not to continue on as normal and keep playing Russian Roulette. If you see fresh leopard tracks maybe it hasn't smelled you yet, but you shouldn't wait to start readying your rifle ready until it making a pounce at you. I also would definitely not call the risk of nuclear war decreasing, with the Ukraine conflict ongoing and multiple nuclear powers experiencing growing instability. The number of friends I have who are knowledgeable about surviving a nuclear strike and/or own a CBRN mask went from 25% to 90% last year.

marginalia_nu 4y ago

I don't think that is a generous interpretation. Substitute AI experiments with playing russian roulette or something similarly dangerous and the logic seems sound. Blowing your head off is possible, but not probable (1 in 6 chance). The magnitude of the perceived risk matters.

23B1 4y ago
A 1 in 6 chance is a discrete number. The chance of annihilation due to AGI is not. You're defending an appeal to probability fallacy with a false equivalence fallacy. https://en.wikipedia.org/wiki/False_equivalence https://en.wikipedia.org/wiki/False_equivalence
- marginalia_nu 4y ago
  https://en.m.wikipedia.org/wiki/Argument_from_fallacy https://en.m.wikipedia.org/wiki/Argument_from_fallacy

deschutes 4y ago

Who knows. It's completely uncharted territory. It's not hard to imagine how malevolent AGI could end us. The illusion of control over something that may well have incomprehensible intelligence is a joke. Tell me, what control does the farm animal have over its own fate?

23B1 4y ago
That it is completely uncharted territory is exactly my point – which is why when people string together a series of predictions to build a narrative that happens to appeal to our most base instincts, e.g. fear, we should be highly skeptical. Obviously there is a wide range of possibilities from the utopian, to the benign, to the apocalyptic.

cubefox 4y ago

There are a lot of arguments that the probability of doom is substantial. Orthogonality thesis, instrumental convergence, goodharting, mesa-optimization, etc. Here is an introduction: https://www.lesswrong.com/posts/vMM6HmSQaKmKadvBi/the-core-of-the-alignment-problem-is-1 https://www.lesswrong.com/posts/vMM6HmSQaKmKadvBi/the-core-o...