5 ms·

The atom bomb was once a hypothetical construct. The Einstein-Szilard letter was talking about a danger that they had no way of proving was even possible. Proof

The atom bomb was once a hypothetical construct. The Einstein-Szilard letter was talking about a danger that they had no way of proving was even possible. Proof is irrelevant outside of mathematics; the only relevant question is cost to expected benefit - or expected harm. With dangerous research, when a conclusive demonstration of possibility is achieved, it is usually too late to do much in the way of prevention. That's why we don't expect the FDA to "prove" that harm from a new medication "is even possible" before regulating it; rather, we expect the pharmaceutical company to provide evidence for its expected harmlessness before we even make the attempt. Think of the FDA what you like, this is certainly a better standard than putting the burden of proof on harmful outcomes.

danaris 4y ago
Part of the problem with this is that conclusive demonstration that AGI is possible is not even in the same ballpark as a conclusive demonstration that it would have consequences like the EA folks seem to believe. From everything I've read about their views, they seem to believe that either the moment an AGI is created or very, very soon thereafter, it will achieve the Singularity and "ascend" to something near godhood from our perspective, and it will perceive humanity as a threat to it and immediately seek our destruction—which, because of its near-godhood, it will be able to achieve far faster than we have the ability to respond to. Thus, what we have to do is either prevent AGI from ever coming to pass (because as soon as it exists, it's effectively too late), or make absolutely certain that by the time it does, we have strong measures in place to either fight back against it, or get humanity the hell out of Dodge. This is built on such a foundation of shaky assumptions that giving it credibility is ludicrous.
- FeepingCreature 4y ago
  I agree with your description of the AGI/ASI position. My take on it is the other way around: every step of the argument seems plainly obvious.
  - yifanl 4y ago
    Plainly, that is not the case, as you are choosing to not spend your every waking moment to stop the antichrist AI coming into existence. So either it's less obvious than you're claiming, or you are taking the threat of creating an evil AGI not seriously enough according to your own beliefs.
    - FeepingCreature 4y ago
      I don't spend every waking moment stopping lots of things that could kill us all. Nuclear war, climate change, gain-of-function research. I just don't care that much about things that kill everyone. I don't think almost anyone does. In the Cold War, the vast majority of the population was not doing absolutely everything to achieve bilateral disarmament. This would lead you to believe that they didn't really expect nuclear war. But maybe they were just apathetic and helpless.
pvg 4y ago
The atom bomb was once a hypothetical construct. The Einstein-Szilard letter was talking about a danger that they had no way of proving was even possible. This isn't an accurate representation, really. The letter was written exactly because it was clear that a fission bomb is very likely within the short-term reach of current technological and industrial capability. This is very much not the case with AGI.
- FeepingCreature 4y ago
  That's a difference of degrees, not kind. You can always ask for a higher class of proof.
  - pvg 4y ago
    How is it a difference of degrees? This is just plain inaccurate: The Einstein-Szilard letter was talking about a danger that they had no way of proving was even possible. Supporting your argument with an inaccurate thing is not 'a difference of degrees'. It's a difference between a good argument and a bad one.
    - FeepingCreature 4y ago
      The disagreement is about the difficulty of "proving". I could say that unfriendly AI is clearly possible because evil humans exist. You could say that unfriendly AI is not proven possible until one has actually been built. But if you applied that standard to the letter, it would likewise fail it. Now, I agree that the case for AI is weaker, maybe much weaker, than the case for nuclear fission when the letter was written - there is, for instance, no clear research agenda that is held by preeminent researchers in the field to lead to it - though there are research groups, such as Deepmind, that have AGI as their explicit goal - but that also has nothing to do with being "proven" possible. My argument is that for contested and dangerous outcomes, proof of possibility is an unreasonable standard, in good part because it massively underspecifies the goalposts.
      - pvg 4y ago

danaris 4y ago

Part of the problem with this is that conclusive demonstration that AGI is possible is not even in the same ballpark as a conclusive demonstration that it would have consequences like the EA folks seem to believe. From everything I've read about their views, they seem to believe that either the moment an AGI is created or very, very soon thereafter, it will achieve the Singularity and "ascend" to something near godhood from our perspective, and it will perceive humanity as a threat to it and immediately seek our destruction—which, because of its near-godhood, it will be able to achieve far faster than we have the ability to respond to. Thus, what we have to do is either prevent AGI from ever coming to pass (because as soon as it exists, it's effectively too late), or make absolutely certain that by the time it does, we have strong measures in place to either fight back against it, or get humanity the hell out of Dodge. This is built on such a foundation of shaky assumptions that giving it credibility is ludicrous.

FeepingCreature 4y ago
I agree with your description of the AGI/ASI position. My take on it is the other way around: every step of the argument seems plainly obvious.
- yifanl 4y ago
  Plainly, that is not the case, as you are choosing to not spend your every waking moment to stop the antichrist AI coming into existence. So either it's less obvious than you're claiming, or you are taking the threat of creating an evil AGI not seriously enough according to your own beliefs.
  - FeepingCreature 4y ago
    I don't spend every waking moment stopping lots of things that could kill us all. Nuclear war, climate change, gain-of-function research. I just don't care that much about things that kill everyone. I don't think almost anyone does. In the Cold War, the vast majority of the population was not doing absolutely everything to achieve bilateral disarmament. This would lead you to believe that they didn't really expect nuclear war. But maybe they were just apathetic and helpless.

pvg 4y ago

The atom bomb was once a hypothetical construct. The Einstein-Szilard letter was talking about a danger that they had no way of proving was even possible. This isn't an accurate representation, really. The letter was written exactly because it was clear that a fission bomb is very likely within the short-term reach of current technological and industrial capability. This is very much not the case with AGI.

FeepingCreature 4y ago
That's a difference of degrees, not kind. You can always ask for a higher class of proof.
- pvg 4y ago
  How is it a difference of degrees? This is just plain inaccurate: The Einstein-Szilard letter was talking about a danger that they had no way of proving was even possible. Supporting your argument with an inaccurate thing is not 'a difference of degrees'. It's a difference between a good argument and a bad one.
  - FeepingCreature 4y ago
    The disagreement is about the difficulty of "proving". I could say that unfriendly AI is clearly possible because evil humans exist. You could say that unfriendly AI is not proven possible until one has actually been built. But if you applied that standard to the letter, it would likewise fail it. Now, I agree that the case for AI is weaker, maybe much weaker, than the case for nuclear fission when the letter was written - there is, for instance, no clear research agenda that is held by preeminent researchers in the field to lead to it - though there are research groups, such as Deepmind, that have AGI as their explicit goal - but that also has nothing to do with being "proven" possible. My argument is that for contested and dangerous outcomes, proof of possibility is an unreasonable standard, in good part because it massively underspecifies the goalposts.
    - pvg 4y ago