Don't buy insurance that costs more than the risk does. Don't take cures that are worse than the disease. There's lots of non-zero risks that we just roll wit

Don't buy insurance that costs more than the risk does. Don't take cures that are worse than the disease. There's lots of non-zero risks that we just roll with, because the risk is less than the cost of the remediation. So your "prove the risk is zero" is the wrong standard. > Is the chance that someone will attach it to a nuclear launch or a bioweapon system zero? No. Was the chance that someone would attach an expert system to a nuclear launch zero? How about just ordinary buggy software? How about perfectly-implemented software that implements insane command-and-control doctrine? You seem to be demanding standards for AI that we haven't applied to anything else. > Not to even mention your or anyone else's ability to prove it today that it has no potential to become self-aware. That's going to be hard without an agreed-on definition for "self-aware". How do you suggest that we prove that it can't do something that we can't even define? And now I'm going to talk out of the other side of my mouth. It's a good thing that I think that the risk of it becoming self-aware are very close to zero, because I also think that if it can become self-aware, it's already too late to stop, at least if GPT is close (say, only one major breakthrough away). I mean, let's say the US and the EU decide to stop all research. Is any company going to secretly keep going in their own labs? (For that matter, will the Pentagon?) Even if they stop, will China? India? Brazil? Even if you are absolutely convinced that totally stopping is the correct answer, can you get buy-in from the entire rest of the world? No, you can't. So whatever the potential for harm is, we're going to see it. If you really think that it has a significant possibility of massive harm, then your best bet is to think about mitigations rather than prevention.