I think it is extremely unlikely we are going to be able to convince every interested party that they should give up the golden goose for the sake of possible c

I think it is extremely unlikely we are going to be able to convince every interested party that they should give up the golden goose for the sake of possible calamity. I think there are risks here, not trying to minimize that, but the coordination problem becomes untenable when the risks/benefits are so large. It is essentially the same problem as the atom bomb: it would have been better if we all agreed not to do it, but thats just not possible. Why should China trust the US or vice versa? Who wants to live in a world where your competitors have world-changing technology but you don't? But here we have a technology with immense militaristic and economic value, so the everyone-wants-it problem is even more pronounced. I don't _like_ this, I just don't see how we can achieve an AI moratorium outside of bombing the data centers (which I also don't think is a good idea). We need to choose the policy with the best distribution of possible outcomes: - The US leads an effort to stop AI development: too much risk that other parties do it anyway - The US continues to lead AI development: hope that P(takeoff) is low and that the good intentions of some US labs are able to achieve safe development I prefer the latter -- this is far from the best hypothetical outcome, but I think it is the best we can do when constrained by reality.