> It is something believed without any evidentiary basis which enables rationally weighing But that's exactly true of either position you take. Imagine I'm arg

> It is something believed without any evidentiary basis which enables rationally weighing But that's exactly true of either position you take. Imagine I'm arguing that aliens probably exist and you're arguing they probably don't. There is no such thing as a belief based on evidence in this case; we only have logical reasoning and assumptions and beliefs. > Depends what you mean by “set its own goals”, I could certainly see an argument that the natural intepretation is impossible in a universe fully governed by physical laws, whether the intelligence is artificial or not. This feels like moving the goal posts... all it needs to be able to do is make decisions that are dangerous to humanity. I felt that definition was obvious. All it needs to do do is mimic an oil executive, because, as you say, human intent is also an existential risk. Imagine an oil executive that could get the upper hand in all possible negotiations and desired to make the entire global economy subservient to the needs of oil production. > delaying superhuman AI could increase existential risk. Oh, yeah I totally agree that we can't truly know one way or the other. But it kind of feels like deciding to invent and deploy nuclear bombs as a deterrent to war. Did it end war? Absolutely not. Many would argue it didn't even that specific war. We are of course fortunate that nuclear war has not yet destroyed civilization, but a) there's still time, b) we aren't necessarily better off than if we had no nukes, and c) not all risks are equal - AI could be more dangerous. > as it would require that every resource that could advance superhuman AI Actually, you're right. I was definitely overstating our progress and didn't think through that argument clearly. But maybe it would be safer if we were devoting all global resources toward a coordinated AGI alignment and development effort...