> There are not repeated practical demonstrations, nor is there any really good theoretical reasons, to think that value alignment mechanisms would be seriously

> There are not repeated practical demonstrations, nor is there any really good theoretical reasons, to think that value alignment mechanisms would be seriously and systematically detrimental to performance & cost in the same way that we have for reinforcement learning/active learning/sequential trials/etc. You can reuse my little trivial proof to argue that UFAIs >= FAIs, but of course they could just be == on intelligence. Then FAIs can be a relatively stable equilibrium because they are not instantly outclassed by any fast-growing UFAI. Some vaguely similar arguments: 1. The race by organizations to develop AI is slowed >10% by worrying about friendliness. In a big efficient global marketplace, the winning organizations will almost certainly be ones who ignored friendliness. 2. A seed AI which is amoral can take immoral actions not available to a moral AI. Even a tiny advantage amplifies over time because of the exponential nature of recursive self-improvement. So the first super intelligent AI will be immoral. 3. Nations who obtain nuclear weapons have immense power. Even if most nations decide not to obtain them, or only obtain them for peaceful purposes, at least a few will get them and then use them to take over the Earth. Therefore, the Earth will be taken over completely by a nuclear-armed nation. All of these arguments have some merit, but none are strong/inevitable. They all require quantification and comparison against countervailing forces (e.g., the effect of laws, the moral motivations of nearby human programmers, the counterthreat provided by many allied nations/AIs). > (It's weird to try to reason that 'value alignment looks hard for the same reason that Tool AI looks hard; but we can do value alignment, thus, we can do Tool AI as well'. Maybe we can't do either? We desperately want value alignment to work, but the universe doesn't owe us anything.) I'm not making any sorts of claims like that.