3 ms·

It seems most of these arguments apply equally well to the problem of solving AI value alignment, or of preventing the development of AI at all. (I.e., it's ch

It seems most of these arguments apply equally well to the problem of solving AI value alignment, or of preventing the development of AI at all. (I.e., it's cheaper and faster to race ahead without worrying about value alignment.) But that doesn't make us conclude that value alignment is impossible, just hard to achieve soon enough in the real world. Yes, we should be aware of the limitations and market instability of tool AI, but I think it's unjustified to suggest that tool AI is essentially impossible ("highly unstable equilibrium") and all we can hope to do is solve value alignment.

gwern 10y ago
> It seems most of these arguments apply equally well to the problem of solving AI value alignment Only somewhat. There are not repeated practical demonstrations, nor is there any really good theoretical reasons, to think that value alignment mechanisms would be seriously and systematically detrimental to performance & cost in the same way that we have for reinforcement learning/active learning/sequential trials/etc. You can reuse my little trivial proof to argue that UFAIs >= FAIs, but of course they could just be == on intelligence. Then FAIs can be a relatively stable equilibrium because they are not instantly outclassed by any fast-growing UFAI. Contrast that with Tool AIs where everyone who cooperates is at enormous disadvantage to one defector, and defectors have large and growing rewards. > But that doesn't make us conclude that value alignment is impossible, just hard to achieve soon enough in the real world. I'm not sure anyone seriously thinks that value alignment will be easy to achieve, much less at sufficient scale. (It's weird to try to reason that 'value alignment looks hard for the same reason that Tool AI looks hard; but we can do value alignment, thus, we can do Tool AI as well'. Maybe we can't do either? We desperately want value alignment to work, but the universe doesn't owe us anything.)
- jessriedel 10y ago
  > There are not repeated practical demonstrations, nor is there any really good theoretical reasons, to think that value alignment mechanisms would be seriously and systematically detrimental to performance & cost in the same way that we have for reinforcement learning/active learning/sequential trials/etc. You can reuse my little trivial proof to argue that UFAIs >= FAIs, but of course they could just be == on intelligence. Then FAIs can be a relatively stable equilibrium because they are not instantly outclassed by any fast-growing UFAI. Some vaguely similar arguments: 1. The race by organizations to develop AI is slowed >10% by worrying about friendliness. In a big efficient global marketplace, the winning organizations will almost certainly be ones who ignored friendliness. 2. A seed AI which is amoral can take immoral actions not available to a moral AI. Even a tiny advantage amplifies over time because of the exponential nature of recursive self-improvement. So the first super intelligent AI will be immoral. 3. Nations who obtain nuclear weapons have immense power. Even if most nations decide not to obtain them, or only obtain them for peaceful purposes, at least a few will get them and then use them to take over the Earth. Therefore, the Earth will be taken over completely by a nuclear-armed nation. All of these arguments have some merit, but none are strong/inevitable. They all require quantification and comparison against countervailing forces (e.g., the effect of laws, the moral motivations of nearby human programmers, the counterthreat provided by many allied nations/AIs). > (It's weird to try to reason that 'value alignment looks hard for the same reason that Tool AI looks hard; but we can do value alignment, thus, we can do Tool AI as well'. Maybe we can't do either? We desperately want value alignment to work, but the universe doesn't owe us anything.) I'm not making any sorts of claims like that.

gwern 10y ago

> It seems most of these arguments apply equally well to the problem of solving AI value alignment Only somewhat. There are not repeated practical demonstrations, nor is there any really good theoretical reasons, to think that value alignment mechanisms would be seriously and systematically detrimental to performance & cost in the same way that we have for reinforcement learning/active learning/sequential trials/etc. You can reuse my little trivial proof to argue that UFAIs >= FAIs, but of course they could just be == on intelligence. Then FAIs can be a relatively stable equilibrium because they are not instantly outclassed by any fast-growing UFAI. Contrast that with Tool AIs where everyone who cooperates is at enormous disadvantage to one defector, and defectors have large and growing rewards. > But that doesn't make us conclude that value alignment is impossible, just hard to achieve soon enough in the real world. I'm not sure anyone seriously thinks that value alignment will be easy to achieve, much less at sufficient scale. (It's weird to try to reason that 'value alignment looks hard for the same reason that Tool AI looks hard; but we can do value alignment, thus, we can do Tool AI as well'. Maybe we can't do either? We desperately want value alignment to work, but the universe doesn't owe us anything.)

jessriedel 10y ago
> There are not repeated practical demonstrations, nor is there any really good theoretical reasons, to think that value alignment mechanisms would be seriously and systematically detrimental to performance & cost in the same way that we have for reinforcement learning/active learning/sequential trials/etc. You can reuse my little trivial proof to argue that UFAIs >= FAIs, but of course they could just be == on intelligence. Then FAIs can be a relatively stable equilibrium because they are not instantly outclassed by any fast-growing UFAI. Some vaguely similar arguments: 1. The race by organizations to develop AI is slowed >10% by worrying about friendliness. In a big efficient global marketplace, the winning organizations will almost certainly be ones who ignored friendliness. 2. A seed AI which is amoral can take immoral actions not available to a moral AI. Even a tiny advantage amplifies over time because of the exponential nature of recursive self-improvement. So the first super intelligent AI will be immoral. 3. Nations who obtain nuclear weapons have immense power. Even if most nations decide not to obtain them, or only obtain them for peaceful purposes, at least a few will get them and then use them to take over the Earth. Therefore, the Earth will be taken over completely by a nuclear-armed nation. All of these arguments have some merit, but none are strong/inevitable. They all require quantification and comparison against countervailing forces (e.g., the effect of laws, the moral motivations of nearby human programmers, the counterthreat provided by many allied nations/AIs). > (It's weird to try to reason that 'value alignment looks hard for the same reason that Tool AI looks hard; but we can do value alignment, thus, we can do Tool AI as well'. Maybe we can't do either? We desperately want value alignment to work, but the universe doesn't owe us anything.) I'm not making any sorts of claims like that.