Not that I disagree, but it's not just a danger of AI, it's a danger of complex self-optimizing systems in general. We've got a pretty robust complex self-opti

Not that I disagree, but it's not just a danger of AI, it's a danger of complex self-optimizing systems in general. We've got a pretty robust complex self-optimizing system already, which goes by names like "society" and "the free market". You point it at bad data and it will reinforce that data. If demographic A is poorer than demographic B, market forces will determine that giving loans to demographic B will get you better risk-adjusted returns, causing demographic B to become even more relatively prosperous, justifying future decisions to prefer giving loans to demographic B. Which is to say that, while I think it's good for folks thinking about AI alignment to worry about this problem, all of us should worry about this problem.