> An AI should be opinionated, at least towards the core values expressed in Western constitutions I think the AI will have to be opinionated, as others have s

> An AI should be opinionated, at least towards the core values expressed in Western constitutions I think the AI will have to be opinionated, as others have said (and as OpenAI and others are actively attempting). But as an information problem, I think it's much harder than just making it opinionated toward current democratic values. Even if we grant that democratic values are currently better than past ones (which I think is true), we could be stuck in a local maximum and AI could make that maximum much more sticky. Imagine, for example, if we had AI in 2015 before the US had marriage equality, and AI lectured us all about the pros and cons of allowing same sex marriage. I think somehow, the AI needs to have its own sense of what's right, and it has to be better than just taking the average of mainstream human ideas. But I think we're currently nowhere close to knowing how that would work.