If you squint a bit, you can see some parallels to the AI alignment issue. Like it's clearly important to align AI, but what exactly should we align it to? How

If you squint a bit, you can see some parallels to the AI alignment issue. Like it's clearly important to align AI, but what exactly should we align it to? How do we come up with one score that measures goodness? And it turns out that if you get the score just a tiny bit wrong, it can lead to big issues and people hating it so much they wish there wasn't any score at all.