First, I agree that we’re currently discussing a sophisticated algorithm that predicts words (though I’m interested and curious about some of the seemingly emer

First, I agree that we’re currently discussing a sophisticated algorithm that predicts words (though I’m interested and curious about some of the seemingly emergent behaviors discussed in recent papers). But what is factually true is not the only thing that matters here. What people believe is also at issue. If an AI gives someone advice, and that advice turns out to be catastrophically harmful, and the person takes the advice because they believe the AI is intelligent, it doesn’t really matter that it’s not. Alignment with human values may involve exploring ways to make the predictions safer in the short term. Long term towards AGI, alignment with human values becomes more literal and increasingly important. But the time to start tackling that problem is now, and at every step on the journey.