I agree, but the push for 'AI Safety' as a whole is driven by the x-risk crowd. There are a few people genuinely concerned, however misguidedly, about the outp

I agree, but the push for 'AI Safety' as a whole is driven by the x-risk crowd. There are a few people genuinely concerned, however misguidedly, about the outputs of LLMs. The big money coming into the field from EAs and the sort of background fear is about fast takeoff, shoggoths, paperclip maximizers, etc. The thought is that, if you can't reliably align an LLM to output what some authoritative source wants, then you can't reliably align the inevitable machine god that will destroy us. In that context, user education doesn't do any good.