Why is the "question to terrify the AI ethicists" always some scifi bullshit and not the thing that's actually here. This thing manifests a supposed interpreta

Why is the "question to terrify the AI ethicists" always some scifi bullshit and not the thing that's actually here. This thing manifests a supposed interpretation from the barest of signals. (We know the language models can output something "meaningful" given basically noise.) Imagine being forced to speak in only the most likely sentences, or having everything you try to say instead transformed into a structurally-similar "more likely" utterance! Why is this not per se scary and unethical as hell?