I don't think that's the right way to think about LLM logits. Fundamentally the logits represent probability of similarity with the text it's been trained on, g

I don't think that's the right way to think about LLM logits. Fundamentally the logits represent probability of similarity with the text it's been trained on, given the current prefix. Mixed in with any correspondence with truth is not only tone, phrasing, dialect, language syntax, but also stuff like the likelihood that specific details are related to general concepts. Even if we're talking about a person with three legs, or a horse riding a man, it'll be hard for the LLM to not assign a fairly high probability to sentences that describe two legs, or a man riding the horse and not the other way around.