I think the better way to look at it is that probabilistic models seem to be an accurate model for human thought. We don't really know how humans think, but we

I think the better way to look at it is that probabilistic models seem to be an accurate model for human thought. We don't really know how humans think, but we know that they probably aren't violating information theoretic principles, and we observe similar phenomena when we compare humans with LLMs.