LLMs sample each next token from a probability distribution. So there's an inherent random process at work. As long as your LLM's "next Turing machine step" out

LLMs sample each next token from a probability distribution. So there's an inherent random process at work. As long as your LLM's "next Turing machine step" output token has anything less than 100.000% odds of being the correct one, you'll get mistakes and unreliability. You could change your sampling to top-token-only. But (and maybe this is hand-wavey) I still suspect the incorrect token will be sometimes predicted on top.