3 ms·
LLMs are notoriously non-deterministic, and even at temperature zero you still can't predict exactly where the weights will take you next. An LLM can be made t
by trollbridge 1mo ago
LLMs are notoriously non-deterministic, and even at temperature zero you still can't predict exactly where the weights will take you next.
An LLM can be made to be completely deterministic. I use them in this mode so I can reproduce test cases. Of course it requires complete control over the model, etc. but this myth that a computer program is non-deterministic needs to end.
You can 100% predict where the weights “will take you” given a set of inputs.
- Folcon 1mo ago> You can 100% predict where the weights “will take you” given a set of inputs. Do you mean reproduce? Sorry it's just if you are saying what your statement implying then either the model is very simple, or you've figured out something incredible
- trollbridge 1mo agoSure. Take the inputs and run it through the model. You now have the output which will be perfectly reproduced with the same inputs.
- techpression 1mo agoYou should publish, likely a Nobel price or Turing award waiting, and generational wealth at some tech giant.
- simonw 1mo agoBy "can't predict exactly where the weights will take you next" I meant with your brain. The blind chess analogy suggests you can predict, using your own thought process, the exact output of a prompt.
- trollbridge 1mo agoI could use my brain to run a set weights, given enough time (on the order of millions of years).
- shakna 1mo agoFloating point matrix calculations are non-deterministic. You need to invent new hardware, that doesn't use floating point math, first. [0] [0] https://arxiv.org/html/2506.09501 https://arxiv.org/html/2506.09501
- mrob 1mo ago>Floating point matrix calculations are non-deterministic. This is not inherent to floating-point math. That actual (true) claim in the article is that different hardware and different hardware configurations produce different results. But deterministic inference is possible, e.g. llama.cpp on CPU is deterministic by default.
- shakna 1mo agoRight... That's why I added "matrix" in there. Because we don't have matrix operations in software, but in hardware.
- feelamee 1mo agousing which floating point standard? IEEE754 is totally deterministic
- shakna 1mo agoThe software standard is. GPU matrix calculations, are not. The hardware, has tiny shifts that rarely matter, except in high finance and... AI modeling.
- trollbridge 1mo agoWhat kind of nonsense is this? Matrix math is entirely deterministic.
- shakna 1mo ago> As the examples demonstrate, although rounding follows deterministic rules, non-associativity introduces nondeterminism, which is further amplified by the larger rounding error of BF16. This becomes particularly relevant in the parallel computations performed in GPUs during LLM inference. Ya might want to just read the paper...
- andai 1mo agoCould you give some examples?
- simonw 1mo agoCan you provide steps to reproduce so I can see one of these deterministic LLMs running myself? API based or local models.
- trollbridge 1mo agoI’d recommend starting with llamafile due to its simplicity and then move up to llamacpp and Unsloth.
- circuit10 1mo agoI think when people say non-deterministic what they mean is closer to chaotic, like https://en.wikipedia.org/wiki/Chaos_theory https://en.wikipedia.org/wiki/Chaos_theory as in very small changes in conditions can produce completely different output making predictions difficult