3 ms·
They cannot be made entirely deterministic. You have cases where the next set of token choices have the same probability, and most people also are using cloud p
by lovich 2mo ago
They cannot be made entirely deterministic. You have cases where the next set of token choices have the same probability, and most people also are using cloud provider served models which can be served across different hardware implementations which can modify the math used in running the model.
can you get some arbitrary number of 9s for consistency? Yea. You can’t get 100% deterministic though.
- win311fwg 2mo agoThere is nothing about LLMs that prevent them from being 100% deterministic, unless you are really stretching the definition. There are implementation shortcuts often used in practice, just as is the case with compilers, that lead to non-determinism but there is no reason you have to rely on those implementation details. Not even if cloud providers do. LLMs can be run in environments under your control. Computers are deterministic. When we talk about non-determinism we're just talking about where there are hidden inputs, but the fundamentals of computing means that all hidden inputs can become visible if you want them to be (although possibly at the cost of things like performance), so, yes, you can reach 100% determinism just fine if you wish to. It's just not worth the tradeoffs in most cases. But if a deterministic solution solved a problem here it would be worth it. However, a deterministic LLM doesn't help here. LLMs don't "make mistakes" because they are typically non-deterministic. They would equally "make mistakes" when deterministic.