3 ms·
I don't think those race conditions are rare. None of the big hosted LLMs provide a temperature=0 plus fixed seed feature which they guarantee won't return diff
by simonw 1y ago
I don't think those race conditions are rare. None of the big hosted LLMs provide a temperature=0 plus fixed seed feature which they guarantee won't return different results, despite clear demand for that from developers.
- xnx 1y agoFair. I dislike "non-deterministic" as a blanket llm descriptor for all llms since it implies some type of magic or quantum effect.
- basch 1y agoI agree its phrased poorly. Better said would be: LLM's are designed to act as if they were non-deterministic.
- deleted 1y ago[deleted]
- dekhn 1y agoI see LLM inference as sampling from a distribution. Multiple details go into that sampling - everything from parameters like temperature to numerical imprecision to batch mixing effects as well as the next-token-selection approach (always pick max, sample from the posterior distribution, etc). But ultimately, if it was truly important to get stable outputs, everything I listed above can be engineered (temp=0, very good numerical control, not batching, and always picking the max probability next token). dekhn from a decade ago cared a lot about stable outputs. dekhn today thinks sampling from a distribution is a far more practical approach for nearly all use cases. I could see it mattering when the false negative rate of a medical diagnostic exceeded a reasonable threshold.
- tanewishly 1y agoErrr... that word implies some type of non-deterministic effect. Like using a randomizer without specifying the seed (ie. sampling from a distribution). I mean, stuff like NFAs (non-deterministic finite automata) isn't magic.
- EdiX 1y agoInteresting, but in general it does not imply that. For example: https://en.wikipedia.org/wiki/Nondeterministic_finite_automaton https://en.wikipedia.org/wiki/Nondeterministic_finite_automa...
- toolslive 1y agoI, naively (an uninformed guess), considered the non-determinism (multiple results possible, even with temperature=0 and fixed seed) stemming from floating point rounding errors propagating through the calculations. How wrong am I ?
- bmicraft 1y agoThey're gonna round the same each time you're running it on the same hardware.
- toolslive 1y agobut they're not: they are scheduled on some infrastructure in the cloud. So the code version might be slightly different, the compiler (settings) might differ, and the actual hardware might differ.
- williamdclt 1y agoAlso uninformed but I can't see how that would be true, floating point rounding errors are entirely deterministic
- saagarjha 1y agoNot if your scheduler causes accumulation in a different order.
- williamdclt 1y agoAre you talking about a DAG of FP calculations, where parallel steps might finish in different order across different executions? That's getting out of my area of knowledge, but I'd believe it's possible
- saagarjha 1y agoWell a very simple example would be if you run a parallel reduce using atomics the result will depend on which workers acquire the accumulator first.
- diggan 1y ago> despite clear demand for that from developers Theorizing about why that is: Could it be possible they can't do deterministic inference and batching at the same time, so the reason we see them avoiding that is because that'd require them to stop batching which would shoot up costs?