3 ms·
Explain this though. The code is deterministic, even if it relies on pseudo random number generation. It doesn't just happen, someone has to make a conscious de
by altcognito 9mo ago
Explain this though. The code is deterministic, even if it relies on pseudo random number generation. It doesn't just happen, someone has to make a conscious decision to force a different code path (or model) if the system is loaded.
- FL33TW00D 9mo agoIt takes a different code path for efficiency. e.g if (batch_size > 1024): kernel_x else: kernel_y
- chrisjj 9mo agoNot deterministic. https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/ https://thinkingmachines.ai/blog/defeating-nondeterminism-in...
- pertymcpert 9mo agoFloating point math isn't associative for operations that are associative in normal math.
- measurablefunc 9mo agoThat would just add up to statistical noise instead of 10% degradation over a week.
- kevin_thibedeau 9mo agoCatastrophic error accumulation can produce more profound effects than noise.
- measurablefunc 9mo agoJust to make sure I got this right. They serve millions of requests a day & somehow catastrophic error accumulation is what is causing the 10% degradation & no one at Anthropic is noticing it. Is that the theory?
- pertymcpert 9mo agoFYI something in that region happened last august/September. Some inference bug triggered worse performance on TPUs vs GPU.
- minimaltom 9mo agoIts not deterministic. Any individual floating point mul/add is deterministic, but in a GPU these are all happening in parallel and the accumulation is in the order they happen to complete. When you add A then B then C, you get a different answer than C then A then B, because floating point, approximation error, subnormals etc.
- bonoboTP 9mo agoIt can be made deterministic. It's not trivial and can slow it down a bit (not much) but there are environment variables you can set to make your GPU computations bitwise reproducible. I have done this in training models with Pytorch.
- minimaltom 9mo agoThere are settings to make it reproducible but they incur a non-negligible drop in performance. Unsurprising given they amount to explicit synchronization to make the order of operations deterministic.
- jmalicki 9mo agoFor all practical purposes any code reliant on the output of a PRNG is non-deterministic in all but the most pedantic senses... And if the LLM temperature isn't set to 0 LLMs are sampling from a distribution. If you're going to call a PRNG deterministic then the outcome of a complicated concurrent system with no guaranteed ordering is going to be deterministic too!
- bonoboTP 9mo agoHow is this related to overloading? The nondeterminism should not be a function of overloading. It should just time out or reply slower. It will only be dumber if it gets rerouted to a dumber, faster model eg quantized.
- gmueckl 9mo agoNo, this isn't right. There are totally legitimate use cases for PRNGs as sources of random number sequences following a certain probability distribution where freezing the seed and getting reproducibility is actually required.
- joquarky 9mo agoTemperature can't be literally zero, or it creates a divide by zero error. When people say zero, it is shorthand for “as deterministic as this system allows”, but it's still not completely deterministic.
- 9mo ago
- make3 9mo agoThere's a million algorithms to make LLM inference more efficient as a tradeoff for performance, like using a smaller model, using quantized models, using speculative decoding with a more permissive rejection threshold, etc etc