10 ms·
LLMs aren’t deterministic even with a seed.
by fzzzy 1y ago
LLMs aren’t deterministic even with a seed.
- jsheard 1y agoDoesn't that depend on the implementation? There's a trade-off between performance and determinism for sure, but if determinism is what you want then it should be possible.
- jb1991 1y agoIf you fix random seeds, disable dropout, and configure deterministic kernels, you can get reproducible outputs locally. But you still have to control for GPU non-determinism, parallelism, and even library version differences. Some frameworks (like PyTorch) have flags (torch.use_deterministic_algorithms(True)) to enforce this.
- jb1991 1y agoThis. I’m still amazed how many people don’t understand how this technology actually works. Even those you would think would have a vested interest in understanding it.
- geor9e 1y agowhat if you set top_p=1, temperature=0, and always run it on the same local hardware
- daemonologist 1y agoMaybe if you run it on CPU. (Maybe on GPU if all batching is disabled, but I wouldn't bet on it.)
- mkarrmann 1y agoHorace He at Thinking Machines just dropped an awesome article describing exactly this: https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/ https://thinkingmachines.ai/blog/defeating-nondeterminism-in... TL;DR: assuming you've squashed all regular non-determinism (itself a tall ask), you either need to ensure you always batch requests deterministically, or ensure all kernels are "batch invariant" (which is absolutely not common practice to do).
- deleted 1y ago[deleted]
- mrheosuper 1y agocosmic wave will get you
- worble 1y agoYes, that's the joke