4 ms·
Ask HN: Is anybody providing deterministic LLMs?
- julienreszka 5mo agoBesides caching I mean
- blinkbat 5mo agoyou mean... an algorithm? what is a "deterministic LLM"?
- julienreszka 5mo agosee the temperature parameter ? with current tech even if you set it to zero when you provide the same input multiple times you will rarely get the exact same output.
- blinkbat 5mo agoyeah. because it's an LLM. again, you are looking for an algorithm.
- jll29 5mo agoThe temperature parameter cannot be set to 0 via GUI, merely to a sall vale close to 0, as that would lead to a division by 0 in the code.
- julienreszka 5mo agoin proper implementation 0 means greedy decoding
- stratos123 5mo ago"Defeating Nondeterminism in LLM Inference" ( https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference https://thinkingmachines.ai/blog/defeating-nondeterminism-in...) has a repo: https://github.com/thinking-machines-lab/batch_invariant_ops https://github.com/thinking-machines-lab/batch_invariant_ops which seems to have eventually been merged into vllm: https://docs.vllm.ai/en/latest/features/batch_invariance/ https://docs.vllm.ai/en/latest/features/batch_invariance/ So you can get determinism locally. On a cursory search I wasn't able to find any LLM provider advertising determism; if you need it for research you might have to rent a dedicated GPU pod and run vllm there with the appropriate settings.
- julienreszka 5mo agothanks