2 ms·
Don't LLMs create the same outputs based on the same inputs if the temperature is 0? Maybe I'm just misunderstanding.
by sureglymop 8mo ago
Don't LLMs create the same outputs based on the same inputs if the temperature is 0? Maybe I'm just misunderstanding.
- deleted 8mo ago[deleted]
- AlotOfReading 8mo agoUnfortunately not. Various implementation details like attention are usually non-deterministic. This is one of the better blog posts I'm aware of: https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/ https://thinkingmachines.ai/blog/defeating-nondeterminism-in...
- 8note 8mo agoi dont think theres anything that makes it essentiall that llms are non-deterministic though if you rewrote the math to be all fixed point precision on big ints, i think you would still get the useful LLM results? if somebody really wanted to make a compiler in an LLM, i dont think that nondetermism is problem id really imagine an llm compiler being a set of specs, dependency versions, and test definitions to use though, and you'd introduce essential nondetermism by changing a version number, even if the only change was the version name from "experimental" to "lts"
- AlotOfReading 8mo agoThey're not inherently non-deterministic, correct. And floating point is deterministic enough, as that blog post is demonstrating.
- h3lp 8mo agoWhen they run on the deterministic hardware, yes. When they run on some large, parallel, varying-unpredictable-load-dependent-latency hardware, no.