3 ms·
Aren't LLMs non-deterministic by choice? That they regularly use random seeds, sampling and batching but that these sources of non-determinism can be removed, f
by eikenberry 1y ago
Aren't LLMs non-deterministic by choice? That they regularly use random seeds, sampling and batching but that these sources of non-determinism can be removed, for instance, by run an LLM locally where you can control these parameters.
- simonw 1y agoUntil very recently that proved surprisingly difficult to achieve. Here's the paper that changed that: https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/ https://thinkingmachines.ai/blog/defeating-nondeterminism-in...