4 ms·
what if you set top_p=1, temperature=0, and always run it on the same local hardware
by geor9e 1y ago
what if you set top_p=1, temperature=0, and always run it on the same local hardware
- daemonologist 1y agoMaybe if you run it on CPU. (Maybe on GPU if all batching is disabled, but I wouldn't bet on it.)
- mkarrmann 1y agoHorace He at Thinking Machines just dropped an awesome article describing exactly this: https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/ https://thinkingmachines.ai/blog/defeating-nondeterminism-in... TL;DR: assuming you've squashed all regular non-determinism (itself a tall ask), you either need to ensure you always batch requests deterministically, or ensure all kernels are "batch invariant" (which is absolutely not common practice to do).
- deleted 1y ago[deleted]
- mrheosuper 1y agocosmic wave will get you