4 ms·
> Namely setting temperature to zero, and turning off all history That's not nearly enough, though. The multi-node/GPU inference and specifically batching (and
by nnevatie 6mo ago
> Namely setting temperature to zero, and turning off all history
That's not nearly enough, though. The multi-node/GPU inference and specifically batching (and ordering in batching) have non-deterministic consequences for the current LLM services.
- 2ndorderthought 6mo agoTrue but for small models it's pretty close. See my comment below about other cases leading to nondeterminism.