3 ms·
I'm pretty sure that the determinism issue is at the floating point math level, or even the hardware level. Just disabling batching and reducing the temperature
by danpalmer 6mo ago
I'm pretty sure that the determinism issue is at the floating point math level, or even the hardware level. Just disabling batching and reducing the temperature to 0 does not result in truly deterministic answers.
- orbital-decay 6mo agoFP math itself is deterministic on real hardware, if the order of operations stays the same. Output reproducibility is much less of a problem than it seems, see for example https://docs.vllm.ai/en/latest/usage/reproducibility/ https://docs.vllm.ai/en/latest/usage/reproducibility/
- nnevatie 6mo agoThe FP math is deterministic. However, the environments in which inference is run and specifically batching make current LLM services practically non-deterministic.