5 ms·
And even on open LLMs, GPU instability can cause non-determinism. For performance reasons, determinism is seldom guaranteed in LLMs in general.
by ijk 1y ago
And even on open LLMs, GPU instability can cause non-determinism. For performance reasons, determinism is seldom guaranteed in LLMs in general.
- rar00 1y agoyep, even with greedy sampling and fixed system state, numerical instability is sufficient to make output sequences diverge when processing the same exact input