4 ms·
This is only ideally true. From the perspective of the user of a large closed LLM, this isn't quite right because of non-associativity, experiments, unversioned
by mgraczyk 1y ago
This is only ideally true. From the perspective of the user of a large closed LLM, this isn't quite right because of non-associativity, experiments, unversioned changes, etc.
It's best to assume that the relationship between input and output of an LLM is not deterministic, similar to something like using a Google search API.
- ijk 1y agoAnd even on open LLMs, GPU instability can cause non-determinism. For performance reasons, determinism is seldom guaranteed in LLMs in general.
- rar00 1y agoyep, even with greedy sampling and fixed system state, numerical instability is sufficient to make output sequences diverge when processing the same exact input