3 ms·
Extremely pedantic, but is "non-deterministic" really the right language? The same input will always produce the same output, provided you haven't intentionally
by the8thbit 2y ago
Extremely pedantic, but is "non-deterministic" really the right language? The same input will always produce the same output, provided you haven't intentionally configured the system to use the model non-deterministically. It seems like the right way to describe it is as a chaotic deterministic system. The same input will always produce the same output, but small shifts in the input or weights can result in dramatic and difficult to predict changes in outputs.
- davedx 2y agoLlms are indeed non deterministic
- visarga 2y ago> The same input will always produce the same output Not guaranteed even with the same seed. If you don't perform all operations in exactly the same order, even a simple float32 sum, if batched differently, will result in different final value. This depends on the load factor and how resources are allocated.
- simonw 2y agoYeah, the fact that floating point multiplication isn't associative is a real pain for producing deterministic outputs - especially when you're running massively parallel computations on GPUs (or multiple GPUs) making the order of operations even less predictable.
- taneq 2y agoThis doesn’t mean LLMs are inherently non-deterministic, just that current common implementations are non-deterministic.