4 ms·
Nit: in practice, even at temperature 0, production LLM implementations have some non-determinism. One reason is because many floating point computations are te
by reasonableklout 1y ago
Nit: in practice, even at temperature 0, production LLM implementations have some non-determinism. One reason is because many floating point computations are technically non-commutative even when the mathematical operation is, and the order can vary if they are carried out in parallel by the GPU. For example, see: https://www.twosigma.com/articles/a-workaround-for-non-determinism-in-tensorflow/ https://www.twosigma.com/articles/a-workaround-for-non-deter...
- jkhdigital 1y agoI ran into this a bit while working on my PhD research that used LLMs for steganography. The output had to be deterministic to reverse the encoding, and it was—as long as you used the same hardware. Encoding a message on a PC and then trying to decode on a phone broke everything.