4 ms·
The algorithms are definitely not deterministic. That said I agree with your general point that experimenting on LLMs as if they're black boxes with unknown int
by kerkeslager 1y ago
The algorithms are definitely not deterministic. That said I agree with your general point that experimenting on LLMs as if they're black boxes with unknown internals is silly.
EDIT: I'm seeing another poster saying "Deterministic with a random seed?" That's a good point--all the non-determinism comes from the seed, which isn't particularly critical to the algorithm. One could easily make an LLM deterministic by simply always using the same seed.
- dist-epoch 1y ago> all the non-determinism comes from the seed not fully true, when using floating point the order of operations matters, and it can vary slightly due to parallelism. I've seen LLMs return different outputs with the same seed.
- onionisafruit 1y agoThat’s an interesting observation. Usually we try to control that, but with LLMs the non-determinism is fine. It seems like that would make it hard to unit test LLM code, but they seem to be managing.
- kerkeslager 1y agoOh, that's really interesting. Hadn't thought of that.