3 ms·
I think you're confusing training and inference. During training there are things like initialization, data shuffling and dropout that depend on random numbers.
by HarHarVeryFunny 2y ago
I think you're confusing training and inference. During training there are things like initialization, data shuffling and dropout that depend on random numbers. At inference time these don't apply.
- jampekka 2y agoDecoding (sampling) uses (pseudo) random numbers. Otherwise same prompt would always give the same response. Computing entropy generally does not. See e.g. https://huggingface.co/blog/how-to-generate https://huggingface.co/blog/how-to-generate
- HarHarVeryFunny 2y agoSure - but that's not the output of the model itself, that's the process of (typically) randomly sampling from the output of the model.
- throwaway314155 2y agoRight, sampling from a model, also known as *inference* (for LLM's). The inference here is perhaps less pure than what you refer to but you're talking to human beings; there's no need for heavy pedantry.