4 ms·

Entropy is also added via a random seed. The model is only deterministic if you use the same random seed.

• HarHarVeryFunny 2y ago
  I think you're confusing training and inference. During training there are things like initialization, data shuffling and dropout that depend on random numbers. At inference time these don't apply.
  • jampekka 2y ago
    Decoding (sampling) uses (pseudo) random numbers. Otherwise same prompt would always give the same response. Computing entropy generally does not. See e.g. https://huggingface.co/blog/how-to-generate https://huggingface.co/blog/how-to-generate
    • HarHarVeryFunny 2y ago
      Sure - but that's not the output of the model itself, that's the process of (typically) randomly sampling from the output of the model.
      • throwaway314155 2y ago
        Right, sampling from a model, also known as *inference* (for LLM's). The inference here is perhaps less pure than what you refer to but you're talking to human beings; there's no need for heavy pedantry.