4 ms·

But if T=0 and you use the exact same input (not a single word or position changes) do you get the same output? Reading your response it implies that the random

• version_five 3y ago
  As a sibling comment mentioned, threading on a gpu is not automatically deterministic so you could randomness from there, although I can't think of anything in the forward pass of a normal LLM they would depend on execution order. So yes, you should get the same, it's basically just matrix multiplication. There may be some implementation details I don't know about that would add other variability though. Look at this minimal implementation (Karpathy's) of LLaMA, the only randomness is in the "sample" function that comes in at non-zero temperature, otherwise its easy to see everything is deterministic: https://github.com/karpathy/llama2.c/blob/master/run.c https://github.com/karpathy/llama2.c/blob/master/run.c Otoh, with MoE like GPT-4 has, it can still vary at zero temperature.