3 ms·

They can be deterministic. We did this at Groq, if you sent a request with exactly the same input token, seed and temperature value you would get precisely the

by jpgvm 2mo ago

They can be deterministic. We did this at Groq, if you sent a request with exactly the same input token, seed and temperature value you would get precisely the same result every time. This is harder to do on other architectures that themselves aren't fully deterministic though.

Iolaum 2mo ago
Was the temperature 0? Cause unless I don't understand it right, any non-zero temperature implies probabilistic next token prediction. You did mention, seed, which I haven't seen available anywhere else (in my limited experience) and it can explain a reproducible inference result, but I feel like this matches the "letter of the law" meaning of deterministic rather than the "spirit of the law".
- olmo23 2mo ago
  If you send the seed along, you can have a non-zero temperature: when it's time to select tokens, a PRNG will be seeded and used for the selection.
- bnjemian 2mo ago
  Using a PRNG and a seed, you can have paeudo-randomness and determinism. The pseudo-randomness is the key piece, it’s random to the extent that it cannot be distinguished from true randomness in polynomial time classically.
- bruce343434 2mo ago
  Perhaps they just used the exact same random seed everywhere?
niea_11 2mo ago
Do you mean that you get the exact same result (as word for word) because from Groq's documentation, it says that determinism is best effort, and also it's not guaranteed across model versions : https://console.groq.com/docs/prompting?_sm_vck=tMW2S7k17245QwqZwFDkjTk6fHwHkPJsZ06NtVHVVJNS7r7672SL#deterministic-outputs-with-seed https://console.groq.com/docs/prompting?_sm_vck=tMW2S7k17245... My understanding is that it's best effort because even if you control the seed/temperature, the model version and hardware specs you still have floating point math that will introduce rounding differences an article explaining this in detail : https://news.ycombinator.com/item?id=45200925 https://news.ycombinator.com/item?id=45200925
- jpgvm 2mo ago
  > harder to do on other architectures that themselves aren't fully deterministic though The TLDR here is that Groq LPUs are completely deterministic hardware, including for floating point. Disclaimer: Formerly of Groq, now of NVIDIA, still working on LPUs.

by jpgvm 2mo ago

Iolaum 2mo ago

Was the temperature 0? Cause unless I don't understand it right, any non-zero temperature implies probabilistic next token prediction. You did mention, seed, which I haven't seen available anywhere else (in my limited experience) and it can explain a reproducible inference result, but I feel like this matches the "letter of the law" meaning of deterministic rather than the "spirit of the law".

olmo23 2mo ago
If you send the seed along, you can have a non-zero temperature: when it's time to select tokens, a PRNG will be seeded and used for the selection.
bnjemian 2mo ago
Using a PRNG and a seed, you can have paeudo-randomness and determinism. The pseudo-randomness is the key piece, it’s random to the extent that it cannot be distinguished from true randomness in polynomial time classically.
bruce343434 2mo ago
Perhaps they just used the exact same random seed everywhere?

niea_11 2mo ago

Do you mean that you get the exact same result (as word for word) because from Groq's documentation, it says that determinism is best effort, and also it's not guaranteed across model versions : https://console.groq.com/docs/prompting?_sm_vck=tMW2S7k17245QwqZwFDkjTk6fHwHkPJsZ06NtVHVVJNS7r7672SL#deterministic-outputs-with-seed https://console.groq.com/docs/prompting?_sm_vck=tMW2S7k17245... My understanding is that it's best effort because even if you control the seed/temperature, the model version and hardware specs you still have floating point math that will introduce rounding differences an article explaining this in detail : https://news.ycombinator.com/item?id=45200925 https://news.ycombinator.com/item?id=45200925

jpgvm 2mo ago
> harder to do on other architectures that themselves aren't fully deterministic though The TLDR here is that Groq LPUs are completely deterministic hardware, including for floating point. Disclaimer: Formerly of Groq, now of NVIDIA, still working on LPUs.