3 ms·

As I understood it, the "randomness" affecting what is selected at any temperature still comes from a PRNG or CSPRNG (or whatever RNG you want, maybe a hardware

As I understood it, the "randomness" affecting what is selected at any temperature still comes from a PRNG or CSPRNG (or whatever RNG you want, maybe a hardware one), and if you where to swap out that with something deterministic you'd get the same results every time (barring non-determinism in other parts of the OS/drivers/maybe even hardware). But theoretically, the output of every LLM is seed-driven (or could be if you wrote the software to isolate it) just like any computer software. It's just none of the software written (even llama.cpp AFAIK) chooses to support stable-seeding due to the changes in stuff like CPU/Vulkan/CUDA/Metal differences making it difficult to make consistent. They could though! Hopefully one day someone implements it into the mainstream LLM-engine software and it gets exposed in the APIs serving the models. It'd do a lot to show folks the "internals" of these models.

microtonal 3mo ago
Stable seeding is not enough. A lot of modern, fast compute kernels are nondeterministic. Floating point multiplication/addition is not strictly associative and e.g. reductions can combine results from different threads in different orders (e.g. through atomic ops). You can write kernels to be deterministic, but it is generally less efficient.
- vlovich123 3mo ago
  They are only non-deterministic when you’re doing batching and a kernel ends up running across a “random” set of token streams. If you’re only processing one user’s request, they’re very much deterministic.
toolslive 3mo ago
It's probably due to the fact that it's a cloud service. You have no guarantee that your next request will go to the same machine. So even with an identical seed, and temp 0 you might get different hardware and hence different accuracy/noise in the floating point operations.
- rightbyte 3mo ago
  How can there be noise in floating point operations? I could buy like completion order for parallized batches i.e. adding a+b+c is different from a+c+b etc.
  - deleted 3mo ago
    [deleted]
  - toolslive 3mo ago
    While the IEEE 754 standard ensures that individual basic operations are deterministic and strictly bounded, it does not guarantee that an entire program will yield bit-identical results on all CPUs. CPUs and their execution environments introduce subtle hardware variations, architecture choices, and compiler optimizations that break bit-level consistency. (same for GPU/TPU, ...)
    - throwaway173738 3mo ago
      It is, after all, a fundamentally voltage-based process, and the logical “no-man’s land” is chosen to limit the likelihood of a weak component producing faulty logic, but it’s impractical to run through the set of all possible starting states and to verify that after an unbounded number of clock steps the machine reaches a predictable end state on all of the devices being manufactured.
    - vlovich123 3mo ago
      Parent is correct - the math is very deterministic if you can guarantee it’s running repeatedly on the same machine and you’re not processing “random” requests in parallel. The compiler is irrelevant because once the code is generated it’s not getting recompiled and thus isn’t a source of non determinism (and generally if you don’t touch the math the compiler will consistently emit the same underlying machine code).
      - simiones 3mo ago
        This sub-thread was about cloud environments, where different requests may be served by different hardware. And it's in fact very likely that there will be a mix of different hardware from different vendors, in any particular LLM cloud for now.
nok22kon 3mo ago
that's incorrect in the presence of batching. it's tough work making it truly deterministic: https://x.com/FireworksAI_HQ/status/2069873437217276015 https://x.com/FireworksAI_HQ/status/2069873437217276015
- vidarh 3mo ago
  It's not that hard. What is hard is making it truly deterministic and retain high throughput.
gaflo 3mo ago
PRNG is deterministic.

microtonal 3mo ago

Stable seeding is not enough. A lot of modern, fast compute kernels are nondeterministic. Floating point multiplication/addition is not strictly associative and e.g. reductions can combine results from different threads in different orders (e.g. through atomic ops). You can write kernels to be deterministic, but it is generally less efficient.

vlovich123 3mo ago
They are only non-deterministic when you’re doing batching and a kernel ends up running across a “random” set of token streams. If you’re only processing one user’s request, they’re very much deterministic.

toolslive 3mo ago

It's probably due to the fact that it's a cloud service. You have no guarantee that your next request will go to the same machine. So even with an identical seed, and temp 0 you might get different hardware and hence different accuracy/noise in the floating point operations.

rightbyte 3mo ago
How can there be noise in floating point operations? I could buy like completion order for parallized batches i.e. adding a+b+c is different from a+c+b etc.
- deleted 3mo ago
  [deleted]
- toolslive 3mo ago
  While the IEEE 754 standard ensures that individual basic operations are deterministic and strictly bounded, it does not guarantee that an entire program will yield bit-identical results on all CPUs. CPUs and their execution environments introduce subtle hardware variations, architecture choices, and compiler optimizations that break bit-level consistency. (same for GPU/TPU, ...)
  - throwaway173738 3mo ago
    It is, after all, a fundamentally voltage-based process, and the logical “no-man’s land” is chosen to limit the likelihood of a weak component producing faulty logic, but it’s impractical to run through the set of all possible starting states and to verify that after an unbounded number of clock steps the machine reaches a predictable end state on all of the devices being manufactured.
  - vlovich123 3mo ago
    Parent is correct - the math is very deterministic if you can guarantee it’s running repeatedly on the same machine and you’re not processing “random” requests in parallel. The compiler is irrelevant because once the code is generated it’s not getting recompiled and thus isn’t a source of non determinism (and generally if you don’t touch the math the compiler will consistently emit the same underlying machine code).
    - simiones 3mo ago
      This sub-thread was about cloud environments, where different requests may be served by different hardware. And it's in fact very likely that there will be a mix of different hardware from different vendors, in any particular LLM cloud for now.

nok22kon 3mo ago

that's incorrect in the presence of batching. it's tough work making it truly deterministic: https://x.com/FireworksAI_HQ/status/2069873437217276015 https://x.com/FireworksAI_HQ/status/2069873437217276015

vidarh 3mo ago
It's not that hard. What is hard is making it truly deterministic and retain high throughput.

gaflo 3mo ago

PRNG is deterministic.