4 ms·

They are usually nondeterministic even at temperature 0 - due to things like parallelism and floating point rounding errors.

Gracana 1y ago
This is dependent on configuration, you can get repeatable results if you need them. I know at least llama.cpp and vllm v0 are deterministic for a given version and backend, and vllm v1 is deterministic if you disable multiprocessing.
PeterStuer 1y ago
floating point rounding errors are still deterministic. Parallelism dynamics can impact results, but those are not specific to LLM's.
- simonw 1y ago
  Here's something that isn't deterministic: a = 0.1, b = 0.2, c = 0.3 a * (b * c) = 0.006 (a * b) * c = 0.006000000000000001 If you are running these operations in parallel you can't guarantee which of those orders the operations will complete in. When you're running models on a GPU (or any other architecture that runs a whole bunch of matrix operations in parallel) you can't guarantee the order of the operations.
  - zelphirkalt 1y ago
    The order of completion doesn't necessarily influence the overall result of a parallelized computation. This depends on how the results are aggregated. For example for reducing floating point error in calculating a sum of floating point numbers, you could have a sorting step before calculating the sum and then start summing up from the lowest values to the higher ones. Then it doesn't matter at all which of the values is calculated first, because you need them all anyway, to sort them and once they are sorted, the result will always be the same, given same input values. So you can see, completion time is a completely orthogonal issue, or can be made one. And even libraries like tensorflow can be made to give reproducible results, when setting the corresponding seeds for the underlying libraries. Have done that myself, speaking from experience in a machine learning setting.

Gracana 1y ago

This is dependent on configuration, you can get repeatable results if you need them. I know at least llama.cpp and vllm v0 are deterministic for a given version and backend, and vllm v1 is deterministic if you disable multiprocessing.

PeterStuer 1y ago

floating point rounding errors are still deterministic. Parallelism dynamics can impact results, but those are not specific to LLM's.

simonw 1y ago
Here's something that isn't deterministic: a = 0.1, b = 0.2, c = 0.3 a * (b * c) = 0.006 (a * b) * c = 0.006000000000000001 If you are running these operations in parallel you can't guarantee which of those orders the operations will complete in. When you're running models on a GPU (or any other architecture that runs a whole bunch of matrix operations in parallel) you can't guarantee the order of the operations.
- zelphirkalt 1y ago
  The order of completion doesn't necessarily influence the overall result of a parallelized computation. This depends on how the results are aggregated. For example for reducing floating point error in calculating a sum of floating point numbers, you could have a sorting step before calculating the sum and then start summing up from the lowest values to the higher ones. Then it doesn't matter at all which of the values is calculated first, because you need them all anyway, to sort them and once they are sorted, the result will always be the same, given same input values. So you can see, completion time is a completely orthogonal issue, or can be made one. And even libraries like tensorflow can be made to give reproducible results, when setting the corresponding seeds for the underlying libraries. Have done that myself, speaking from experience in a machine learning setting.