4 ms·

The backend OpenAI APIs are not deterministic even with temperature 0, and they might upgrade the model weights/tuning behind your back too? (Not sure about the

The backend OpenAI APIs are not deterministic even with temperature 0, and they might upgrade the model weights/tuning behind your back too? (Not sure about the upgrade, they might just put out a new model id param...)

varunkmohan 4y ago
It's very strange that they don't guarantee determinism. We trained a model at Codeium that does have determinism at inference time - it's not super hard if you don't just use random seeds everywhere. This makes it a lot harder to rely on OpenAI's API if you are doing any sort of caching / evaluation workloads.
- xyzzy123 4y ago
  Not promising determinism gives them a lot more implementation wiggle room to update / change things. My guess is that promising it right now is not a priority. It would be a great "pro" or "enterprise" type feature tho.
- swyx 4y ago
  Pytorch/CUDA is apparently sometimes nondeterministic https://twitter.com/taliaringer/status/1511411984398487564?s=46&t=L75UJ1fe78VCiBPUK6PddA https://twitter.com/taliaringer/status/1511411984398487564?s...
  - varunkmohan 4y ago
    Interesting, I guess we implemented this op properly :) It is strange that cub does not guarantee determinism here - it's not too hard an op to implement in CUDA.
nasir 4y ago
normally you can choose the model version so that's not a concern.
ilaksh 4y ago
They seem pretty consistent with temperature 0. Maybe not 100% but very close.
- justsid 4y ago
  “Pretty consistent” and “deterministic” are not at all the same
- AgentME 4y ago
  It's close but not fully deterministic. I remember seeing a theory that if the system is considering multiple possible completions that have the same rounded score percentage, then its choice between them is nondeterministic.
  - teebs 4y ago
    It's likely because GPU calculations are non-deterministic and small differences in floating point numbers could lead to different outcomes (either in the way you described or somewhere deeper in the model)
    - janalsncm 4y ago
      > GPU calculations are non-deterministic Tensorflow is non-deterministic for some operations due to thread scheduling. PyTorch doesn’t have this issue.
      - coolness 4y ago
        Some of the underlying CuDNN algorithms have nondeterministic implementations, which applies to PyTorch as well. See https://pytorch.org/docs/stable/notes/randomness.html https://pytorch.org/docs/stable/notes/randomness.html

varunkmohan 4y ago

It's very strange that they don't guarantee determinism. We trained a model at Codeium that does have determinism at inference time - it's not super hard if you don't just use random seeds everywhere. This makes it a lot harder to rely on OpenAI's API if you are doing any sort of caching / evaluation workloads.

xyzzy123 4y ago
Not promising determinism gives them a lot more implementation wiggle room to update / change things. My guess is that promising it right now is not a priority. It would be a great "pro" or "enterprise" type feature tho.
swyx 4y ago
Pytorch/CUDA is apparently sometimes nondeterministic https://twitter.com/taliaringer/status/1511411984398487564?s=46&t=L75UJ1fe78VCiBPUK6PddA https://twitter.com/taliaringer/status/1511411984398487564?s...
- varunkmohan 4y ago
  Interesting, I guess we implemented this op properly :) It is strange that cub does not guarantee determinism here - it's not too hard an op to implement in CUDA.

nasir 4y ago

normally you can choose the model version so that's not a concern.

ilaksh 4y ago

They seem pretty consistent with temperature 0. Maybe not 100% but very close.

justsid 4y ago
“Pretty consistent” and “deterministic” are not at all the same
AgentME 4y ago
It's close but not fully deterministic. I remember seeing a theory that if the system is considering multiple possible completions that have the same rounded score percentage, then its choice between them is nondeterministic.
- teebs 4y ago
  It's likely because GPU calculations are non-deterministic and small differences in floating point numbers could lead to different outcomes (either in the way you described or somewhere deeper in the model)
  - janalsncm 4y ago
    > GPU calculations are non-deterministic Tensorflow is non-deterministic for some operations due to thread scheduling. PyTorch doesn’t have this issue.
    - coolness 4y ago
      Some of the underlying CuDNN algorithms have nondeterministic implementations, which applies to PyTorch as well. See https://pytorch.org/docs/stable/notes/randomness.html https://pytorch.org/docs/stable/notes/randomness.html