5 ms·

Wait. Is temperature=0 "pretty deterministic", or is it deterministic?

• LeoPanthera 4y ago
  It’s my understanding that 0 is completely deterministic, except for when the model is updated, which does happen occasionally.
  • valgaze 4y ago
    @haolez I just tested w/ davinci-003 w/ temperature to 0 Prompt: https://imgur.com/YtQ4fbf https://imgur.com/YtQ4fbf -- Reveal the question marks in an interesting way: The dog goes ????????????? -- With temperature == 0, it consistently ("pretty deterministically"?) generated "woof!" ex. The dog goes woof!
  • gwillen 4y ago
    Their docs say (somewhat recently updated, I think) that even 0 is not perfectly deterministic in all cases (though it's very close). Some people had previously observed this and speculated that it was some kind of floating point roundoff issue, when two outputs have almost identical scores.
    • Jensson 4y ago
      GPU's often do floating point computations in nondeterministic order for performance reasons, so you would get small differences even for exactly the same order. It is probably that.
• genmon 4y ago
  In the strict definition, it's deterministic: you get the same response for the same prompt, each time (given the exact same model). But the prompt is parameterised. The bulk of the prompt is requesting a list of guests and speakers to be extracted, and the episode synopsis is appended as the "parameter". And I've noticed that the variation of the parameter changes what the overall prompt returns... so it might start being less reliable at responding with valid JSON, for example. So it's instance-deterministic but, across a range of parameters, class-fuzzy, if that makes sense?
• acchow 4y ago
  On a related note, has anyone noticed the growing use of "pretty" as a hedge against ever being wrong? Also, "not", as in "not the fastest" or "not the hottest". It could be 46C/115F outside and you say "wow, today is unbearably hot", to which someone retorts "nah, it's fine. It's not the hottest day". That's pretty good hedging. You can make infinite technically correct statements this way without ever saying anything meaningful
  • messe 4y ago
    > Also, "not", as in "not the fastest" or "not the hottest". Mid twenties Hiberno-English speaker here and that's always been a fairly common form of hedging, and I've not noticed an increase.
    • igravious 4y ago
      fairly, lol
• tedsanders 4y ago
  Temperature=0 is not perfectly deterministic for the OpenAI API. It's mentioned briefly in the OpenAI text completion guide: https://platform.openai.com/docs/guides/completion/introduction https://platform.openai.com/docs/guides/completion/introduct... If you have two possible tokens with probability 40% and 30%, you'll always get the 40% token at T=0. But if you have two possible tokens at 40% and 39.99%, you may get the 39.99% token on occasion, even if at T=0. (Numbers illustrative.)
  • minkzilla 4y ago
    Where does the inherent randomness come from?
    • ChatGTP 4y ago
      It's injected into the model, you can deviate from no randomness to get "creative".
      • minkzilla 4y ago
        Isn't the temperature the "injected" randomness? The way I imagine it would be somewhere in the model you would do input += (Math.random - 0.5) * Coefficient * temperature so setting temperature to 0 would mean no randomness. On thinking further about why there is inherent randomness I believe it is from a lack of associativity in floating point operations. They obviously do A LOT of parallel floating point operations.
• maccaw 4y ago
  Well technically everything is deterministic.