3 ms·

Are any of these stable? I mean when using temperature=0, do you get the same reply for the same prompt? I am using gpt-4-1106-preview quite a lot, but it is h

by mg 3y ago

Are any of these stable? I mean when using temperature=0, do you get the same reply for the same prompt? I am using gpt-4-1106-preview quite a lot, but it is hard to optimize prompts when you cannot build a test-suite of questions and correct replies against which you can test and improve the instruction prompt. Even when using temperature=0, gpt-4-1106-preview outputs different answers for the same prompt.

tomschwiha 3y ago
Wouldn't be for reproducability the usage of a seed be a better fit? [0] https://platform.openai.com/docs/api-reference/chat/create#chat-create-seed https://platform.openai.com/docs/api-reference/chat/create#c...
- mg 3y ago
  I just tried the same prompt twice, both with 'temperature': 0, 'seed': 1, And I got two different replies. The 'system_fingerprint' in the reply was the same in both of the json responses. So it seems that even when you get the same 'system_fingerprint' back, replies for the same prompt will not be the same.
phillipcarter 3y ago
> [...] but it is hard to optimize prompts when you cannot build a test-suite of questions and correct replies against which you can test and improve the instruction prompt. I think this is because your approach isn't right. This tech isn't really unit-testable in the same sense. In fact, for many use cases, you may want non-deterministic results by design. Instead, you probably need evaluations. The idea is that you're still building out "test" cases, but instead of expecting a specific result each time, you get a result that you can score through some means. Each test case produces a score, and you get a rollup score for the suite, and that's how you can track regressions over time. For example, in our use case, we produce structured JSON that has to match a spec, but we also want to have the contents of that valid-to-spec JSON object be "useful". So there's a function that defines "usefulness" based on some criteria that I've put together since I'm a domain expert. This is something I can evolve over time, using real-world inputs that produce bad or unsatisfying outputs as new evaluations for the evaluation suite. Fair warning though, it's not very easy to get started with, and there's not a whole lot of information about doing it well online.
- mg 3y ago
  This is what I do. I calculate a score over a sample of questions and replies. I'm not doing unit tests. Comparing the scores of two prompts will not give you a definitive answer which one is superior. But the prediction which one is superior would be better without the noise added by the randomness in the execution of the LLM.
  - phillipcarter 3y ago
    > Comparing the scores of two prompts will not give you a definitive answer which one is superior. Yes, but it can tell you which is likely to be superior, which is perhaps good enough? Offline evals are only a part of the equation though, which is why online evaluations are perhaps even more important. Good observability and a way to systematically measure "what is a good response" on production data is what ultimately gets us closer to real truth.
Xenoamorphous 3y ago
Aren't these models non-deterministic by nature?
- kragen 3y ago
  no, they run on computers. anything that happens in a computer can be made reproducible
  - mg 3y ago
    I'm not sure. Googling for "gpu non-determinism" and reading around the results seems to indicate that GPUs are often not deterministic. I don't know why, but it is easy to imagine a scenario where non-determinism makes sense. For example the computer could say "I am supposed to do this floating point division. Let's see ... my very exact FP-Division unit is pretty hot at the moment. 82°C. Let's pass this calculation to the slightly less exact FP-Division unit then".
    - kragen 3y ago
      cpus can be nondeterministic as well, generally because of race conditions between different cores, and as i understand it, that's what happens with gpus too, just more often because there are more cores i think the kind of scenario you're describing does not occur in commercially sold hardware, because it poses problems not only for customers but also for verification engineers at the design house. also, in the specific case of floating-point division, it would violate the ieee-754 standard. it might happen in an experimental academic project

by mg 3y ago

tomschwiha 3y ago

Wouldn't be for reproducability the usage of a seed be a better fit? [0] https://platform.openai.com/docs/api-reference/chat/create#chat-create-seed https://platform.openai.com/docs/api-reference/chat/create#c...

mg 3y ago
I just tried the same prompt twice, both with 'temperature': 0, 'seed': 1, And I got two different replies. The 'system_fingerprint' in the reply was the same in both of the json responses. So it seems that even when you get the same 'system_fingerprint' back, replies for the same prompt will not be the same.

phillipcarter 3y ago

> [...] but it is hard to optimize prompts when you cannot build a test-suite of questions and correct replies against which you can test and improve the instruction prompt. I think this is because your approach isn't right. This tech isn't really unit-testable in the same sense. In fact, for many use cases, you may want non-deterministic results by design. Instead, you probably need evaluations. The idea is that you're still building out "test" cases, but instead of expecting a specific result each time, you get a result that you can score through some means. Each test case produces a score, and you get a rollup score for the suite, and that's how you can track regressions over time. For example, in our use case, we produce structured JSON that has to match a spec, but we also want to have the contents of that valid-to-spec JSON object be "useful". So there's a function that defines "usefulness" based on some criteria that I've put together since I'm a domain expert. This is something I can evolve over time, using real-world inputs that produce bad or unsatisfying outputs as new evaluations for the evaluation suite. Fair warning though, it's not very easy to get started with, and there's not a whole lot of information about doing it well online.

mg 3y ago
This is what I do. I calculate a score over a sample of questions and replies. I'm not doing unit tests. Comparing the scores of two prompts will not give you a definitive answer which one is superior. But the prediction which one is superior would be better without the noise added by the randomness in the execution of the LLM.
- phillipcarter 3y ago
  > Comparing the scores of two prompts will not give you a definitive answer which one is superior. Yes, but it can tell you which is likely to be superior, which is perhaps good enough? Offline evals are only a part of the equation though, which is why online evaluations are perhaps even more important. Good observability and a way to systematically measure "what is a good response" on production data is what ultimately gets us closer to real truth.

Xenoamorphous 3y ago

Aren't these models non-deterministic by nature?

kragen 3y ago
no, they run on computers. anything that happens in a computer can be made reproducible
- mg 3y ago
  I'm not sure. Googling for "gpu non-determinism" and reading around the results seems to indicate that GPUs are often not deterministic. I don't know why, but it is easy to imagine a scenario where non-determinism makes sense. For example the computer could say "I am supposed to do this floating point division. Let's see ... my very exact FP-Division unit is pretty hot at the moment. 82°C. Let's pass this calculation to the slightly less exact FP-Division unit then".
  - kragen 3y ago
    cpus can be nondeterministic as well, generally because of race conditions between different cores, and as i understand it, that's what happens with gpus too, just more often because there are more cores i think the kind of scenario you're describing does not occur in commercially sold hardware, because it poses problems not only for customers but also for verification engineers at the design house. also, in the specific case of floating-point division, it would violate the ieee-754 standard. it might happen in an experimental academic project