3 ms·
How do you create repeatable tests in a non-deterministic system? Every time you send the same prompt you get a different answer.
by cloudking 12d ago
How do you create repeatable tests in a non-deterministic system? Every time you send the same prompt you get a different answer.
- ssivark 12d agoThe actual tokens might be non-deterministic, but you could look for proxy measures that are supposed to be invariant. Eg. correctness/performance on benchmarks, "thinking level" on complex problems, etc
- CharlesW 12d agoThis is a good overview of how this is done: https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents https://www.anthropic.com/engineering/demystifying-evals-for...
- 6gvONxR4sf7o 12d agoThat's like the entire field of statistics.