4 ms·
> Comparing the scores of two prompts will not give you a definitive answer which one is superior. Yes, but it can tell you which is likely to be superior, whi
by phillipcarter 3y ago
> Comparing the scores of two prompts will not give you a definitive answer which one is superior.
Yes, but it can tell you which is likely to be superior, which is perhaps good enough?
Offline evals are only a part of the equation though, which is why online evaluations are perhaps even more important. Good observability and a way to systematically measure "what is a good response" on production data is what ultimately gets us closer to real truth.