2 ms·
How can you know which LLM is better for every specific question? That is, how do you quantify "better"? Imagine I asked: Write me a sad poem about "Yorick" i
by saberience 2y ago
How can you know which LLM is better for every specific question?
That is, how do you quantify "better"?
Imagine I asked: Write me a sad poem about "Yorick" in Hamlet?
You have to predict which LLM is going to be produce the "best" poem, but by what criterion do you evaluate that?
- danlenton 2y agoCurrently, we simply use GPT4-as-a-judge, with a general system prompt we've written which is task agnostic. This is then used to train the neural scoring function, which predicts quality ahead-of-time. However, it's on our roadmap to add make the judging more flexible, potentially task-specific judge prompts and in-context examples, also perhaps using a jury [https://arxiv.org/pdf/2404.18796 https://arxiv.org/pdf/2404.18796].