3 ms·

Thanks for sharing this, this is awesome! I noticed on the evaluations, you're looking at the structure of the responses (and I agree this is important.) But h

Thanks for sharing this, this is awesome! I noticed on the evaluations, you're looking at the structure of the responses (and I agree this is important.) But how do I check the factual content of the responses automatically? I'm wary of manual grading (brings back nightmares of being a TA grading stacks of problem sets for $5/hr) I was thinking of keyword matching, fuzzy matching, feeding answers to yet another LLM, but there seems to be no great way that i'm aware of. Any suggestions on tooling here?

typpo 3y ago
The library supports the model-graded factuality prompt used by OpenAI in their own evals. So, you can do automatic grading if you wish (using GPT 4 by default, or your preferred LLM). Example here: https://promptfoo.dev/docs/guides/factuality-eval https://promptfoo.dev/docs/guides/factuality-eval
- westurner 3y ago
  OpenAI/evals > Building an eval: https://github.com/openai/evals/blob/main/docs/build-eval.md https://github.com/openai/evals/blob/main/docs/build-eval.md "Robustness of Model-Graded Evaluations and Automated Interpretability" (2023) https://www.lesswrong.com/posts/ZbjyCuqpwCMMND4fv/robustness-of-model-graded-evaluations-and-automated https://www.lesswrong.com/posts/ZbjyCuqpwCMMND4fv/robustness... : > The results inspire future work and should caution against unqualified trust in evaluations and automated interpretability. From https://news.ycombinator.com/item?id=37451534 https://news.ycombinator.com/item?id=37451534 : add'l benchmarks: TheoremQA, Legalbench