4 ms·
Hamel wrote a whole lot more about the "LLM as a judge" pattern (where you use LLMs to evaluate the output of other LLMs) here: https://hamel.dev/blog/posts/llm
by simonw 1y ago
Hamel wrote a whole lot more about the "LLM as a judge" pattern (where you use LLMs to evaluate the output of other LLMs) here: https://hamel.dev/blog/posts/llm-judge/ https://hamel.dev/blog/posts/llm-judge/
- hamelsmu 1y agoAppreciate it, Simon! I have now edited my post to include links to "intro to evals" for those not familiar.
- padolsey 1y agoI really recommend people study the measurement frailties and prompting sensitivities of LLM judges before employing them. They're valuable, but should be used with complete understanding of the risks: https://www.cip.org/blog/llm-judges-are-unreliable https://www.cip.org/blog/llm-judges-are-unreliable