4 ms·
There is a real need for this. I have to admit most of my testing right now is vibes based. Problem is, these LLM evaluation platforms get in-between me and my
by justanotheratom 2y ago
There is a real need for this. I have to admit most of my testing right now is vibes based. Problem is, these LLM evaluation platforms get in-between me and my LLM.
- Onawa 2y agoI use promptfoo for our internal evaluations. There are a ton of different "assertions" (tests) that you can write, including model-graded evaluations using rubrics. This is far from a solved problem, but there are options out there for systematic testing of LLMs.
- senko 2y agoProbably non-optimal long term, but you can easily whip up rudimentary tests with your LLM/lib of choice (see my other comment at https://news.ycombinator.com/item?id=42322794 https://news.ycombinator.com/item?id=42322794) without using any of the eval platforms.
- vinnymac 2y agoOffline evaluators such as laminar and (evalite wip) stay out of the way fortunately. Evals can be very useful when they do.