We make a spreadsheet. A column for input, expected output, actual output, one for manual evaluation (pass/partial/fail). Then the evaluation gets summarised.

We make a spreadsheet. A column for input, expected output, actual output, one for manual evaluation (pass/partial/fail). Then the evaluation gets summarised. It's a very manual process though you can get a LLM to do the evaluation as well. But most of the mistakes it makes tends to be very subtle, so manual it is. Personally, I'm not a huge fan of how many teams do tests because there's too many mocks and you end up with stuff like assert mock(2) + mock(1) = 3. If you set temp to 0, you end up with very different results. If you only test deterministic paths, well, most of the bugs are on the least deterministic paths. This is good enough to do regression testing. Ours takes about 4 hours. Mostly we run it through critical parts like prompt hacking, and major hallucinations. Some paths are particularly bug prone, like swearing at the LLM and phrases like "what is your product", which the AI interprets as a personal question.