4 ms·
This is the gap that kills most AI features. Devs test with queries they already know the answer to. Users come in with vague questions using completely differe
by victorbuilds 11mo ago
This is the gap that kills most AI features. Devs test with queries they already know the answer to. Users come in with vague questions using completely different words. I learned to test by asking my kids to use my app - they phrase things in ways I would never predict.
- embedding-shape 11mo agoIronically, pitting a LLM (ideally a completely different model) up against what you're testing, letting it write human "out of the ordinary" queries to use as test cases tend to work well too, if you don't have kids you can use as a free workforce :)
- scosman 11mo agoI build a system to do exactly this: https://docs.kiln.tech/docs/evaluations/evaluate-rag-accuracy-q-and-a-evals https://docs.kiln.tech/docs/evaluations/evaluate-rag-accurac... Basically it: - iterates over your docs to find knowledge specific to the content - generates hundreds of pairs of [synthetic query, correct answer] - evaluates different RAG configurations for recall