4 ms·
do you somehow control how non-trivial the queries are? The LLM generates them, right? what if every engine returns garbage, or on the other hand, handles them
by terno 1mo ago
do you somehow control how non-trivial the queries are? The LLM generates them, right?
what if every engine returns garbage, or on the other hand, handles them too well?
building a benchmark like this in a genuinely fair way seems extremely hard to me. I’m very curious about the details, of course within what you can share.
- matt4711 1mo agoIt is hard to be fair I agree. We tried to be open about what we do here: github.com/keenableai/needle The queries from what I can tell are not trivial. The actual github repo of the benchmark has a judgement/query browser where you can inspect the different query streams: https://keenableai.github.io/needle/ https://keenableai.github.io/needle/