17 ms·
I'm sure there's tonnes of research and prior art on this subject, but it's an interesting inquiry. off the top of my head, there's two meaning - performance b
by chii 5y ago
I'm sure there's tonnes of research and prior art on this subject, but it's an interesting inquiry.
off the top of my head, there's two meaning - performance benchmarking (how fast the search results comes back), and accuracy/fit-for-purposeness benchmarking (how good it is at finding something the user intends).
Performance is easy. It's the accuracy/fit-for-purposeness that would be an interesting benchmark.
I wonder if you have to use an empirical measurement for accuracy - that is, give a random sample of people a target piece of code (or file) to find, and see how long or how many queries it takes to find it.
- toast0 5y agoFor quality, you really do need to do human qualititative measures to get a full measure, with all of the fun that involves. However, you can do things like generate search terms from your top N documents through some method, and then do the queries and confirm the document you generated the term from shows up in the top M results. This can be circular though if you're not careful; the top N documents may not include important documents that nobody could find.