3 ms·
> I'm curious how people evaluate them quickly. Are there search benchmarks to be found somewhere? There must be. If you want to write a search engine, you n
by amelius 5y ago
> I'm curious how people evaluate them quickly.
Are there search benchmarks to be found somewhere?
There must be. If you want to write a search engine, you need a way to validate the results.
- marginalia_nu 5y agoThere are benchmarks within the adjacent field of information retrieval, but in general it's hard to properly validate a search engine because real data is so noisy and misbehaved, and sample data is so different from real data.
- kreeben 5y agoSure, the problem of information retrieval is not exactly that of web search but they're pretty close. So, from such a knowledgeable person such as yourself, when it comes to this topic, could you remind us, what are some of those benchmarks?
- marginalia_nu 5y agohttps://en.m.wikipedia.org/wiki/Precision_and_recall https://en.m.wikipedia.org/wiki/Precision_and_recall For some standard corpus.
- kreeben 5y ago>> Precision and recall The phrase I was looking for. Thx a bunch! Gonna marginalia that now.
- marginalia_nu 5y agoHaha, ironically it lacks both precision and recall for that topic.
- jll29 5y agoThe "Web track" task at the annual US NIST TREC conference ( https://trec.nist.gov/ https://trec.nist.gov/ ) is an open innovation benchmark that everyone can contribute; participants get a set of queries that they have to run on exactly the same corpus. Then they return the top-k results to a team that evaluates them. Here is an example (2014) Web track paper of the 23th TREC: https://trec.nist.gov/pubs/trec23/papers/overview-web.pdf https://trec.nist.gov/pubs/trec23/papers/overview-web.pdf (TREC has a plentitude of difference benchmark tasks and you can submit your own: https://trec.nist.gov/pubs/trec29/trec2020.html https://trec.nist.gov/pubs/trec29/trec2020.html - recent TREC 2020 papers)
- vivekraghunatha 5y ago(Founder of Neeva) Eval for search engines is about as hard as ranking for search engines. You need rating templates, raters, querysets, tooling and lots of time and patience. There are a number of vendors who can help you with the raters part, but the rest is still painstaking work you have to do yourself. Email me if you need help; we are happy to share findings.