3 ms·
Yeah there needs to be some kind of acid test that will compare these products on equal footing and show the pitfalls.
by keyle 4y ago
Yeah there needs to be some kind of acid test that will compare these products on equal footing and show the pitfalls.
- sanxiyn 4y agoThis is very very difficult, but Tantivy tried: see https://github.com/quickwit-oss/search-benchmark-game https://github.com/quickwit-oss/search-benchmark-game
- jasfi 4y agoThat would be great. However if you wanted to benchmark relevance ranking, how would you do that?
- sanxiyn 4y agoYou need a dataset and an evaluation metric. The usual evaluation metric is NDCG(Normalized Discounted Cumulative Gain): https://en.wikipedia.org/wiki/NDCG https://en.wikipedia.org/wiki/NDCG An example dataset is BEIR(BEnchmarking Information Retrieval), published in NIPS 2021: https://github.com/beir-cellar/beir https://github.com/beir-cellar/beir
- DeathArrow 4y agoHere is a performance benchmark: https://db-benchmarks.com/test-hn/#manticore-search-columnar-storage-vs-elasticsearch https://db-benchmarks.com/test-hn/#manticore-search-columnar...