6 ms·
You get really good performance on not much hardware. Can you share some technical details? - file formats, particularly the postings - query evaluation strat
by ccleve 10y ago
You get really good performance on not much hardware. Can you share some technical details?
- file formats, particularly the postings
- query evaluation strategy
- update strategy
I poked around in the source code a bit, but couldn't find these things.
- deusu 10y agoFile formats will be documented when I publish the data-files in a few weeks. What do you mean with postings? The main index is split into 32 shards (there is also an additional news-index which is updated about every 5-10 minutes). Each shard is updated and queried seperately. The query actually runs 2/3 on a Windows server and 1/3 on a Linux server. The latter in Docker containers. I want to move everything to Linux over time. Query has two phases. First only a rough - but fast - ranking is done. Then the top results of all shards are combined and completely re-ranked. This is basically a meta search engine hidden within. First query phase is in src/searchservernew.dpr, and the second phase is in src/cgi/PostProcess.pas.
- ccleve 10y agoThank you. "Postings" is another word for the format of the doc ids and related information in the inverted file. A google for "inverted index postings" will turn up a bunch of references.