4 ms·
If I understand, having only 4096 bytes of data per term causes multiple terms in the same query to intersect to little or no results. The purpose seems to cut
by davidebaldini 3y ago
If I understand, having only 4096 bytes of data per term causes multiple terms in the same query to intersect to little or no results. The purpose seems to cut cost in compromise of completeness.
- marginalia_nu 3y agoYeah. That seems like a design decision that will scale poorly. For reference, even in my dinky 100M index I have individual terms with several gigabytes of associated document references. In general hash map table index designs don't tend to be very efficient. If you use a skip list or something similar, you can calculate the intersection between sets in sublinear time.
- daoudc 3y agoWe actually just take the union and then re-rank. Because the lists are all small, this is cheap.
- marginalia_nu 3y agoPoint is, with a skip list (or similar), the lists don't need to be small. You can intersect data sets that are enormous very quickly using this algo[1] where a single linear read of both lists is the worst case scenario. [1] https://nlp.stanford.edu/IR-book/html/htmledition/faster-postings-list-intersection-via-skip-pointers-1.html https://nlp.stanford.edu/IR-book/html/htmledition/faster-pos...
- daoudc 3y agoYes, you're correct on the purpose. We mitigate it a little by also indexing on bigrams.