3 ms·
The size of the index is far more dependent on the search text than the number of searched items. I believe Google documented some of this in its early days, n
by dredmorbius 3y ago
The size of the index is far more dependent on the search text than the number of searched items.
I believe Google documented some of this in its early days, noting that a search index returns the relevant metadata matching a specific query. The query space itself is largely based on both raw keywords and tuples (2- or 3-word ngrams if memory serves, though I'm hazy on this), the latter meeting some minimum frequency requirement. Longer search terms can be constructed from shorter ngrams.
A typical advanced native-tongue English vocabulary is about 40,000 words. An expansive dictionary might contain fewer than 250,000 words, including obsolete ones.
Mapping a vocabulary to works citing those words is relatively straightforward. Ngrams experience combinatorial expansion, but are still a reasonably constrained space. And we now have well over a quarter-century's experience indexing written content at Web scale.
A laptop could probably make a decent cut at providing a useful index of many millions of books, though you'd probably want a somewhat larger system for a more comprehensive index, in particular to rank-index the search space, which is probably the more considerable challenge.
- carlosjobim 3y agoThat was a great reply, and made me understand how these things work a lot better. Thank you!
- dredmorbius 3y agoThe key concept here is the inverted index: <https://en.wikipedia.org/wiki/Inverted_index https://en.wikipedia.org/wiki/Inverted_index>