3 ms·
Deciding to do things this way is a tradeoff. Writing data into a search index is maybe 10x more expensive than writing it into a log. But doing actual search
by fizx 10y ago
Deciding to do things this way is a tradeoff. Writing data into a search index is maybe 10x more expensive than writing it into a log. But doing actual search typically happens in something like log(n) time. Even for pathological queries, they're still far more efficient than scans.
From a cost perspective, this system is a bet that most data it writes is rarely or never read from.
- foota 10y agoThe articles also discusses that it makes more complicated queries much easier to process. For instance, allowing regular expressions, or special character aware searches.
- rakoo 10y agoIt is possible to index content and then retrieve it with (simple) regular expressions, see (https://swtch.com/~rsc/regexp/regexp4.html https://swtch.com/~rsc/regexp/regexp4.html). The trick is that each regular expressions can be seen as a tree where leaves are n-grams to be searched for, and nodes are AND and OR logical operations of n-grams and intermediate nodes. So, a regexp like /banana/ can be seen as a search for the trigrams "ban" and "ana" and "nan" and "ana", all in that order. A regexp like /ca[tp]/ can be seen as a search for "cat" or "cap". Once you have simple n-grams like that it's trivial to index content on the way in (note that what you index doesn't actually need to interpret content, all you need is to split every n characters) and use that index to restrict potential matches.
- Retric 10y agoThis quickly falls down for searches like: /a[0-9][0-9][0-9]k[0-9][0-9][0-9]c