3 ms·

> To make the most of phrase matching, stop words need to go. Perhaps I am misunderstanding; does this mean occurrences of stop words like "the" are stored now

by gary_0 2y ago

> To make the most of phrase matching, stop words need to go. Perhaps I am misunderstanding; does this mean occurrences of stop words like "the" are stored now instead of ignored? That seems like it would add a lot of bloat. Are there any optimizations in place? Just a shot-in-the-dark suggestion, but if you are storing some bits with each keyword occurrence, can you add a few more bits to store whether the term is adjacent to a common stop word? So maybe if you have to=0 or=1, "to be or not to be" would be able to match the data `0be 1not 0be`, where only "be" and "not" are actual keywords. But the extra metadata bits can be ignored, so pages containing "The Clash" will match both the literal query (via the "the" bit), and just "clash" (without the "the" bit).

marginalia_nu 2y ago
It's not as bad as you might think, we're speaking dozens of GB across the entire index. I don't think stopwords as an optimization makes sense when you go beyond BM25. The search engine behaves worse and adding a bunch of optimizations makes an already incrediby complex piece of software more so. So overall I don't think the juice is worth the squeeze.
ValleZ 2y ago
Removing stop words is usually a bad advice which is beneficial only in a limited set of circumstances. Google keeps all the “the”: https://www.google.com/search?q=the https://www.google.com/search?q=the
- efilife 2y ago
  I don't think it's as simple for them as just keeping the thes. It's probably very context dependent
heikkilevanto 2y ago
One of the problems with stop words is that they vary between languages. "The" is a good candidate in English, but in Danish it just means "tea", which should be a valid search term. And even in English, what looks like a serious stop word, can be an integral part of the phrase. "How to use The in English".
joking 2y ago
in reality, terms are stored with the position, so if you use stopwords to be or not to be should be indexed as 2-be 6-be and a phrase match should match exactly. the problem would be that it would match also "be not to or be", as the distance between the 2 "be" is also 3. Long time ago it was necessary, but nowadays you loose more that what you gain using stopwords.

by gary_0 2y ago

marginalia_nu 2y ago

It's not as bad as you might think, we're speaking dozens of GB across the entire index. I don't think stopwords as an optimization makes sense when you go beyond BM25. The search engine behaves worse and adding a bunch of optimizations makes an already incrediby complex piece of software more so. So overall I don't think the juice is worth the squeeze.

ValleZ 2y ago

Removing stop words is usually a bad advice which is beneficial only in a limited set of circumstances. Google keeps all the “the”: https://www.google.com/search?q=the https://www.google.com/search?q=the

efilife 2y ago
I don't think it's as simple for them as just keeping the thes. It's probably very context dependent

heikkilevanto 2y ago

One of the problems with stop words is that they vary between languages. "The" is a good candidate in English, but in Danish it just means "tea", which should be a valid search term. And even in English, what looks like a serious stop word, can be an integral part of the phrase. "How to use The in English".

joking 2y ago

in reality, terms are stored with the position, so if you use stopwords to be or not to be should be indexed as 2-be 6-be and a phrase match should match exactly. the problem would be that it would match also "be not to or be", as the distance between the 2 "be" is also 3. Long time ago it was necessary, but nowadays you loose more that what you gain using stopwords.