4 ms·
>To query the index, we are going to apply the same tokenizer and filters we used for indexing From developing a Lucene-based search engine years ago, I learne
by injb 6y ago
>To query the index, we are going to apply the same tokenizer and filters we used for indexing
From developing a Lucene-based search engine years ago, I learned a very useful trick for things like stemming and synonyms: don't use the same tokenizer/filters that you used for indexing. It's much more flexible if you instead expand the query terms, replacing a given term with a list of OR clauses with all the synonyms.
So during indexing, if the document has "fishing", then you just add that to the index.
Later, during querying, the user can search for "fish" with optional stemming enabled, and their query will get re-written into
"fish" OR "fishing" OR "fishy"
This way, the user has control at query-time, and you can manage your synonym/stemming database without having to rebuild your index.
Of course, I had to write a custom parser for this but it was well worth it imo.
- blueline 6y agohow do you populate/maintain the synonym/stemming database? is this something that “standard” datasets exist for? asking because this problem will fall on my plate at work sometimes in the next few months :-)
- injb 6y agoThere was a standard set of synonyms that everyone was using at the time (around 12 years ago) that came in the form of a Lucene index. I'm not sure if it's still around or where you'd find it. IIRC the stemming was done by an algorithm and not from a database...but I don't remember the exact details.
- pseudalopex 6y agoStemming uses algorithms or Hunspell dictionaries.[1] [1] https://cwiki.apache.org/confluence/display/solr/LanguageAnalysis#LanguageAnalysis-Stemming https://cwiki.apache.org/confluence/display/solr/LanguageAna...
- matlin 6y agoI like that approach especially since index is more true to the source text.
- dumb1224 6y agoGood old days with Lucene powered indices! So many memories. That query expansion trick would have saved me many hours of tweaking.
- aflag 6y agoWouldn't having a stemmed index and a non-stemmed one also allow the same usability? The user could then just select which index to use.
- inertiatic 6y agoNot a universal trick. Depending on how smart you end up getting and how big your queries are you might end up with an extremely large number of terms in your queries.
- amelius 6y agoNot by more than a small constant factor.
- inertiatic 6y agoThe number of terms times the number of expansions times the types of expansion times the target fields. In effect, a problem in many applications.