3 ms·

how do you populate/maintain the synonym/stemming database? is this something that “standard” datasets exist for? asking because this problem will fall on my pl

how do you populate/maintain the synonym/stemming database? is this something that “standard” datasets exist for? asking because this problem will fall on my plate at work sometimes in the next few months :-)

injb 6y ago
There was a standard set of synonyms that everyone was using at the time (around 12 years ago) that came in the form of a Lucene index. I'm not sure if it's still around or where you'd find it. IIRC the stemming was done by an algorithm and not from a database...but I don't remember the exact details.
- vaidhy 6y ago
  when I was building this in circa 2006, I used porter stemmer. I do not know if that is still state of the art, but it worked really well for my purpose.
  - sanxiyn 6y ago
    These days we use Snowball stemmer. OP also used it.
pseudalopex 6y ago
Stemming uses algorithms or Hunspell dictionaries.[1] [1] https://cwiki.apache.org/confluence/display/solr/LanguageAnalysis#LanguageAnalysis-Stemming https://cwiki.apache.org/confluence/display/solr/LanguageAna...