4 ms·

One thing which is missing from this implementation is the use of lemmas. Rather than treating words like "house", "houses", "housing" all as separate terms, th

One thing which is missing from this implementation is the use of lemmas. Rather than treating words like "house", "houses", "housing" all as separate terms, they all get reduced to the stem "house". http://lemmatise.ijs.si/ http://lemmatise.ijs.si/ is a good resource for this.

mmavnn 15y ago
He covers this in some detail, under the term stemming (which I've heard more often in this context). He even makes a library recommendation.
- lignuist 15y ago
  Lemmatizing and stemming are similar, but not the same. Example for possible results: Lemmatizing: indices -> index Stemming: indices -> indi http://en.wikipedia.org/wiki/Stemming#Lemmatisation_algorithms http://en.wikipedia.org/wiki/Stemming#Lemmatisation_algorith...
  - bad_user 15y ago
    Btw, in case you want a good lemmatisation implementation, there's one in NLTK that uses the WordNet database.
  - nl 15y ago
    It's probably worth noting that in this context stemming is probably most appropriate. Lemmatizing is unlikely to give better results and in most implementations you suffer a significant performance hit.