3 ms·
When you lemmatize, what are you doing exactly? Are you merely reducing words to a more common form, thereby reducing IDF? For example, are you reducing "walkin
by ccleve 12y ago
When you lemmatize, what are you doing exactly? Are you merely reducing words to a more common form, thereby reducing IDF? For example, are you reducing "walking" or "walked" to "walk", and then using the IDF of walk?
- bane 12y agoYes, exactly. related is the idea of "stemming" which uses an algorithm to try to reduce inflection, to find a common form of a word that various versions come from. Porters algorithm is a well known stemming algorithm. However, sometimes you end up with weird "non-inflected" tokens at the end. (e.g. 'enhancement' might become 'enhanc') However, lemmatization is considered "better" in that it uses a dictionary of inflected forms that map back to the non-inflected form. So in theory, if the dictionary is comprehensive, you can properly replace inflected forms with their correct non-inflected forms. (e.g. 'enhancement' -> 'enhance') If your dictionary isn't comprehensive and comes across a token it doesn't recognize, you can try falling back onto a stemming algorithm.