4 ms·
I found that note very interesting: > consider "house" and "housing" ... the former is used less frequently in a spammy context then the later. The point is t
by robertskmiles 15y ago
I found that note very interesting:
> consider "house" and "housing" ... the former is used less frequently in a spammy context then the later.
The point is that stemming is valuable unless P(spam|root) is substantially different from P(spam|derivedWord). But that data is available to you through Bayes. Has anyone tried a 'conditional stemming' system where the word is stemmed iff
|P(spam|root) - P(spam|derivedWord)| < someThreshold
? I wonder if that would improve classification accuracy. I suppose it might be a big performance hit on the stemming algorithm though.
- sigil 15y agoInteresting idea. I bet within some specific application domain, stemmed probabilities either diverge or don't across the board, so it wouldn't pay to conditionally stem term-by-term.
- robertskmiles 15y agoYeah, probably. That's something that's fairly easy to measure for any given data set, so you can check to see if it's worthwhile before you do it.