3 ms·
You are right about log probabilities ... I added a note about it (I'm the author of the article). Also on spam filtering, there's a note there about stemming
by bad_user 15y ago
You are right about log probabilities ... I added a note about it (I'm the author of the article).
Also on spam filtering, there's a note there about stemming not being optimal for spam, but the spam classifier was just an example, otherwise stemming is still useful when bootstrapping with a small dataset.
- irahul 15y agoOn a related note, I generally simplify the implementation further: P(spam/doc) = P(w1/spam) * P(w2/spam) * ... * P(wn/spam) ------------------------------------------- P(w1) * P(w2) * ... * P(wn) Of course, I do the log and anti-log dance(if non-log values are required, otherwise leave it at the log probabilities), or else both denominator and numerator tend to 0. I ignore P(spam) because more often than not, I don't have the representative value for P(spam). If I am using a training dataset, P(spam) isn't chance - it is just the training data I used. If I am given a value for it, I don't believe P(spam) is a constant for all sorts of data, and I think using it skews the results. Also, if you assume P(spam) is constant, the scale remains the same if the constant is dropped. That won't give us the actual probability, but this is naive bayes which assumes w1 and w2 are independent, when they are not. I don't think we get the actual probabilities in any case - that isn't an issue as we aren't concerned with actual probabilities but a heuristic which we can use to classify. I tried using extended form of the bayes theorem, but in my sample runs using cross validation, it gave me worse results. Now, as a rule of thumb, I stick to Peter Norwig's advice - "If your training dataset is relevant, use the simplest algorithm and it will work. If your training data isn't accurate, even the complicated algorithms won't buy you much." In my experience, I always had better results with short bayes compared to extended bayes. One missing aspect about this naive bayes implementation is it assumes all features are relevant. I have tried logistic regressions with regularization, and they tended to give me same or worse results. So, for most of the classification cases, bayes is my goto algorithm - I don't do any stemming or lemmatization or feature selection. I run it, and if results aren't relevant, then I re-examine the dataset. If they still aren't relevant, I go for feature selection. If they still aren't relevant, well, I mostly give up.
- robertskmiles 15y agoI found that note very interesting: > consider "house" and "housing" ... the former is used less frequently in a spammy context then the later. The point is that stemming is valuable unless P(spam|root) is substantially different from P(spam|derivedWord). But that data is available to you through Bayes. Has anyone tried a 'conditional stemming' system where the word is stemmed iff |P(spam|root) - P(spam|derivedWord)| < someThreshold ? I wonder if that would improve classification accuracy. I suppose it might be a big performance hit on the stemming algorithm though.
- sigil 15y agoInteresting idea. I bet within some specific application domain, stemmed probabilities either diverge or don't across the board, so it wouldn't pay to conditionally stem term-by-term.
- robertskmiles 15y agoYeah, probably. That's something that's fairly easy to measure for any given data set, so you can check to see if it's worthwhile before you do it.