6 ms·

Naive Bayes and Text Classification I – Introduction and Theory

115 points by rasbt 12y ago

rasbt 12y ago
I posted this article some while ago, and I thought it may not be a bad idea to archive it on arXiv. What do you think? The (Latex) PDF version would look like this: http://sebastianraschka.com/PDFs/articles/naive_bayes_1.pdf http://sebastianraschka.com/PDFs/articles/naive_bayes_1.pdf I just signed up on arXiv and see that the catch is that I'd need 2 recommendations for the categories Computer Science -> Learning or Statistics -> Machine Learning. Would anyone here who would want to give me a recommendation? I would really appreciate it!
en4bz 12y ago
I'm currently working on a school project to classify abstracts from papers in to 4 categories. Currently Bernoulli Naive Bayes scores 85% accuracy. For comparison kNN scores about the same.
- rasbt 12y ago
  Sounds cool. I think the most crucial part is really the data (or feature) set. The choice of classifiers makes a difference too, but having good data (salient, invariant, discriminatory) makes the largest difference. One think about naive Bayes and kNN: Although the performance may be similar, I would consider Naive beneficial if the data set is large. kNN requires to re-evaluate the data. Naive Bayes makes the classification computationally cheaper and with "on-line" learning, you can still improve or adjust it over time
MojoJolo 12y ago
I actually really like doing text classification using Naive Bayes. I'm still new to it and still learning a lot. But one thing I'm having a hard time answering is explaining Naive Bayes classification in simple terms. If that was asked to you, how can you explain Naive Bayes classification in simple terms?
- yen223 12y ago
  I know a lot of spam has the word "Viagra", and very few legitimate emails have the word "Viagra". So if an email contains the word "Viagra", it's more likely to be spam. Naive Bayes classification just applies that to all words in all your emails.
  - rasbt 12y ago
    I like this simple and intuitive explanation. Hm, maybe a short mentioning of the "a priori" knowledge would be also worthwhile!
syllogism 12y ago
> Naive Bayes classifiers, a family of classifiers that are based on the popular Bayes’ probability theorem, are known for creating simple yet well performing models, especially in the fields of document classification and disease prediction. But this is simply not true! They _don't_ perform well. There's really no reason to teach people Naive Bayes any more, except as a footnote when explaining log-linear/MaxEnt models. MaxEnt is not so complicated, and it makes Naive Bayes fully obsolete. And if MaxEnt is in some way too complicated/expensive, Averaged Perceptron is generally much better than NB, can be implemented in 50 lines of Python, and has far fewer hyper-parameters. A common way for machine learning courses to suck is to teach students about a bunch of crap, obsolete algorithms they should never use, simply for historical reasons --- they used to be in the course, so they stay in the course.
- Radim 12y ago
  Now you're just being silly. Yes, other models may perform better, but Naive Bayes is a simple model that performs quite well, especially in the fields mentioned. The statement is completely uncontroversial.
  - syllogism 12y ago
    We can argue about what "well" means I guess. On many problems the gap will be small, but you should never expect NB to out-perform the better models. And on lots of problems, the difference will be substantial. NB is strictly worse than MaxEnt. Why settle for "quite well"? Just use the better algorithm, always. It's important to understand when solutions have been superseded.
    - Radim 12y ago
      That part I agree on. I guess my point is that pragmatic progress in science is hardly ever as earth-shattering and all-changing as the academic circles flaunt it to be (understandably so, in these "publish or perish" times). "Old stuff" doesn't suddenly become rubbish just because "new stuff" comes out. Even when the new stuff offers a better, more predictive/complete model of data, there's typically a class of problems where the improvement is inconsequential (cf. gravity ala Newton vs. general relativity). But yes, when you understand the implications of switching, and the cost is simply a different `pip install`, switch away.
dj-wonk 12y ago
There are typos: 2.6.3. The correct term is "Additive Smoothing" https://en.wikipedia.org/wiki/Additive_smoothing https://en.wikipedia.org/wiki/Additive_smoothing not "smoothening". 3.3. Letters transposed. The author meant "Multi-variate".
- rasbt 12y ago
  Thanks a lot, I always called it "smoothening" for whatever reason, but it should be fixed now :)
dj-wonk 12y ago
Re: "Empirical studies showed that the mutli-variate Bernoulli model is inferior to the multinomial Bayes model, and the latter has been shown to reduce the error by 27% on average [13]." I remain skeptical of statements that claim that one kind of model is categorically inferior. Also, I have a hard time believing this is a fair summary of the literature. First, one study is cited, but the sentence implies many. Second, the citation in the original post is not a fair summary of the cited research. The abstract of "A Comparison of Event Models for Naive Bayes Text Classification" says: "This paper aims to clarify the confusion by describing the differences and details of these two models, and by empirically comparing their classification performance on five text corpora. We find that the multi-variate Bernoulli performs well with small vocabulary sizes, but that the multinomial performs usually performs even better at larger vocabulary sizes—providing on average a 27% reduction in error over the multi-variate Bernoulli model at any vocabulary size."
- rasbt 12y ago
  Thanks for the comment, and I agree! I rephrased it a little bit to "Empirical comparisons provide evidence that the multinomial model tends to outperform the multi-variate Bernoulli model if the vocabulary size is relatively large [13]. However, the performance of machine learning algorithms is highly dependent on the appropriate choice of features. In the case of naive Bayes classifiers and text classification, large differences in performance can be attributed to the choices of stop word removal, stemming, and token-length [14]. In practice, it is recommended that the choice between a multi-variate Bernoulli or multinomial model for text classification should precede comparative studies including different combinations of feature extraction and selection steps."
  - dj-wonk 12y ago
    Thanks for the updates.
meaty_sausages 12y ago
Lexicon-based approaches are better than all these approaches. The difference between MaxEnt and NB is similar (the same?) as a binomial/multinomial regression. They have all of the drawbacks and only a few of the benefits. A decent word list would be about as accurate as a regression (70% or so). Account for syntax and it will get you higher.

115 points by rasbt 12y ago

rasbt 12y ago

I posted this article some while ago, and I thought it may not be a bad idea to archive it on arXiv. What do you think? The (Latex) PDF version would look like this: http://sebastianraschka.com/PDFs/articles/naive_bayes_1.pdf http://sebastianraschka.com/PDFs/articles/naive_bayes_1.pdf I just signed up on arXiv and see that the catch is that I'd need 2 recommendations for the categories Computer Science -> Learning or Statistics -> Machine Learning. Would anyone here who would want to give me a recommendation? I would really appreciate it!

en4bz 12y ago

I'm currently working on a school project to classify abstracts from papers in to 4 categories. Currently Bernoulli Naive Bayes scores 85% accuracy. For comparison kNN scores about the same.

rasbt 12y ago
Sounds cool. I think the most crucial part is really the data (or feature) set. The choice of classifiers makes a difference too, but having good data (salient, invariant, discriminatory) makes the largest difference. One think about naive Bayes and kNN: Although the performance may be similar, I would consider Naive beneficial if the data set is large. kNN requires to re-evaluate the data. Naive Bayes makes the classification computationally cheaper and with "on-line" learning, you can still improve or adjust it over time

MojoJolo 12y ago

I actually really like doing text classification using Naive Bayes. I'm still new to it and still learning a lot. But one thing I'm having a hard time answering is explaining Naive Bayes classification in simple terms. If that was asked to you, how can you explain Naive Bayes classification in simple terms?

yen223 12y ago
I know a lot of spam has the word "Viagra", and very few legitimate emails have the word "Viagra". So if an email contains the word "Viagra", it's more likely to be spam. Naive Bayes classification just applies that to all words in all your emails.
- rasbt 12y ago
  I like this simple and intuitive explanation. Hm, maybe a short mentioning of the "a priori" knowledge would be also worthwhile!

syllogism 12y ago

> Naive Bayes classifiers, a family of classifiers that are based on the popular Bayes’ probability theorem, are known for creating simple yet well performing models, especially in the fields of document classification and disease prediction. But this is simply not true! They _don't_ perform well. There's really no reason to teach people Naive Bayes any more, except as a footnote when explaining log-linear/MaxEnt models. MaxEnt is not so complicated, and it makes Naive Bayes fully obsolete. And if MaxEnt is in some way too complicated/expensive, Averaged Perceptron is generally much better than NB, can be implemented in 50 lines of Python, and has far fewer hyper-parameters. A common way for machine learning courses to suck is to teach students about a bunch of crap, obsolete algorithms they should never use, simply for historical reasons --- they used to be in the course, so they stay in the course.

Radim 12y ago
Now you're just being silly. Yes, other models may perform better, but Naive Bayes is a simple model that performs quite well, especially in the fields mentioned. The statement is completely uncontroversial.
- syllogism 12y ago
  We can argue about what "well" means I guess. On many problems the gap will be small, but you should never expect NB to out-perform the better models. And on lots of problems, the difference will be substantial. NB is strictly worse than MaxEnt. Why settle for "quite well"? Just use the better algorithm, always. It's important to understand when solutions have been superseded.
  - Radim 12y ago
    That part I agree on. I guess my point is that pragmatic progress in science is hardly ever as earth-shattering and all-changing as the academic circles flaunt it to be (understandably so, in these "publish or perish" times). "Old stuff" doesn't suddenly become rubbish just because "new stuff" comes out. Even when the new stuff offers a better, more predictive/complete model of data, there's typically a class of problems where the improvement is inconsequential (cf. gravity ala Newton vs. general relativity). But yes, when you understand the implications of switching, and the cost is simply a different `pip install`, switch away.

dj-wonk 12y ago

There are typos: 2.6.3. The correct term is "Additive Smoothing" https://en.wikipedia.org/wiki/Additive_smoothing https://en.wikipedia.org/wiki/Additive_smoothing not "smoothening". 3.3. Letters transposed. The author meant "Multi-variate".

rasbt 12y ago
Thanks a lot, I always called it "smoothening" for whatever reason, but it should be fixed now :)

dj-wonk 12y ago

Re: "Empirical studies showed that the mutli-variate Bernoulli model is inferior to the multinomial Bayes model, and the latter has been shown to reduce the error by 27% on average [13]." I remain skeptical of statements that claim that one kind of model is categorically inferior. Also, I have a hard time believing this is a fair summary of the literature. First, one study is cited, but the sentence implies many. Second, the citation in the original post is not a fair summary of the cited research. The abstract of "A Comparison of Event Models for Naive Bayes Text Classification" says: "This paper aims to clarify the confusion by describing the differences and details of these two models, and by empirically comparing their classification performance on five text corpora. We find that the multi-variate Bernoulli performs well with small vocabulary sizes, but that the multinomial performs usually performs even better at larger vocabulary sizes—providing on average a 27% reduction in error over the multi-variate Bernoulli model at any vocabulary size."

rasbt 12y ago
Thanks for the comment, and I agree! I rephrased it a little bit to "Empirical comparisons provide evidence that the multinomial model tends to outperform the multi-variate Bernoulli model if the vocabulary size is relatively large [13]. However, the performance of machine learning algorithms is highly dependent on the appropriate choice of features. In the case of naive Bayes classifiers and text classification, large differences in performance can be attributed to the choices of stop word removal, stemming, and token-length [14]. In practice, it is recommended that the choice between a multi-variate Bernoulli or multinomial model for text classification should precede comparative studies including different combinations of feature extraction and selection steps."
- dj-wonk 12y ago
  Thanks for the updates.

meaty_sausages 12y ago

Lexicon-based approaches are better than all these approaches. The difference between MaxEnt and NB is similar (the same?) as a binomial/multinomial regression. They have all of the drawbacks and only a few of the benefits. A decent word list would be about as accurate as a regression (70% or so). Account for syntax and it will get you higher.