6 ms·

> Naive Bayes classifiers, a family of classifiers that are based on the popular Bayes’ probability theorem, are known for creating simple yet well performing m

> Naive Bayes classifiers, a family of classifiers that are based on the popular Bayes’ probability theorem, are known for creating simple yet well performing models, especially in the fields of document classification and disease prediction. But this is simply not true! They _don't_ perform well. There's really no reason to teach people Naive Bayes any more, except as a footnote when explaining log-linear/MaxEnt models. MaxEnt is not so complicated, and it makes Naive Bayes fully obsolete. And if MaxEnt is in some way too complicated/expensive, Averaged Perceptron is generally much better than NB, can be implemented in 50 lines of Python, and has far fewer hyper-parameters. A common way for machine learning courses to suck is to teach students about a bunch of crap, obsolete algorithms they should never use, simply for historical reasons --- they used to be in the course, so they stay in the course.

Radim 12y ago
Now you're just being silly. Yes, other models may perform better, but Naive Bayes is a simple model that performs quite well, especially in the fields mentioned. The statement is completely uncontroversial.
- syllogism 12y ago
  We can argue about what "well" means I guess. On many problems the gap will be small, but you should never expect NB to out-perform the better models. And on lots of problems, the difference will be substantial. NB is strictly worse than MaxEnt. Why settle for "quite well"? Just use the better algorithm, always. It's important to understand when solutions have been superseded.
  - Radim 12y ago
    That part I agree on. I guess my point is that pragmatic progress in science is hardly ever as earth-shattering and all-changing as the academic circles flaunt it to be (understandably so, in these "publish or perish" times). "Old stuff" doesn't suddenly become rubbish just because "new stuff" comes out. Even when the new stuff offers a better, more predictive/complete model of data, there's typically a class of problems where the improvement is inconsequential (cf. gravity ala Newton vs. general relativity). But yes, when you understand the implications of switching, and the cost is simply a different `pip install`, switch away.
MrQuincle 12y ago
If the features are independent, it is better to use NB, because then you don't need to find it out the hard way. So your hypothesis of maxent being always better is flawed. Moreover NB can be seen as a joint maxent method, so don't compare it on problems with conditional structure. This website explains nicely the connection between them: http://www.denizyuret.com/2010/11/naive-bayes-is-joint-maximum-entropy.html?m=1 http://www.denizyuret.com/2010/11/naive-bayes-is-joint-maxim...
- Chronic29 12y ago
  No, Naive Bayes is not better.
- andreasvc 12y ago
  Assumptions such as independence are always dutifully mentioned when discussing statistical methods, but I never see a discussion about in which sort of practical problems such assumptions hold. In the case of independence and text classification, it seems to me that it doesn't hold, certainly not on the scale of a paragraph. As the blog post mentions, naive Bayes performs well despite violating the independence assumption; I'm curious to understand why that is.
  - rasbt 12y ago
    In practice, I would say that it is a very challenging task to determine whether variables are totally independent or not: It is more about the strength of independence. Every time something is "estimated", you'd introduce an additional source of error, and in cases of weak independence the violation may be negligible. Modeling the conditional dependences is basically the idea of Bayesian Belief Networks, and you might find the paper "Comparing Bayesian Network Classifiers" interesting in this context (includes comparisons to Naive Bayes) http://link.springer.com/article/10.1023/A:1007465528199 http://link.springer.com/article/10.1023/A:1007465528199
Dn_Ab 12y ago
As others have mentioned, when the assumptions of conditional independence are met, Gaussian NaiveBayes and MaxEnt will asymptotically learn the same classifier. Except NB will learn it in O(log dimensions) instead of O(dimensions) examples. So in those cases and where you have fewer examples you'll want to use NB. Even when the independence assumptions are not met, NB will often produce a good classifier if you don't care about the accuracy of the probabilities. NB is also online, resistant to the curse of dimensionality and for categorical data will learn polynomial decision boundaries. So the gain in the trade-off is in ease of implementation (you can write one in a dozen lines or less) that you can throw something close to your raw data at and often get good enough results without any ahead of time training, kernels or regularization. You are right though that MaxEnt (some sparsity capturing algorithms are as powerful as SVMs) will in general outperform naive bayes and that the averaged perceptron probably has, on average, the best performance/ease of implementation ratio. But for many implementors/problems, where clarity is paramount, those distinctions will not be worth the cost. Also, you have a wonderful website.
- instakill 12y ago
  What's his website? It's not in his bio.
  - andreasvc 12y ago
    http://honnibal.wordpress.com/ http://honnibal.wordpress.com/
  - syllogism 12y ago
    Thanks, fixed.
- syllogism 12y ago
  Thanks for the kind compliment :). And I should probably be less shouty here. Okay, it's not true that NB is strictly dominated by MaxEnt. But, look at the two example problems the author gave, where Naive Bayes was said to be a good choice. The parameters there definitely, definitely won't be conditionally independent. And probably you'll have enough data. Naive Bayes is a bad choice here, as it is in most other situations. So, I think the caveats you've raised are all true...but, I still wouldn't be raising them in a class I was teaching. I think it's easy to have less useful discussion, made up of individually more true statements. I think a common problem in technical discussion is too much attention to every qualification, and every caveat. People assume that there's some sort of proportionality between the importance of an idea/topic, and the airtime you give it. And I think they do this implicitly, in a way that's really hard to consciously over-rule. So I think it's really important to editorialise. I think a lot of technical discussion would be better off by making statements that are untrue at the edges, accepting the imprecision, and moving on. That's why I'm dismayed to see someone's started writing a detailed tutorial on Naive Bayes, especially set to two problem domains it's a bad choice for. Even if it's composed of entirely true statements, I think its net effect is to miseducate people.

Radim 12y ago

Now you're just being silly. Yes, other models may perform better, but Naive Bayes is a simple model that performs quite well, especially in the fields mentioned. The statement is completely uncontroversial.

syllogism 12y ago
We can argue about what "well" means I guess. On many problems the gap will be small, but you should never expect NB to out-perform the better models. And on lots of problems, the difference will be substantial. NB is strictly worse than MaxEnt. Why settle for "quite well"? Just use the better algorithm, always. It's important to understand when solutions have been superseded.
- Radim 12y ago
  That part I agree on. I guess my point is that pragmatic progress in science is hardly ever as earth-shattering and all-changing as the academic circles flaunt it to be (understandably so, in these "publish or perish" times). "Old stuff" doesn't suddenly become rubbish just because "new stuff" comes out. Even when the new stuff offers a better, more predictive/complete model of data, there's typically a class of problems where the improvement is inconsequential (cf. gravity ala Newton vs. general relativity). But yes, when you understand the implications of switching, and the cost is simply a different `pip install`, switch away.

MrQuincle 12y ago

If the features are independent, it is better to use NB, because then you don't need to find it out the hard way. So your hypothesis of maxent being always better is flawed. Moreover NB can be seen as a joint maxent method, so don't compare it on problems with conditional structure. This website explains nicely the connection between them: http://www.denizyuret.com/2010/11/naive-bayes-is-joint-maximum-entropy.html?m=1 http://www.denizyuret.com/2010/11/naive-bayes-is-joint-maxim...

Chronic29 12y ago
No, Naive Bayes is not better.
andreasvc 12y ago
Assumptions such as independence are always dutifully mentioned when discussing statistical methods, but I never see a discussion about in which sort of practical problems such assumptions hold. In the case of independence and text classification, it seems to me that it doesn't hold, certainly not on the scale of a paragraph. As the blog post mentions, naive Bayes performs well despite violating the independence assumption; I'm curious to understand why that is.
- rasbt 12y ago
  In practice, I would say that it is a very challenging task to determine whether variables are totally independent or not: It is more about the strength of independence. Every time something is "estimated", you'd introduce an additional source of error, and in cases of weak independence the violation may be negligible. Modeling the conditional dependences is basically the idea of Bayesian Belief Networks, and you might find the paper "Comparing Bayesian Network Classifiers" interesting in this context (includes comparisons to Naive Bayes) http://link.springer.com/article/10.1023/A:1007465528199 http://link.springer.com/article/10.1023/A:1007465528199

Dn_Ab 12y ago

As others have mentioned, when the assumptions of conditional independence are met, Gaussian NaiveBayes and MaxEnt will asymptotically learn the same classifier. Except NB will learn it in O(log dimensions) instead of O(dimensions) examples. So in those cases and where you have fewer examples you'll want to use NB. Even when the independence assumptions are not met, NB will often produce a good classifier if you don't care about the accuracy of the probabilities. NB is also online, resistant to the curse of dimensionality and for categorical data will learn polynomial decision boundaries. So the gain in the trade-off is in ease of implementation (you can write one in a dozen lines or less) that you can throw something close to your raw data at and often get good enough results without any ahead of time training, kernels or regularization. You are right though that MaxEnt (some sparsity capturing algorithms are as powerful as SVMs) will in general outperform naive bayes and that the averaged perceptron probably has, on average, the best performance/ease of implementation ratio. But for many implementors/problems, where clarity is paramount, those distinctions will not be worth the cost. Also, you have a wonderful website.

instakill 12y ago
What's his website? It's not in his bio.
- andreasvc 12y ago
  http://honnibal.wordpress.com/ http://honnibal.wordpress.com/
- syllogism 12y ago
  Thanks, fixed.
syllogism 12y ago
Thanks for the kind compliment :). And I should probably be less shouty here. Okay, it's not true that NB is strictly dominated by MaxEnt. But, look at the two example problems the author gave, where Naive Bayes was said to be a good choice. The parameters there definitely, definitely won't be conditionally independent. And probably you'll have enough data. Naive Bayes is a bad choice here, as it is in most other situations. So, I think the caveats you've raised are all true...but, I still wouldn't be raising them in a class I was teaching. I think it's easy to have less useful discussion, made up of individually more true statements. I think a common problem in technical discussion is too much attention to every qualification, and every caveat. People assume that there's some sort of proportionality between the importance of an idea/topic, and the airtime you give it. And I think they do this implicitly, in a way that's really hard to consciously over-rule. So I think it's really important to editorialise. I think a lot of technical discussion would be better off by making statements that are untrue at the edges, accepting the imprecision, and moving on. That's why I'm dismayed to see someone's started writing a detailed tutorial on Naive Bayes, especially set to two problem domains it's a bad choice for. Even if it's composed of entirely true statements, I think its net effect is to miseducate people.