4 ms·
> Naive Bayes assumes variable independence, for one thing That's true, but every model makes assumptions that are wrong. The puzzling thing about Naive Bayes
by ced 11y ago
> Naive Bayes assumes variable independence, for one thing
That's true, but every model makes assumptions that are wrong. The puzzling thing about Naive Bayes was how well it performs in practice in spite of its assumptions being so wrong. I believe that there have been papers explaining this. I would look at Russell and Norvig's book for a start.
One thing that Naive Bayes sucks at is providing good probabilistic estimates. It is nearly always overconfident in its predictions (eg. P(rain) = 0.99999999) even though its classification accuracy can be pretty good (relative to its simplicity). Logistic regression fares a lot better for probabilities.
- deleted 11y ago[deleted]
- ch4s3 11y agoYeah, Naive Bayes isn't very good for probabilistic estimates, but its often quite good at classification. I've had great luck using it to classify poorly coded medical data. I maintain this ruby gem [0] that implements Naive Bayes as a text classifier, and for a lot of tasks, it's good enough. [0]https://github.com/jekyll/classifier-reborn https://github.com/jekyll/classifier-reborn
- mrdrozdov 11y agoHave you tried a Perceptron or MaxEnt model for your medical data? They're also fairly simple (MaxEnt slightly less so), but will give you solid results.
- ch4s3 11y agoI looked into perceptron, but it seemed overkill and I didn't want to write a classifier from scratch in Ruby or Java. I wasn't previously aware of MaxEnt, which looks cool. Bayes is nice for the data I'm looking at, because my features are pretty independent.
- _dps 11y ago> One thing that Naive Bayes sucks at is providing good probabilistic estimates. This is true based on how it's described in textbooks, but in practice it should always be combined with a calibration algorithm (which add a trivial O(n) cost to the process). The common choices here are Platt [0] or Isotonic Regression [1] (the latter should, itself, be combined with a regularization algorithm because it can easily overfit for outliers). Given a calibration algorithm Naive Bayes produces probability estimates every bit as reasonable as any other algorithm. [0] https://en.wikipedia.org/wiki/Platt_scaling https://en.wikipedia.org/wiki/Platt_scaling [1] https://en.wikipedia.org/wiki/Isotonic_regression https://en.wikipedia.org/wiki/Isotonic_regression
- otabdeveloper 11y ago> but in practice it should always be combined with a calibration algorithm (which add a trivial O(n) cost to the process). Why not just use logistic regression at this point? The only benefit of Naive Bayes over logistic regression is that Naive Bayes is simpler to code.
- _dps 11y agoThe calibration cost is trivial compared to the coefficient learning cost. Very roughly, calibration is O(records) whereas coefficient learning is O(records * features). So the tiny add-on cost of calibration shouldn't affect anyone's evaluation of the relative merits of algorithms. NB still retains its computational advantage. One thing that is often discounted in theoretical discussions is that NB takes much less I/O than something like LR, typically in the range of 5-100x (depending on how many iterations you want to do updating your LR coefficients). If you're doing, for example, a MapReduce implementation then NB has huge computational advantages. In LR each coefficient update costs you another map/reduce pass across your entire data set (whereas NB is always done in exactly one iteration). So if NB + calibration gets you something close to LR for vastly less computation and I/O, why wouldn't you use it? Having said that, if you're talking about small amounts of data that fit into RAM and you can "just load into R", then sure use LR over NB. For that matter use a Random Forest [0]. The reason NB is still around is because it offers a point in the design space where you spend almost no resources and still get something surprisingly useful (and recalibration narrows the utility gap between NB and better methods even more). [0] And you should still consider calibrating your random forest's output.
- Houshalter 11y agoGeoff Hinton has a theory that Naive Bayes is just logistic regression with dropout: https://www.youtube.com/watch?v=DleXA5ADG78 https://www.youtube.com/watch?v=DleXA5ADG78 And that if you divide all the parameters by a certain amount, you would get accurate probability estimates.
- eveningcoffee 11y agoThis is amazing talk. He has many other very interesting ideas.