8 ms·

A practical explanation of a Naive Bayes classifier

350 points by feconroses 9y ago

schuetze 9y ago
Considering the relative ease of implementation, classification accuracy with smaller datasets, and computational efficiency of Naive Bayes classifiers, I am surprised that they are not mentioned as often as other machine learning competitors, such as random forest. Are there major drawbacks to Naive Bayes classifiers? Is it just that they aren't as accurate on large datasets?
- data_scientist 9y ago
  It doesn't have a good accuracy. I have yet to see a real-life dataset where it's better than to just call LogisticRegressionCV from scikit-learn. For bigger datasets, you may use vowpal wabbit or Fasttext. It may be a little bit slower for the training (but not so much), and as fast as LR for the training. What is the purpose of using an algorithm when another one is just better?
  - boto3 9y ago
    There's a reason for that. NB needs to fit more params than LR (and other linear discriminant learners), e.g. for binary classification, NB needs to fit 2*N params while LR just N.
    - imh 9y ago
      Not true. Let's derive it. Here's NB for a binary classification: p(c|w)*p(w) = p(w|c) * p(c) p(c|w) = p(w|c) * p(c) / p(w) p(c|w) = p(w|c) * p(c) / [sum_i p(w|c_i) * p(c_i)] Let's look at the probability of class 1. p(c_1|w) = p(w|c_1) * p(c_1) / [sum_i p(w|c_i) * p(c_i)] Notice how the numerator is going to show up in the denominator. We can simplify that by bringing it into the denominator: p(c_1|w) = 1 / [sum_i p(w|c_i) * p(c_i) / p(w|c_1) / p(c_1] Then cancel it out: p(c_1|w) = 1 / [1 + p(w|c_0) / p(w|c_1) * p(c_0) / p(c_1)] Not let's apply the NB assumptions: p(w|c_0) / p(w|c_1) = prod_i p(w_i | c_0) / p(w_i | c_1) Now, if you take the log of the final p(c_1|w) I derived, the product in p(w|c_0) / p(w|c_1) turns into a exponentiated sum, giving you one parameter per word, plus an intercept for the log of p(c_0) / p(c_1). You end up with exactly the same 1/(1+exp(linear stuff)) with the same parametrization and form you have in logistic regression [0]. This is a broader thing that a given graphical model with a given fixed parametrization can be generatively or discriminatively trained. They will end up learning different models, but that's because they make different assumptions, not because of different parametrizations. [0] linear stuff = intercept + sum_i (coefficient of word_i)
      - abhgh 9y ago
        I may be missing something, but how do you take a log on the RHS of p(c_1|w)? The denominator has a sum i.e. "1+ something", which the log doesn't distribute over.
        imh 9y ago
        Heh, nope you're right I was writing to quickly. I meant to say to take a log of of the probability term in p(c_1|w). So you take p(c_1|w) = 1/(1+stuff) and turn it into p(c_1|w) = 1/(1+exp(log(stuff))). stuff is a product, so log(stuff) is a sum. Good catch.
        abhgh 9y ago
        Oh, I see it now. Thanks for clarifying! In that case, you needn't have kept the denominator around at all. Since P(w) is not based on a class i.e. its not class conditioned, the classifier could have directly calculated P(C_1|w)/P(C_0|w). The P(w) term cancels out, and you end up with the product of ratios of feature probabilities conditioned on the classes. Note though that, for K-classes, K>2, the number of parameters you need to store would blow up. You would need have these N ratios: P(w_j|C_x)/P(w_j|C_y) for all possible classes x,y. Here N is the number of features. So, in all N*C(K,2) values. On the contrary, multiclass softmax regression (the discriminative analogue for NB) would need NK parameters.
        imh 9y ago
        In the multiclass problem, they have the same number of degrees of parameters. You can see this by choosing a reference class c_y. The NB parameter for class c_x and feature w_j is f_jxy = log(p(w_j|c_x)/p(w_j|c_y)). If you want a new reference class c_w, you can see that f_jxw = f_jxy - f_jwy. No new learned parameters needed. You need one of those parameters per N features and (K-1) classes that aren't c_y. So you get N(K-1) features. This is the same as for multiclass softmax regression: N(K-1) instead of the NK you wrote (which you can see by working it out with a reference class c_y). It really is the same parametrization. The analogue with NK parametrized softmax may be more straightforward if you just use NK naive bayes features of the form f_jx = log(p(w_j|c_x)) for each class and check the equivalence with softmax. It really is just a special case of a more general rule that a given PGM with a fixed parametrization can be trained discriminatively or generatively.
- imh 9y ago
  The (conditional) independence assumptions it makes are pretty strong and usually inaccurate. In the text example, consider "Steph Curry" and "Warriors." These are going to show up very frequently together, and both provide strong evidence that the topic is sports. However, they don't provide strongly independent evidence of sports. Supposing Curry announced he was making an investment in a local restaurant, the article would probably mention the Warriors just because it's Curry. The problem is that seeing "Warriors" when you see "Steph Curry" shouldn't make you that much more confident that it's sports than just seeing "Steph Curry" alone. Just like seeing "University" with "Stanford" shouldn't change any inferences much from seeing "Stanford" alone. In naive bayes models, these are treated as independent pieces of evidence and that can lead to overconfidence and errors. Naive bayes is a generative model, so in flipping bayes rule around to discriminate classes, you have to be sure your probability model is decent. In a discriminative model, you just go straight to learning the p(class | observations) and have no requirement for a decent model of p(observations). p(observations) is the kind of model that would have to know about "university" and "stanford" being likely to be observed together. Often, it's better to go straight to the discriminative model. In this case, logistic regression.
- j2kun 9y ago
  Boosted decision trees account for >50% of all Kaggle winners. That is the real surprise, since people rarely talk about boosted decision trees.
  - abhgh 9y ago
    Deep Learning is in vogue so I think we see a lot around it written about in blogs, media etc. GBDTs are used in the industry though.
  - nl 9y ago
    There is no greater discussion topic on Kaggle Slack than a comparison of XGB vs LightGBM vs CatGBM.
- autokad 9y ago
  many competition solutions use stacking, where you use the probabilities of several classifiers are used instead of their predictions. Since naive bayes makes an independence assumption, and most competitions have features that are highly correlated, its probability / confidence of its prediction is probably a lot more useless than other classifiers. besides that, nb doesn't do feature selection, regularization, and interactions like many tree based algorithms (such as XGBoost) do out of the box. interactions is a big one for me. With NB, you have to add an interaction feature by multiplying two together, but in creating the feature you must steal some probability away from the other stuff, and finding a good interaction among many features is hard. The tree structure of random forests naturally captures that edit: to your last point, nb does worse on larger data sets. when you dont have a lot of data, you have to make larger assumptions. NB makes a big independence assumption, thus it does well on very small data sets but falls short on large ones
- abhgh 9y ago
  Accuracy is a problem. NBs are linear classifiers, and more so, the linear boundary is found by assuming features to be independent. So, in cases: (1) data cannot be separated linearly - NBs don't do well for the same reason as any other linear classifier wouldn't work well (2) for linearly separable data - if the feature independence assumption is incompatible with the data, linear classifiers that don't make this assumption will beat it. For ex you can prove that logistic regression beats (or is at least as good as) Naive Bayes asymptotically. [1] [1] See proposition 1 here: "On Discriminative vs Generative classifiers ... Andrew ng, Michael Jordan. https://ai.stanford.edu/~ang/papers/nips01-discriminativegenerative.pdf https://ai.stanford.edu/~ang/papers/nips01-discriminativegen... [PDF]
mooman219 9y ago
The examples other people are using are fairly narrow. I would like to substantiate that text categorization via naive bayes classifier is surprisingly accurate and simple. This paper[1] uses ngrams and a simple out of place measure to compare articles against different verticals and often sees greater than 99% accuracy for relatively small blocks of text. The out of place measure also adds a penalty to features not found in the document, which helps establish the individuality for the category classification. Raw matching performance is also fairly impressive; A less naive implementation is also highly parallelizable. [1] http://odur.let.rug.nl/~vannoord/TextCat/textcat.pdf http://odur.let.rug.nl/~vannoord/TextCat/textcat.pdf
- abhgh 9y ago
  From experience, I suspect NB does well on text primarily for two reasons: (1) High dimensionality - this is also the reason why SVMs with linear kernels do so well on text (NBs find a linear boundary too - but a different linear boundary than linear-kernel SVMs) (2) this reason applies to certain cases only: text where "token" based estimates are sufficiently discriminatory. For ex take a dataset that has two kinds of documents - one talking about product A and the other talking about product B. And you want to label the documents based on which product they're talking about. Here, just noticing if the term "A" or "B" appears in the document is good enough for classification. You don't have to have powerful models that infer "connotations" of words based on context. NB will do well here, esp if bundled with a feature selection technique that weeds out noisy features (like sthg based on Normalized Mutual Information)
moultano 9y ago
A practical issue for Naive Bayes that also infects linear models is bias w.r.t. document length. Typically when you are detecting a rare, relatively compact class such as sports articles (or spam) you will tend to have a strongly negative prior, many positive features, and few negative ones. As a consequence, as the length of your text increases, not only does the variance of your prediction increase, but the mean tends to as well. This leads to all very long documents being classified as positive, regardless of their text. You can observe this by training your model and then classifying /usr/dict/words. This is the most common mistake I've seen in production use of linear models on document text. Invariably, they'll misfire on any unusually long document.
- intune 9y ago
  Is there some way to normalize the document length?
  - moultano 9y ago
    Lots of reasonable hacks. 1. Use only the beginning of the document, as that's probably the most important part anyways, and it's fast. 2. Divide the sum of your feature scores by sqrt(n) to give it constant variance, and hopefully keep it comparable with your prior. 3. Split the doc into reasonably sized chunks, and average their scores rather than adding them.
    - _dps 9y ago
      I'll add to this that you can add a very crude (separate) model for the document length and number of distinct words, and use that to flag outlier documents that might bump into the known weaknesses with respect to document length.
    - geezerjay 9y ago
      > 1. Use only the beginning of the document, as that's probably the most important part anyways, and it's fast. That seems to be a solution devised for news articles, as the standard news writing style involves providing answers to the Five Ws up front on the article.
mattbettinson 9y ago
https://github.com/bettinson/bayesian-tag-suggestion https://github.com/bettinson/bayesian-tag-suggestion I wrote one of these in Ruby to classify links into tags! Was fun. I think it got me a job.
rgarreta 9y ago
I would add that another practical aspect about Naive Bayes classifiers is that you can make use of the conditional probabilities for each feature that contributes to the predictions. That gives you some introspection on how the model is working and it's useful when "debugging" classifiers by finding features that should/shouldn't be used. https://monkeylearn.com/blog/how-to-create-text-classifiers-machine-learning/#keyword-cloud https://monkeylearn.com/blog/how-to-create-text-classifiers-...
- abhgh 9y ago
  A side note here: the classification probabilities NB produces are not very accurate and usually need some correction using a process called "calibration".
superasn 9y ago
I created a small program that finds the best sub-reddit given any title text[1] using this algorithm. I'm a total ML noob but it was a interesting project and the results were pretty accurate. I basically used reddit's Bigquery data for the dataset (it's huge!). If you need a practical example of this algo, the algorithm and code is here[2]. [1] https://storage.googleapis.com/superasn/script.html https://storage.googleapis.com/superasn/script.html [2] https://www.reddit.com/r/learnmachinelearning/comments/6hqd6o/p_automatic_reddit_categorizer_update_first/ https://www.reddit.com/r/learnmachinelearning/comments/6hqd6...
drefgert 9y ago
Bayee does a great job of filtering spam but it drives me nuts that obvious spam still appears in my inbox. Suffix trees would fix this in most cases, so why the heck isn't spam filtering using them to remove the obvious spam?
b_ttercup 9y ago
Is Naive Bayes really ever the most practical choice? Yes it is a simple, fast algorithm, but it's usually a non trivial step below other simple models in my experience and doesn't seem to show any major advantages. The results shown here seem good but bag of words models usually do better than you might think on supervised NLP. So what's the motivation?
- tyingq 9y ago
  I thought it was the typical approach for identifying email spam. Has that changed?
- Houshalter 9y ago
  The scikit-learn flowchart recommends it for text data with less than 100k samples when linear SVC doesn't work: http://scikit-learn.org/stable/tutorial/machine_learning_map/index.html http://scikit-learn.org/stable/tutorial/machine_learning_map... AFAIK it's by far the fastest machine learning method and one of the only ones that can be learned "online". I.e. it can just update the model each time it gets a datapoint, and then throw it away without saving it for future training. These are nice properties if you are doing something at a very large scale or in an environment with very limited resources. And if your data happens to actually meet the naive bayes assumptions (that all the features are conditionally independent) then it's literally mathematically optimal and you can't do any better than it. It seems to work fairly well even when that isn't the case though.
  - phunge 9y ago
    Logistic regression can easily be made online too, keep in mind! sklearn has an implementation of online gradient descent, and vowpal wabbit is also excellent at those problems. Naive bayes can be parallelized in ways that SGD can't, that's a whole other conversation.
    - Houshalter 9y ago
      Gradient descent can be made online. But it's very slow and suffers from catastrophic forgetting. Typical gradient descent needs to iterate over the dataset many times, while naive Bayes only needs one pass.
torbjorn 9y ago
I just did an jupyter notebook on Naive Bayes for Siraj Ravel's Math of Intelligence YouTube course. https://github.com/NoahLidell/math-of-intelligence/blob/master/probability_theory/bayesian-classification.ipynb https://github.com/NoahLidell/math-of-intelligence/blob/mast... I used naive bayes to classify raps from Biggie and 2pac.
anthonysarkis 9y ago
C++ implementation for self driving car (school project) https://github.com/swirlingsand/self-driving-car-nanodegree-nd013/blob/master/p11/naive_bayes_cpp/classifier.cpp https://github.com/swirlingsand/self-driving-car-nanodegree-...

350 points by feconroses 9y ago

Considering the relative ease of implementation, classification accuracy with smaller datasets, and computational efficiency of Naive Bayes classifiers, I am surprised that they are not mentioned as often as other machine learning competitors, such as random forest. Are there major drawbacks to Naive Bayes classifiers? Is it just that they aren't as accurate on large datasets?

data_scientist 9y ago
It doesn't have a good accuracy. I have yet to see a real-life dataset where it's better than to just call LogisticRegressionCV from scikit-learn. For bigger datasets, you may use vowpal wabbit or Fasttext. It may be a little bit slower for the training (but not so much), and as fast as LR for the training. What is the purpose of using an algorithm when another one is just better?
- boto3 9y ago
  There's a reason for that. NB needs to fit more params than LR (and other linear discriminant learners), e.g. for binary classification, NB needs to fit 2*N params while LR just N.
  - imh 9y ago
    Not true. Let's derive it. Here's NB for a binary classification: p(c|w)*p(w) = p(w|c) * p(c) p(c|w) = p(w|c) * p(c) / p(w) p(c|w) = p(w|c) * p(c) / [sum_i p(w|c_i) * p(c_i)] Let's look at the probability of class 1. p(c_1|w) = p(w|c_1) * p(c_1) / [sum_i p(w|c_i) * p(c_i)] Notice how the numerator is going to show up in the denominator. We can simplify that by bringing it into the denominator: p(c_1|w) = 1 / [sum_i p(w|c_i) * p(c_i) / p(w|c_1) / p(c_1] Then cancel it out: p(c_1|w) = 1 / [1 + p(w|c_0) / p(w|c_1) * p(c_0) / p(c_1)] Not let's apply the NB assumptions: p(w|c_0) / p(w|c_1) = prod_i p(w_i | c_0) / p(w_i | c_1) Now, if you take the log of the final p(c_1|w) I derived, the product in p(w|c_0) / p(w|c_1) turns into a exponentiated sum, giving you one parameter per word, plus an intercept for the log of p(c_0) / p(c_1). You end up with exactly the same 1/(1+exp(linear stuff)) with the same parametrization and form you have in logistic regression [0]. This is a broader thing that a given graphical model with a given fixed parametrization can be generatively or discriminatively trained. They will end up learning different models, but that's because they make different assumptions, not because of different parametrizations. [0] linear stuff = intercept + sum_i (coefficient of word_i)
    - abhgh 9y ago
      I may be missing something, but how do you take a log on the RHS of p(c_1|w)? The denominator has a sum i.e. "1+ something", which the log doesn't distribute over.
      - imh 9y ago
        Heh, nope you're right I was writing to quickly. I meant to say to take a log of of the probability term in p(c_1|w). So you take p(c_1|w) = 1/(1+stuff) and turn it into p(c_1|w) = 1/(1+exp(log(stuff))). stuff is a product, so log(stuff) is a sum. Good catch.
        abhgh 9y ago
        Oh, I see it now. Thanks for clarifying! In that case, you needn't have kept the denominator around at all. Since P(w) is not based on a class i.e. its not class conditioned, the classifier could have directly calculated P(C_1|w)/P(C_0|w). The P(w) term cancels out, and you end up with the product of ratios of feature probabilities conditioned on the classes. Note though that, for K-classes, K>2, the number of parameters you need to store would blow up. You would need have these N ratios: P(w_j|C_x)/P(w_j|C_y) for all possible classes x,y. Here N is the number of features. So, in all N*C(K,2) values. On the contrary, multiclass softmax regression (the discriminative analogue for NB) would need NK parameters.
        imh 9y ago
        In the multiclass problem, they have the same number of degrees of parameters. You can see this by choosing a reference class c_y. The NB parameter for class c_x and feature w_j is f_jxy = log(p(w_j|c_x)/p(w_j|c_y)). If you want a new reference class c_w, you can see that f_jxw = f_jxy - f_jwy. No new learned parameters needed. You need one of those parameters per N features and (K-1) classes that aren't c_y. So you get N(K-1) features. This is the same as for multiclass softmax regression: N(K-1) instead of the NK you wrote (which you can see by working it out with a reference class c_y). It really is the same parametrization. The analogue with NK parametrized softmax may be more straightforward if you just use NK naive bayes features of the form f_jx = log(p(w_j|c_x)) for each class and check the equivalence with softmax. It really is just a special case of a more general rule that a given PGM with a fixed parametrization can be trained discriminatively or generatively.
imh 9y ago
The (conditional) independence assumptions it makes are pretty strong and usually inaccurate. In the text example, consider "Steph Curry" and "Warriors." These are going to show up very frequently together, and both provide strong evidence that the topic is sports. However, they don't provide strongly independent evidence of sports. Supposing Curry announced he was making an investment in a local restaurant, the article would probably mention the Warriors just because it's Curry. The problem is that seeing "Warriors" when you see "Steph Curry" shouldn't make you that much more confident that it's sports than just seeing "Steph Curry" alone. Just like seeing "University" with "Stanford" shouldn't change any inferences much from seeing "Stanford" alone. In naive bayes models, these are treated as independent pieces of evidence and that can lead to overconfidence and errors. Naive bayes is a generative model, so in flipping bayes rule around to discriminate classes, you have to be sure your probability model is decent. In a discriminative model, you just go straight to learning the p(class | observations) and have no requirement for a decent model of p(observations). p(observations) is the kind of model that would have to know about "university" and "stanford" being likely to be observed together. Often, it's better to go straight to the discriminative model. In this case, logistic regression.
j2kun 9y ago
Boosted decision trees account for >50% of all Kaggle winners. That is the real surprise, since people rarely talk about boosted decision trees.
- abhgh 9y ago
  Deep Learning is in vogue so I think we see a lot around it written about in blogs, media etc. GBDTs are used in the industry though.
- nl 9y ago
  There is no greater discussion topic on Kaggle Slack than a comparison of XGB vs LightGBM vs CatGBM.
autokad 9y ago
many competition solutions use stacking, where you use the probabilities of several classifiers are used instead of their predictions. Since naive bayes makes an independence assumption, and most competitions have features that are highly correlated, its probability / confidence of its prediction is probably a lot more useless than other classifiers. besides that, nb doesn't do feature selection, regularization, and interactions like many tree based algorithms (such as XGBoost) do out of the box. interactions is a big one for me. With NB, you have to add an interaction feature by multiplying two together, but in creating the feature you must steal some probability away from the other stuff, and finding a good interaction among many features is hard. The tree structure of random forests naturally captures that edit: to your last point, nb does worse on larger data sets. when you dont have a lot of data, you have to make larger assumptions. NB makes a big independence assumption, thus it does well on very small data sets but falls short on large ones
abhgh 9y ago
Accuracy is a problem. NBs are linear classifiers, and more so, the linear boundary is found by assuming features to be independent. So, in cases: (1) data cannot be separated linearly - NBs don't do well for the same reason as any other linear classifier wouldn't work well (2) for linearly separable data - if the feature independence assumption is incompatible with the data, linear classifiers that don't make this assumption will beat it. For ex you can prove that logistic regression beats (or is at least as good as) Naive Bayes asymptotically. [1] [1] See proposition 1 here: "On Discriminative vs Generative classifiers ... Andrew ng, Michael Jordan. https://ai.stanford.edu/~ang/papers/nips01-discriminativegenerative.pdf https://ai.stanford.edu/~ang/papers/nips01-discriminativegen... [PDF]

mooman219 9y ago

The examples other people are using are fairly narrow. I would like to substantiate that text categorization via naive bayes classifier is surprisingly accurate and simple. This paper[1] uses ngrams and a simple out of place measure to compare articles against different verticals and often sees greater than 99% accuracy for relatively small blocks of text. The out of place measure also adds a penalty to features not found in the document, which helps establish the individuality for the category classification. Raw matching performance is also fairly impressive; A less naive implementation is also highly parallelizable. [1] http://odur.let.rug.nl/~vannoord/TextCat/textcat.pdf http://odur.let.rug.nl/~vannoord/TextCat/textcat.pdf

abhgh 9y ago
From experience, I suspect NB does well on text primarily for two reasons: (1) High dimensionality - this is also the reason why SVMs with linear kernels do so well on text (NBs find a linear boundary too - but a different linear boundary than linear-kernel SVMs) (2) this reason applies to certain cases only: text where "token" based estimates are sufficiently discriminatory. For ex take a dataset that has two kinds of documents - one talking about product A and the other talking about product B. And you want to label the documents based on which product they're talking about. Here, just noticing if the term "A" or "B" appears in the document is good enough for classification. You don't have to have powerful models that infer "connotations" of words based on context. NB will do well here, esp if bundled with a feature selection technique that weeds out noisy features (like sthg based on Normalized Mutual Information)

moultano 9y ago

A practical issue for Naive Bayes that also infects linear models is bias w.r.t. document length. Typically when you are detecting a rare, relatively compact class such as sports articles (or spam) you will tend to have a strongly negative prior, many positive features, and few negative ones. As a consequence, as the length of your text increases, not only does the variance of your prediction increase, but the mean tends to as well. This leads to all very long documents being classified as positive, regardless of their text. You can observe this by training your model and then classifying /usr/dict/words. This is the most common mistake I've seen in production use of linear models on document text. Invariably, they'll misfire on any unusually long document.

intune 9y ago
Is there some way to normalize the document length?
- moultano 9y ago
  Lots of reasonable hacks. 1. Use only the beginning of the document, as that's probably the most important part anyways, and it's fast. 2. Divide the sum of your feature scores by sqrt(n) to give it constant variance, and hopefully keep it comparable with your prior. 3. Split the doc into reasonably sized chunks, and average their scores rather than adding them.
  - _dps 9y ago
    I'll add to this that you can add a very crude (separate) model for the document length and number of distinct words, and use that to flag outlier documents that might bump into the known weaknesses with respect to document length.
  - geezerjay 9y ago
    > 1. Use only the beginning of the document, as that's probably the most important part anyways, and it's fast. That seems to be a solution devised for news articles, as the standard news writing style involves providing answers to the Five Ws up front on the article.

mattbettinson 9y ago

https://github.com/bettinson/bayesian-tag-suggestion https://github.com/bettinson/bayesian-tag-suggestion I wrote one of these in Ruby to classify links into tags! Was fun. I think it got me a job.

rgarreta 9y ago

I would add that another practical aspect about Naive Bayes classifiers is that you can make use of the conditional probabilities for each feature that contributes to the predictions. That gives you some introspection on how the model is working and it's useful when "debugging" classifiers by finding features that should/shouldn't be used. https://monkeylearn.com/blog/how-to-create-text-classifiers-machine-learning/#keyword-cloud https://monkeylearn.com/blog/how-to-create-text-classifiers-...

abhgh 9y ago
A side note here: the classification probabilities NB produces are not very accurate and usually need some correction using a process called "calibration".

superasn 9y ago

I created a small program that finds the best sub-reddit given any title text[1] using this algorithm. I'm a total ML noob but it was a interesting project and the results were pretty accurate. I basically used reddit's Bigquery data for the dataset (it's huge!). If you need a practical example of this algo, the algorithm and code is here[2]. [1] https://storage.googleapis.com/superasn/script.html https://storage.googleapis.com/superasn/script.html [2] https://www.reddit.com/r/learnmachinelearning/comments/6hqd6o/p_automatic_reddit_categorizer_update_first/ https://www.reddit.com/r/learnmachinelearning/comments/6hqd6...

drefgert 9y ago

Bayee does a great job of filtering spam but it drives me nuts that obvious spam still appears in my inbox. Suffix trees would fix this in most cases, so why the heck isn't spam filtering using them to remove the obvious spam?

b_ttercup 9y ago

Is Naive Bayes really ever the most practical choice? Yes it is a simple, fast algorithm, but it's usually a non trivial step below other simple models in my experience and doesn't seem to show any major advantages. The results shown here seem good but bag of words models usually do better than you might think on supervised NLP. So what's the motivation?

tyingq 9y ago
I thought it was the typical approach for identifying email spam. Has that changed?
Houshalter 9y ago
The scikit-learn flowchart recommends it for text data with less than 100k samples when linear SVC doesn't work: http://scikit-learn.org/stable/tutorial/machine_learning_map/index.html http://scikit-learn.org/stable/tutorial/machine_learning_map... AFAIK it's by far the fastest machine learning method and one of the only ones that can be learned "online". I.e. it can just update the model each time it gets a datapoint, and then throw it away without saving it for future training. These are nice properties if you are doing something at a very large scale or in an environment with very limited resources. And if your data happens to actually meet the naive bayes assumptions (that all the features are conditionally independent) then it's literally mathematically optimal and you can't do any better than it. It seems to work fairly well even when that isn't the case though.
- phunge 9y ago
  Logistic regression can easily be made online too, keep in mind! sklearn has an implementation of online gradient descent, and vowpal wabbit is also excellent at those problems. Naive bayes can be parallelized in ways that SGD can't, that's a whole other conversation.
  - Houshalter 9y ago
    Gradient descent can be made online. But it's very slow and suffers from catastrophic forgetting. Typical gradient descent needs to iterate over the dataset many times, while naive Bayes only needs one pass.

torbjorn 9y ago

I just did an jupyter notebook on Naive Bayes for Siraj Ravel's Math of Intelligence YouTube course. https://github.com/NoahLidell/math-of-intelligence/blob/master/probability_theory/bayesian-classification.ipynb https://github.com/NoahLidell/math-of-intelligence/blob/mast... I used naive bayes to classify raps from Biggie and 2pac.

anthonysarkis 9y ago

C++ implementation for self driving car (school project) https://github.com/swirlingsand/self-driving-car-nanodegree-nd013/blob/master/p11/naive_bayes_cpp/classifier.cpp https://github.com/swirlingsand/self-driving-car-nanodegree-...