3 ms·

_up we evaluated thoroughly perceptron which are somewhat close to Bayesian networks. Basically a perceptron is a one layer NN and is therefore quite similar to

by spolu 12y ago

_up we evaluated thoroughly perceptron which are somewhat close to Bayesian networks. Basically a perceptron is a one layer NN and is therefore quite similar to a bayesian network in the fact that it encodes a linear regression. That being said... studying bayesian networks more thoroughly might raise better results indeed. Don't know though if Gmail is using bayesian networks or deep learning?

tensor 12y ago
My guess is that gmail is using a linear classifier. Both because of the scale of the data, and because up until very recently linear classifiers have been state of the art on text classification. In the few cases where NN have achieved new state of the art on text, the stanford sentiment analysis work and a few more recent works, a full sentence parse is needed. Sentence parses do achieve 95% accuracy, but only on well structured text in a given domain. Plus, they are hugely time intensive compared to the large scale linear classifiers like vowpal wabbit or sophia-ml. Regarding perceptrons, a basic perceptron, although it is a linear classifier, will not achieve state of the art. Averaged perceptrons get you closer, but what you really want is a discriminatively trained linear classifier with regularization. If I had to bet, gmail is probably using something closer to https://code.google.com/p/sofia-ml/ https://code.google.com/p/sofia-ml/ than a NN. Maybe a Googler will surprise me though!
- wamatt 12y ago
  > My guess is that gmail is using a linear classifier. Yup, the Google "Priority Inbox" feature does indeed use a linear classifier, in particular logistic regression [1] for the reason of scale as you point out. Also, IIRC Gmail's original spam detection used naive bayes. It may have evolved since then. [1] http://static.googleusercontent.com/media/research.google.com/en/us/pubs/archive/36955.pdf http://static.googleusercontent.com/media/research.google.co...
brandonb 12y ago
I don't know about GMail specifically, but I suspect your approach of using neural networks is the best one given the state-of-the-art in machine learning today. You can think of naive bayes and a perceptron as roughly equivalent in terms of expressiveness--they're both linear models--but a perceptron is usually better since it can account for correlations between input variables. As you say, a perceptron is a one-layer neural network, so with a large enough training set, a multi-layer neural network will almost certainly perform better since it can recognize combinations of features that work well together. Bayesian filtering for spam detection is a good starting point since it's easy to implement, and was very popular in the mid-2000s, but with all the advancements in deep learning since 2006, I'd almost certainly bet on a neural network these days.
- pedrosorio 12y ago
  Arguing that using a neural network instead of a simpler classifier is a good idea because of recent developments in deep learning is like arguing that driving a car from the 50s is safer than taking a train because of recent developments in airbag/cruise control technology.

by spolu 12y ago

tensor 12y ago

My guess is that gmail is using a linear classifier. Both because of the scale of the data, and because up until very recently linear classifiers have been state of the art on text classification. In the few cases where NN have achieved new state of the art on text, the stanford sentiment analysis work and a few more recent works, a full sentence parse is needed. Sentence parses do achieve 95% accuracy, but only on well structured text in a given domain. Plus, they are hugely time intensive compared to the large scale linear classifiers like vowpal wabbit or sophia-ml. Regarding perceptrons, a basic perceptron, although it is a linear classifier, will not achieve state of the art. Averaged perceptrons get you closer, but what you really want is a discriminatively trained linear classifier with regularization. If I had to bet, gmail is probably using something closer to https://code.google.com/p/sofia-ml/ https://code.google.com/p/sofia-ml/ than a NN. Maybe a Googler will surprise me though!

wamatt 12y ago
> My guess is that gmail is using a linear classifier. Yup, the Google "Priority Inbox" feature does indeed use a linear classifier, in particular logistic regression [1] for the reason of scale as you point out. Also, IIRC Gmail's original spam detection used naive bayes. It may have evolved since then. [1] http://static.googleusercontent.com/media/research.google.com/en/us/pubs/archive/36955.pdf http://static.googleusercontent.com/media/research.google.co...

brandonb 12y ago

I don't know about GMail specifically, but I suspect your approach of using neural networks is the best one given the state-of-the-art in machine learning today. You can think of naive bayes and a perceptron as roughly equivalent in terms of expressiveness--they're both linear models--but a perceptron is usually better since it can account for correlations between input variables. As you say, a perceptron is a one-layer neural network, so with a large enough training set, a multi-layer neural network will almost certainly perform better since it can recognize combinations of features that work well together. Bayesian filtering for spam detection is a good starting point since it's easy to implement, and was very popular in the mid-2000s, but with all the advancements in deep learning since 2006, I'd almost certainly bet on a neural network these days.

pedrosorio 12y ago
Arguing that using a neural network instead of a simpler classifier is a good idea because of recent developments in deep learning is like arguing that driving a car from the 50s is safer than taking a train because of recent developments in airbag/cruise control technology.