5 ms·

I've wanted to build this for a while, I think an SVM-based spam solution could be amazing. Obviously, like the article mentions, when trying to categorize spam

by dvt 14y ago

I've wanted to build this for a while, I think an SVM-based spam solution could be amazing. Obviously, like the article mentions, when trying to categorize spam, a purely Bayesian approach is not great -- and neither is an ANN (although, with a large enough pool of hidden layers, it can get pretty decent). I think that the issue lies in the problem set. Spam cannot be treated like a linearly-separable model. There are papers[1][2] that outline possible benefits of SVM-based spam filtering. Unfortunately, SVMs are still in their infancy and not many people know how to implement and use them. I do think the are the future, however. [1] http://trec.nist.gov/pubs/trec15/papers/hit.spam.final.final.pdf http://trec.nist.gov/pubs/trec15/papers/hit.spam.final.final... [2] http://classes.soe.ucsc.edu/cmps290c/Spring12/lect/14/00788645-SVMspam.pdf http://classes.soe.ucsc.edu/cmps290c/Spring12/lect/14/007886...

Radim 14y ago
Assuming you mean Support Vector Machines with "SVM", you may be idealizing them a bit. SVMs have been around for almost two decades now, which is an eternity in the ML world, rather than infancy. SVMs don't require the problem set to be linearly separable. Please note that there's a myriad of robust, scalable SVM implementations -- SVMlight, HeroSVM, LIBSVM, liblinear... (the latter two also have wrappers in scikit-learn, a Python library mentioned in the OP).
- notimetorelax 14y ago
  Would only add that unless you're writing a PHD on SVM don't write your own implementation. As Radim wrote there are quite a few to chose from.
- JacobiX 14y ago
  Unfortunately the vast majority of scalable SVM implementations are only for non commercial use.
- dvt 14y ago
  That's exactly my point. Most real-world problems (see spam filtering) are NOT linearly separable. Using highly-granular kernels would yield better results when compared to ANNs and Bayesian methods, etc. Like I said, I haven't worked (much) with SVMs but they really do seem like the future. Unfortunately, they are difficult to work with. And finally, the first workable SVM algorithm was proposed in 1995 (not nearly the two decades you claim) and implemented several years later. SVMs are very much still in their infancy -- especially considering that not much work is being done on SVMs since ANNs are much (much) easier to work with.
TomAnthony 14y ago
I'm interested to know why ANNs wouldn't be a good fit for this? ANNs aren't my area but I have a basic grasp, and I thought they could be an interesting approach to try here?
- dvt 14y ago
  ANNs excel in systems that are linearly separable, but have a much harder time with data that is not. Google the XOR problem, for example.
wfn 14y ago
As Radim noted, there might be better approaches to using SVMs. One example may be Restricted Boltzmann Machines (see Hinton's google tech talk [0]). Some folks have tried using them to detect spam (or at least using RBMs as a part in another architecture), achieving better results than SVMs (they actually did a proper comparison). [1] [2] Might be something worth to be looked at ;) At any rate, RBMs are rather fascinating, I also plan to try experimenting with them when I have time. [0] https://www.youtube.com/watch?v=AyzOUbkUf3M https://www.youtube.com/watch?v=AyzOUbkUf3M [1] For a general overview of the study, here are its slides (pptx): http://users.cs.uoi.gr/~gtzortzi/docs/publications/Deep%20Belief%20Networks%20for%20Spam%20Filtering.pptx http://users.cs.uoi.gr/~gtzortzi/docs/publications/Deep%20Be... [2] The same study in full (pdf): http://users.cs.uoi.gr/~gtzortzi/docs/publications/Deep%20Belief%20Networks%20for%20Spam%20Filtering.pdf http://users.cs.uoi.gr/~gtzortzi/docs/publications/Deep%20Be...
dkersten 14y ago
If I were to build this, I would try deep belief networks. Deep learning can learn the features from a lot of text in an unsupervised manner and then correlate those features to a category with a relatively small training set, which allows it to generalize much better than traditional ANN techniques. Geoff Hinton's claims that DBN's outperform traditional (shallow) ANN's, random forests and SVM's for any of the tasks he has tried them for (including classifying textual documents), though I have not benchmarked how well they work myself (not yet anyway!).
- jsrn 14y ago
  What would you use as input for the deep network? I have only worked with text classification methods where I chose the features myself. As I understand it, a deep network still has (like a 'traditional'/non-deep ANN) a fixed number of inputs in its input layer, i.e. one would have to process each input text somehow before feeding it into the network (to make the input sizes equal). Is there a usual way to do this without doing feature-extraction?
  - textminer 14y ago
    If using raw bag of words or n-grams, why not hash the strings to maybe 2^13-1 slots, with something like MurmurHash3 or with multiple hashes to prevent collisions, then use that sparse vector as input to a deep learning model?
    - jsrn 14y ago
      Thanks, that makes sense. So the parameter would be the number of slots (== number of input units of the deep NN). And the transformation of the text into bag-of-words / n-grams would not be considered feature-engineering - or at least only 'low level feature engineering' - the higher level features will be learned by the deep network. I guess one could go lower level still and even do away with bag-of-words / n-grams : limit the text size to e.g. 20000 characters, represent each character value with a numerical value (e.g. its ASCII code point when dealing with mainly English texts) and then simply feed this vector of codepoints to the input layer of the deep network. Given enough input data, it should learn location-invariant representations like bag-of-words / n-grams (or even better ones) itself, right?

by dvt 14y ago

Radim 14y ago

Assuming you mean Support Vector Machines with "SVM", you may be idealizing them a bit. SVMs have been around for almost two decades now, which is an eternity in the ML world, rather than infancy. SVMs don't require the problem set to be linearly separable. Please note that there's a myriad of robust, scalable SVM implementations -- SVMlight, HeroSVM, LIBSVM, liblinear... (the latter two also have wrappers in scikit-learn, a Python library mentioned in the OP).

notimetorelax 14y ago
Would only add that unless you're writing a PHD on SVM don't write your own implementation. As Radim wrote there are quite a few to chose from.
JacobiX 14y ago
Unfortunately the vast majority of scalable SVM implementations are only for non commercial use.
dvt 14y ago
That's exactly my point. Most real-world problems (see spam filtering) are NOT linearly separable. Using highly-granular kernels would yield better results when compared to ANNs and Bayesian methods, etc. Like I said, I haven't worked (much) with SVMs but they really do seem like the future. Unfortunately, they are difficult to work with. And finally, the first workable SVM algorithm was proposed in 1995 (not nearly the two decades you claim) and implemented several years later. SVMs are very much still in their infancy -- especially considering that not much work is being done on SVMs since ANNs are much (much) easier to work with.

TomAnthony 14y ago

I'm interested to know why ANNs wouldn't be a good fit for this? ANNs aren't my area but I have a basic grasp, and I thought they could be an interesting approach to try here?

dvt 14y ago
ANNs excel in systems that are linearly separable, but have a much harder time with data that is not. Google the XOR problem, for example.

wfn 14y ago

As Radim noted, there might be better approaches to using SVMs. One example may be Restricted Boltzmann Machines (see Hinton's google tech talk [0]). Some folks have tried using them to detect spam (or at least using RBMs as a part in another architecture), achieving better results than SVMs (they actually did a proper comparison). [1] [2] Might be something worth to be looked at ;) At any rate, RBMs are rather fascinating, I also plan to try experimenting with them when I have time. [0] https://www.youtube.com/watch?v=AyzOUbkUf3M https://www.youtube.com/watch?v=AyzOUbkUf3M [1] For a general overview of the study, here are its slides (pptx): http://users.cs.uoi.gr/~gtzortzi/docs/publications/Deep%20Belief%20Networks%20for%20Spam%20Filtering.pptx http://users.cs.uoi.gr/~gtzortzi/docs/publications/Deep%20Be... [2] The same study in full (pdf): http://users.cs.uoi.gr/~gtzortzi/docs/publications/Deep%20Belief%20Networks%20for%20Spam%20Filtering.pdf http://users.cs.uoi.gr/~gtzortzi/docs/publications/Deep%20Be...

dkersten 14y ago

If I were to build this, I would try deep belief networks. Deep learning can learn the features from a lot of text in an unsupervised manner and then correlate those features to a category with a relatively small training set, which allows it to generalize much better than traditional ANN techniques. Geoff Hinton's claims that DBN's outperform traditional (shallow) ANN's, random forests and SVM's for any of the tasks he has tried them for (including classifying textual documents), though I have not benchmarked how well they work myself (not yet anyway!).

jsrn 14y ago
What would you use as input for the deep network? I have only worked with text classification methods where I chose the features myself. As I understand it, a deep network still has (like a 'traditional'/non-deep ANN) a fixed number of inputs in its input layer, i.e. one would have to process each input text somehow before feeding it into the network (to make the input sizes equal). Is there a usual way to do this without doing feature-extraction?
- textminer 14y ago
  If using raw bag of words or n-grams, why not hash the strings to maybe 2^13-1 slots, with something like MurmurHash3 or with multiple hashes to prevent collisions, then use that sparse vector as input to a deep learning model?
  - jsrn 14y ago
    Thanks, that makes sense. So the parameter would be the number of slots (== number of input units of the deep NN). And the transformation of the text into bag-of-words / n-grams would not be considered feature-engineering - or at least only 'low level feature engineering' - the higher level features will be learned by the deep network. I guess one could go lower level still and even do away with bag-of-words / n-grams : limit the text size to e.g. 20000 characters, represent each character value with a numerical value (e.g. its ASCII code point when dealing with mainly English texts) and then simply feed this vector of codepoints to the input layer of the deep network. Given enough input data, it should learn location-invariant representations like bag-of-words / n-grams (or even better ones) itself, right?