4 ms·

There is great, big data driven research coming out of Stanford using Common Crawl. For example, see http://www-nlp.stanford.edu/projects/glove/ http://www-nlp

There is great, big data driven research coming out of Stanford using Common Crawl. For example, see http://www-nlp.stanford.edu/projects/glove/ http://www-nlp.stanford.edu/projects/glove/ . They successfully train an 840 billion token corpus. Vapnik is a big theory guy. Though I am not sure he has done anything of big practical importance recently, his immense contribution to ML (the SVM) was done at a time when machines were many orders of magnitudes weaker than they are now.

mturmon 12y ago
"In writing this book I had one more goal in mind: I wanted to stress the practical power of abstract reasoning. The point is that during the last few years at different computer science conferences, I heard reiteration of the following claim: Complex theories do not work, simple algorithms do. "One of the goals of this book is to show that, at least in the problems of statistical inference, this is not true. I would like to demonstrate that in this area of science a good old principle is valid: Nothing is more practical than a good theory. -- From Vapnik's preface to The Nature of Statistical Learning Theory * Vapnik is not well-described as a "theory guy". That implies that he's not interested in connections between theory and practice, and this is most profoundly not the case. He has arguably been the most successful ML researcher ever as far as connecting abstract theory to real-world outcomes. Besides the SVM: the VC dimension started out as a lemma regarding set counting, and he pushed it to the surprising (even shocking) conclusion of universal consistency for very general classes of estimators.
- robrenaud 12y ago
  I guess it depends on what kind of semantics you apply to "theory guy". In my mind it's not at all dismissive. I mean it in it a foundation sense, rather than an applications sense. He has done great work with a whiteboard and pure thought, without the need for terabytes of data and thousands of machines.
  - mturmon 12y ago
    Remember, though, that the AT&T group Vladimir came from, and that informed his work, was much in the mold of linking theory and practice. Where "practice" (at that time) was working on the handwritten digit problem -- the now-cliche NIST dataset.
nl 12y ago
There is great, big data driven research coming out of Stanford using Common Crawl. For example, see http://www-nlp.stanford.edu/projects/glove/ http://www-nlp.stanford.edu/projects/glove/ . They successfully train an 840 billion token corpus. I haven't seen this paper before (thanks!!). How different is it to Word2Vec? Clearly the pre-trained vectors at that scale (and much bigger than the ones released with Word2Vec) are new and very exciting.
- Smerity 12y ago
  The paper compares in detail against word2vec, but (spoiler alert) GloVe using 42 billion tokens from Common Crawl beats word2vec using 100 billion tokens from the Google News corpus! They don't actually use the 840 billion token model in the paper as it was made with some parameters that didn't allow for direct comparison, but the code and the models are all released for anyone to use from their site. This is one of many great examples of open datasets like Common Crawl allowing talented people from academia and start-ups to compete with the large proprietary datasets of Google or Bing. (disclaimer: data scientist at Common Crawl who does the crawling)
  - nl 12y ago
    The paper compares in detail against word2vec, but (spoiler alert) GloVe using 42 billion tokens from Common Crawl beats word2vec using 100 billion tokens from the Google News corpus! Damn!! Background for those who don't follow this field: Word2Vec is an apparently miraculous demonstration and poster-child of the unreasonable effectiveness of big data. Beating it at all is impressive, assuming the performance is as robust as Word2Vec is against different metrics. Beating it with only 42% of the tokens is wondrous.
  - ajtulloch 12y ago
    The GloVe vs word2vec is not as clear-cut as that - see https://plus.google.com/114479713299850783539/posts/BYvhAbgG8T2 https://plus.google.com/114479713299850783539/posts/BYvhAbgG..., https://docs.google.com/document/d/1ydIujJ7ETSZ688RGfU5IMJJsbxAi-kRl8czSwpti15s/ https://docs.google.com/document/d/1ydIujJ7ETSZ688RGfU5IMJJs... for more discussion.

mturmon 12y ago

"In writing this book I had one more goal in mind: I wanted to stress the practical power of abstract reasoning. The point is that during the last few years at different computer science conferences, I heard reiteration of the following claim: Complex theories do not work, simple algorithms do. "One of the goals of this book is to show that, at least in the problems of statistical inference, this is not true. I would like to demonstrate that in this area of science a good old principle is valid: Nothing is more practical than a good theory. -- From Vapnik's preface to The Nature of Statistical Learning Theory * Vapnik is not well-described as a "theory guy". That implies that he's not interested in connections between theory and practice, and this is most profoundly not the case. He has arguably been the most successful ML researcher ever as far as connecting abstract theory to real-world outcomes. Besides the SVM: the VC dimension started out as a lemma regarding set counting, and he pushed it to the surprising (even shocking) conclusion of universal consistency for very general classes of estimators.

robrenaud 12y ago
I guess it depends on what kind of semantics you apply to "theory guy". In my mind it's not at all dismissive. I mean it in it a foundation sense, rather than an applications sense. He has done great work with a whiteboard and pure thought, without the need for terabytes of data and thousands of machines.
- mturmon 12y ago
  Remember, though, that the AT&T group Vladimir came from, and that informed his work, was much in the mold of linking theory and practice. Where "practice" (at that time) was working on the handwritten digit problem -- the now-cliche NIST dataset.

nl 12y ago

There is great, big data driven research coming out of Stanford using Common Crawl. For example, see http://www-nlp.stanford.edu/projects/glove/ http://www-nlp.stanford.edu/projects/glove/ . They successfully train an 840 billion token corpus. I haven't seen this paper before (thanks!!). How different is it to Word2Vec? Clearly the pre-trained vectors at that scale (and much bigger than the ones released with Word2Vec) are new and very exciting.

Smerity 12y ago
The paper compares in detail against word2vec, but (spoiler alert) GloVe using 42 billion tokens from Common Crawl beats word2vec using 100 billion tokens from the Google News corpus! They don't actually use the 840 billion token model in the paper as it was made with some parameters that didn't allow for direct comparison, but the code and the models are all released for anyone to use from their site. This is one of many great examples of open datasets like Common Crawl allowing talented people from academia and start-ups to compete with the large proprietary datasets of Google or Bing. (disclaimer: data scientist at Common Crawl who does the crawling)
- nl 12y ago
  The paper compares in detail against word2vec, but (spoiler alert) GloVe using 42 billion tokens from Common Crawl beats word2vec using 100 billion tokens from the Google News corpus! Damn!! Background for those who don't follow this field: Word2Vec is an apparently miraculous demonstration and poster-child of the unreasonable effectiveness of big data. Beating it at all is impressive, assuming the performance is as robust as Word2Vec is against different metrics. Beating it with only 42% of the tokens is wondrous.
- ajtulloch 12y ago
  The GloVe vs word2vec is not as clear-cut as that - see https://plus.google.com/114479713299850783539/posts/BYvhAbgG8T2 https://plus.google.com/114479713299850783539/posts/BYvhAbgG..., https://docs.google.com/document/d/1ydIujJ7ETSZ688RGfU5IMJJsbxAi-kRl8czSwpti15s/ https://docs.google.com/document/d/1ydIujJ7ETSZ688RGfU5IMJJs... for more discussion.