17 ms·

Boosting was, I suppose, invented by Schapire and Freund, both CS guys, but its most successful form, gradient boosting was invented by Friedman, a stat profess

by dthal 12y ago

Boosting was, I suppose, invented by Schapire and Freund, both CS guys, but its most successful form, gradient boosting was invented by Friedman, a stat professor. Random forests were developed by Breiman and Cutler, both stats professors. SVM's probably weren't developed in statistics because they don't have a probability model.

mturmon 12y ago
It sounds like you're challenging my central claim: conventional statistics has largely ignored the impact of computational technologies on inference, and dismissed the resulting failure as a mere marketing problem. You are wrong. Boosting is a generic technique, and reducing it to gradient boosting and then claiming it as "mainstream statistics" is silly. Boosting is not, even today, part of the standard statistics curriculum, despite its wide utility. But the bootstrap is. I wonder why? Describing Leo Breiman as a stats professor is similarly twisted. He was always an outlier in traditional statistics; in fact, he left a position in the math department at UCLA because he didn't fit in, and began independent consulting, during which time he developed CART. And during his time, years later, as a bona fide stats professor at UC Berkeley, he spent considerable effort overcoming a tendency of mainstream stats to ignore the implications of computing. For instance, he was one of a small number of stats professors who was active in the NIPS community. (His bio at http://projecteuclid.org/download/pdf_1/euclid.ss/1009213290 http://projecteuclid.org/download/pdf_1/euclid.ss/1009213290 is absolutely fascinating, see pages 195-6 for his feelings about conventional statistics.) And don't forget that a lot of the popularity of CART was due to the C4.5 implementation, due to Ross Quinlan, a CS professor with a physics background, which allowed easy building of classifiers on top of trees. Which points out another thing, that these advances have in many cases not come from, strictly speaking, computer science; physicists and engineers have been very influential. SVM's were developed by Vladimir Vapnik and collaborators, who came out of a Russian math/physics background. Who cares whether SVMs have a model or not? This is an example of the myopia that afflicts old-fashioned statistics. SVMs are highly effective predictors with deep mathematical properties and admit a statistical analysis of their performance and properties. (You don't think every statistician who uses logistic regression really believes this is a model for the data generation process! Yet logistic regression is part of the stats canon.) The SVM analysis ("large margin classifiers") largely took place, again, outside mainstream statistics. We could say the same thing about neural networks. And we could say the same thing about deep learning. NNs: 1980s-1990s. SVMs: 1990s-2000s. Deep learning: 2000s-2010s. In ten years, we'll be adding another example to the list, because this myopia is not just poor marketing skills.

by dthal 12y ago

mturmon 12y ago

It sounds like you're challenging my central claim: conventional statistics has largely ignored the impact of computational technologies on inference, and dismissed the resulting failure as a mere marketing problem. You are wrong. Boosting is a generic technique, and reducing it to gradient boosting and then claiming it as "mainstream statistics" is silly. Boosting is not, even today, part of the standard statistics curriculum, despite its wide utility. But the bootstrap is. I wonder why? Describing Leo Breiman as a stats professor is similarly twisted. He was always an outlier in traditional statistics; in fact, he left a position in the math department at UCLA because he didn't fit in, and began independent consulting, during which time he developed CART. And during his time, years later, as a bona fide stats professor at UC Berkeley, he spent considerable effort overcoming a tendency of mainstream stats to ignore the implications of computing. For instance, he was one of a small number of stats professors who was active in the NIPS community. (His bio at http://projecteuclid.org/download/pdf_1/euclid.ss/1009213290 http://projecteuclid.org/download/pdf_1/euclid.ss/1009213290 is absolutely fascinating, see pages 195-6 for his feelings about conventional statistics.) And don't forget that a lot of the popularity of CART was due to the C4.5 implementation, due to Ross Quinlan, a CS professor with a physics background, which allowed easy building of classifiers on top of trees. Which points out another thing, that these advances have in many cases not come from, strictly speaking, computer science; physicists and engineers have been very influential. SVM's were developed by Vladimir Vapnik and collaborators, who came out of a Russian math/physics background. Who cares whether SVMs have a model or not? This is an example of the myopia that afflicts old-fashioned statistics. SVMs are highly effective predictors with deep mathematical properties and admit a statistical analysis of their performance and properties. (You don't think every statistician who uses logistic regression really believes this is a model for the data generation process! Yet logistic regression is part of the stats canon.) The SVM analysis ("large margin classifiers") largely took place, again, outside mainstream statistics. We could say the same thing about neural networks. And we could say the same thing about deep learning. NNs: 1980s-1990s. SVMs: 1990s-2000s. Deep learning: 2000s-2010s. In ten years, we'll be adding another example to the list, because this myopia is not just poor marketing skills.