29 ms·

Any commentary on why would anyone consider this paper of interest today? The peculiarities of the least squares fit are no secret, the assumptions of the "gene

by stiff 12y ago

Any commentary on why would anyone consider this paper of interest today? The peculiarities of the least squares fit are no secret, the assumptions of the "generalizability of fit theorem" presented are highly unrealistic (compares equal weights to weights chosen uniformly at random), and we are now living in the age of things like Support Vector Machines, Structural risk minimization, VC dimension, boosting and so forth, which all address the issue of overfitting very well. This reads like prehistory of the field, and not very significant at that.

yummyfajitas 12y ago
The paper is interesting because it is a highly readable and extremely elementary introduction to the topic. Most HN readers don't even know the prehistory of the field, so I think it's useful to them.
- stiff 12y ago
  The rudiments of modern theory can be understood with little more difficulty, and it is clear from those that any significant restriction of the number of degrees of freedom of a model reduces the chances of overfitting occurring, but also decreases the fraction of predicitions the model will get right, so the real issue is where exactly do we draw a line, and this is now understood quite well - the approach from the paper, for practical purposes, throws out the baby with the bathwater. The first few lectures of the course of machine learning by Yaser Abu-Mostafa are a really engaging introduction to those topics: https://work.caltech.edu/ https://work.caltech.edu/ By the way Howard Wainer is a noted author of semi-popular (some formulas actually appear etc.) statistics books, so if enjoyed the writing, maybe a better use of time would be to read his newer and more general stuff: http://www.amazon.com/Howard-Wainer/e/B000AP7SUU/ref=sr_ntt_srch_lnk_1?qid=1401098828&sr=8-1 http://www.amazon.com/Howard-Wainer/e/B000AP7SUU/ref=sr_ntt_...
michaelochurch 12y ago
It is interesting that an obviously "wrong" approach can have more success than expected. However, most people these days would use more modern forms of regularization (e.g. ridge, Lasso, elastic net, stagewise with early stopping). Modern optimization software can handle it. As I noted in my other comment, equal-weight constraints are still used in one domain: neural network architectures (e.g. convolutional neural nets). This is not quite the same thing because neural nets are so highly parameterized, and the weights interact in so many nonlinear ways, that individual weights don't have any meaning independent of others. It does show us that the idea of equal-weight regression has still lived on in one specific place.
- nkurz 12y ago
  Equal weights are also used in creating ensemble averages. It's always bothered me that the IPCC predictions for climate are done by averaging the results across models of highly varying quality, with each one weighted equally. Instinctively, I'd think you'd want to put greater trust in the 'better quality' models, but perhaps equal weighting is more defensible than it appears.
nkurz 12y ago
I think there may be more substance to the concept of equal weighting than there appears to be at first glance. I came across this paper via a blog post from Andrew Gelman. In that post he endorses the conclusions of a 2013 paper that reaches a similar seemingly-implausible conclusion: http://andrewgelman.com/2013/08/14/the-robust-beauty-of-improper-linear-models-in-decision-making/ http://andrewgelman.com/2013/08/14/the-robust-beauty-of-impr.... That lead me to a post by John Cook on the same topic: http://www.johndcook.com/blog/2013/03/05/robustness-of-equal-weights/ http://www.johndcook.com/blog/2013/03/05/robustness-of-equal..., where this paper was mentioned in the comments. The Dawes "The Robust Beauty of Improper Linear Models in Decision Making" paper is here: http://heatherlench.com/wp-content/uploads/2008/07/dawes2.pdf http://heatherlench.com/wp-content/uploads/2008/07/dawes2.pd...

by stiff 12y ago

yummyfajitas 12y ago

The paper is interesting because it is a highly readable and extremely elementary introduction to the topic. Most HN readers don't even know the prehistory of the field, so I think it's useful to them.

stiff 12y ago
The rudiments of modern theory can be understood with little more difficulty, and it is clear from those that any significant restriction of the number of degrees of freedom of a model reduces the chances of overfitting occurring, but also decreases the fraction of predicitions the model will get right, so the real issue is where exactly do we draw a line, and this is now understood quite well - the approach from the paper, for practical purposes, throws out the baby with the bathwater. The first few lectures of the course of machine learning by Yaser Abu-Mostafa are a really engaging introduction to those topics: https://work.caltech.edu/ https://work.caltech.edu/ By the way Howard Wainer is a noted author of semi-popular (some formulas actually appear etc.) statistics books, so if enjoyed the writing, maybe a better use of time would be to read his newer and more general stuff: http://www.amazon.com/Howard-Wainer/e/B000AP7SUU/ref=sr_ntt_srch_lnk_1?qid=1401098828&sr=8-1 http://www.amazon.com/Howard-Wainer/e/B000AP7SUU/ref=sr_ntt_...

michaelochurch 12y ago

It is interesting that an obviously "wrong" approach can have more success than expected. However, most people these days would use more modern forms of regularization (e.g. ridge, Lasso, elastic net, stagewise with early stopping). Modern optimization software can handle it. As I noted in my other comment, equal-weight constraints are still used in one domain: neural network architectures (e.g. convolutional neural nets). This is not quite the same thing because neural nets are so highly parameterized, and the weights interact in so many nonlinear ways, that individual weights don't have any meaning independent of others. It does show us that the idea of equal-weight regression has still lived on in one specific place.

nkurz 12y ago
Equal weights are also used in creating ensemble averages. It's always bothered me that the IPCC predictions for climate are done by averaging the results across models of highly varying quality, with each one weighted equally. Instinctively, I'd think you'd want to put greater trust in the 'better quality' models, but perhaps equal weighting is more defensible than it appears.

nkurz 12y ago

I think there may be more substance to the concept of equal weighting than there appears to be at first glance. I came across this paper via a blog post from Andrew Gelman. In that post he endorses the conclusions of a 2013 paper that reaches a similar seemingly-implausible conclusion: http://andrewgelman.com/2013/08/14/the-robust-beauty-of-improper-linear-models-in-decision-making/ http://andrewgelman.com/2013/08/14/the-robust-beauty-of-impr.... That lead me to a post by John Cook on the same topic: http://www.johndcook.com/blog/2013/03/05/robustness-of-equal-weights/ http://www.johndcook.com/blog/2013/03/05/robustness-of-equal..., where this paper was mentioned in the comments. The Dawes "The Robust Beauty of Improper Linear Models in Decision Making" paper is here: http://heatherlench.com/wp-content/uploads/2008/07/dawes2.pdf http://heatherlench.com/wp-content/uploads/2008/07/dawes2.pd...