4 ms·

I've been looking into Stein's paradox last year more deeply as I could not wrap my head around it. The baseball example is easy. It is quite intuitive that som

by Hermel 11y ago

I've been looking into Stein's paradox last year more deeply as I could not wrap my head around it. The baseball example is easy. It is quite intuitive that some of those who scored well probably just did so by accident, and are actually somewhat less skilled (and vice versa for the lowest scorers). However, the Stein paradox is deeper than that. According to Wikipedia, it apparently also applies to unrelated variables, for example the population of Ulan Bator, the temperature on Mars, and the yearly chocolate consumption of Switzerland. This goes counter all my intuition and I found the following weakness in the theory behind Stein's paradox: the improved estimators all seem to depend on things like the mean or the variance among the included variables. For example, if you estimate Ulan Bator to have 1 million inhabitants and the temperature on Mars to be 200 Kelvin, you would adjust the former estimate a little downwards and the latter estimate a little upwards (towards the common mean). However, this implicitely assumes that the population and the temperature have been drawn from a distribution whose mean exists. My guess, that this is not the case. Obviously, you can always calculate a sample mean and a sample variance, but they might be meaningless if the sample stems from a distribution such as Cauchy.

hyperbovine 11y ago
The "correct" way to understand this phenomenon is via regression, in which case it boils down to the simple fact that the regression lines E(θ|X) and E(X|θ) are different. (This "Galtonian perspective" is the subject of one of my favorite papers of all time, http://projecteuclid.org/euclid.ss/euclid.ss/1177012274 http://projecteuclid.org/euclid.ss/euclid.ss/1177012274). This is also the only intuitive explanation I am aware of which explains why Stein's phenomenon only occurs in dimensions three and higher.
tedsanders 11y ago
Thanks for the elaboration. One thing I don't understand is how quantities with different units can be compared. For instance, you give the example of 1,000,000 people living in Ulan Bator and 200 Kelvin being the temperature on Mars. If you follow this procedure and nudge them toward one another (bringing 1,000,000 down and 200 up), then you're supposed to end up with a better predictor. But what if our units had been millions of people and milliKelvins. Then our quantities would have been 1 and 200,000, respectively. The procedure would have us nudge our estimates in the opposite directions. And that surely wouldn't also improve our estimates, right? Clearly I'm misunderstanding something, so I'm going to read some of these papers. Edit: It seems from the Galtonian perspective paper, all the distributions are assumed to have a constant standard deviation. So perhaps we shouldn't be measuring these quantities in terms of people or milliKelvins, but rather in terms of standard deviations? E.g., the mean is +4 standard deviations above 0?

by Hermel 11y ago

hyperbovine 11y ago

The "correct" way to understand this phenomenon is via regression, in which case it boils down to the simple fact that the regression lines E(θ|X) and E(X|θ) are different. (This "Galtonian perspective" is the subject of one of my favorite papers of all time, http://projecteuclid.org/euclid.ss/euclid.ss/1177012274 http://projecteuclid.org/euclid.ss/euclid.ss/1177012274). This is also the only intuitive explanation I am aware of which explains why Stein's phenomenon only occurs in dimensions three and higher.

tedsanders 11y ago

Thanks for the elaboration. One thing I don't understand is how quantities with different units can be compared. For instance, you give the example of 1,000,000 people living in Ulan Bator and 200 Kelvin being the temperature on Mars. If you follow this procedure and nudge them toward one another (bringing 1,000,000 down and 200 up), then you're supposed to end up with a better predictor. But what if our units had been millions of people and milliKelvins. Then our quantities would have been 1 and 200,000, respectively. The procedure would have us nudge our estimates in the opposite directions. And that surely wouldn't also improve our estimates, right? Clearly I'm misunderstanding something, so I'm going to read some of these papers. Edit: It seems from the Galtonian perspective paper, all the distributions are assumed to have a constant standard deviation. So perhaps we shouldn't be measuring these quantities in terms of people or milliKelvins, but rather in terms of standard deviations? E.g., the mean is +4 standard deviations above 0?