6 ms·

Understanding Stein's Paradox (2021)

103 points by robertvc 2y ago

TibbityFlanders 2y ago
I'm horrible at stats, but is this saying that if I have 5 jars of pennies, and I guess the amount in each one. Then I find the average of all my guesses, and the variance between the guesses, then I can adjust each guess to a more likely answer with this method?
- eru 2y ago
  No, I don't think these problems are related.
- kgwgk 2y ago
  Not necessarily "more likely" but "better" in some "loss" sense. It could be "more likely" in the jars example where estimates may convey some relevant information for each other. But consider this example from wikipedia: "Suppose we are to estimate three unrelated parameters, such as the US wheat yield for 1993, the number of spectators at the Wimbledon tennis tournament in 2001, and the weight of a randomly chosen candy bar from the supermarket. Suppose we have independent Gaussian measurements of each of these quantities. Stein's example now tells us that we can get a better estimate (on average) for the vector of three parameters by simultaneously using the three unrelated measurements." https://en.wikipedia.org/wiki/Stein%27s_example#Example https://en.wikipedia.org/wiki/Stein%27s_example#Example
mitthrowaway2 2y ago
Sorry, I'm siding with the physicists here. If you're going to declare that your seemingly arbitrary choice of coordinate system is actually not arbitrary and part of your prior information about where the mean of the distribution is suspected to be, you have to put that in the initial problem statement.
- kgwgk 2y ago
  There is nothing magical about the origin, the shrinkage can be done towards any point and in fact when estimating multiple means it's customary to move each point closer to their average. https://www.math.drexel.edu/~tolya/EfronMorris.pdf https://www.math.drexel.edu/~tolya/EfronMorris.pdf
  - mitthrowaway2 2y ago
    There is something magical about the origin when the result does not respect translational symmetry. In fact, in a real world setting I would probably use my first measurement to define the origin, having no other reference to reach for.
    - kgwgk 2y ago
      What does not respect translational symmetry? You have an estimator. If you apply shrinkage towards the origin you have another estimator. If you apply shrinkage towards [42, 42, ..., 42] you have yet another estimator. Etc. Is it a problem that different estimators produce different results?
      - mitthrowaway2 2y ago
        The James-Stein estimator does not respect translational symmetry. If I do a change of variables x2 = (x - offset), for an arbitrary offset, it gives me a different result! Whereas an estimator that just says I should guess that the mean is x, is unaffected by a change of coordinate system. This is a big problem if the coordinate system itself is not intended to contain information about the location of the mean. This makes sense if "zero" is physically meaningful, for example if negative values are not allowed in the problem domain (number of spectators at Wimbledon stadium, etc). Although in that case, my distribution probably shouldn't be Gaussian!
        kgwgk 2y ago
        This is what the original paper from Stein says: "We choose an arbitrary point in the sample space independent of the outcome of the experiment and call it the origin. Of course, in the way we have expressed the problem this choice has already been made, but in a correct coordinate-free presentation, it would appear as an arbitrary choice of one point in an affine space." The James-Stein estimator in its general form is about shrinking towards an arbitrary point (which usually is not the origin). It respects translational symmetry if you transform that arbitrary point like everything else.
pfortuny 2y ago
I do not get it: if variance is too large, a random sample is very little representative of the mean. As simple as that? Now the specific formula may be complicated. But otherwise I do not understand the “paradox”? Or am I missing something?
- hcks 2y ago
  In the 1D case the single point will be the best estimator for the mean no matter what the variance is
  - pfortuny 2y ago
    OK, this is a good reply, and very informative, many thanks. Now I understand why the thing is paradoxical.
credit_guy 2y ago
Stein's paradox is bogus. Somebody needs to say that. Here's one wikipedia example: > Suppose we are to estimate three unrelated parameters, such as the US wheat yield for 1993, the number of spectators at the Wimbledon tennis tournament in 2001, and the weight of a randomly chosen candy bar from the supermarket. Suppose we have independent Gaussian measurements of each of these quantities. Stein's example now tells us that we can get a better estimate (on average) for the vector of three parameters by simultaneously using the three unrelated measurements. Here's what's bogus about this: the "better estimate (on average)" is mathematically true ... for a certain definition of "better estimate". But whatever that definition is, it is irrelevant to the real world. If you believe you get a better estimate of the US wheat yield by estimating also the number of Wimbledon spectators and the weight of a candy bar in a shop, then you probably believe in telepathy and astrology too.
- wavemode 2y ago
  (Disclaimer, stats noob here) - I thought the point was that, you have a better chance of being -overall- closer to the mean (i.e., the 3D euclidean distance between your guess and the mean would be the smallest, on average), even though you may not necessarily have improved your odds of guessing any of the single individual means. So it's not that "you get a better estimate of the US wheat yield by estimating also the number of Wimbledon spectators and the weight of a candy bar in a shop", it's simply that you get a better estimate for the combined vector of the three means. (Which, in this case, the vector of the three means is probably meaningless, since the three data sets are entirely unrelated. But we could also imagine scenarios where that vector is meaningful.) Am I misunderstanding something?
  - credit_guy 2y ago
    You are most likely right. I am personally bothered by the way it is presented as a "paradox", with the implication that it would have real world applications. I have zero doubts that you can't improve the estimate of the US wheat yields by looking at some other unrelated things, like candy bars. Presenting the result as if it a real "improvement" is false advertisement. On the other hand, if we look at related observations, then the improvement is not a paradox at all. Let's say I want to estimate the average temperature in the US and in Europe. They are related, and combining the estimates will result to a better result, to nobody's surprise.
jprete 2y ago
My intuition is that the problem is in using squares for the error. The volume of space available for a given distance of error in 3-space is O(N^3) the magnitude of the error, so an error term of O(N^2) doesn't grow fast enough compared to the volume that can contain that magnitude of error. But I really don't know, it's just an intuition with no formalism behind it.
fromMars 2y ago
Can someone confirm the validity of the section called ”Can we derive the James-Stein estimator rigorously?"? The claim that the best estimator must be smooth seemed surprising to me.
hyperbovine 2y ago
Had a bit of a chuckle at the very-2024 definition of the Stein shrinkage estimator: \hat{mu} = ReLU(…)
- toth 2y ago
  Ditto. I think that ship has sailed, but I think it's unfortunate that "ReLU(x)" became a popular notation for "max(0,x)". And using the name "rectified linear unit" for basically "positive part" seems like a parody, like insisting on calling water "dihydrogen monoxide".
  - BlueTemplar 2y ago
    It hasn't "sailed" as long as they want to communicate with non-machine-learning people.
rssoconnor 2y ago
I think the part on "How arbitrary is the origin, really?" is not correct. The origin is arbitrary. As the Wikipedia article points you you can pick any point, whether or not it is the origin, and use the James-Stein estimator to push your estimate towards that point and it will improve one's mean squared error. If you pick a point to the left of your sample, then moving your estimate to the left will improve your mean squared error on average. If you pick a point to the right of your sample, then moving your estimate to the right will improve your mean squared error as well. I'm still trying to come to grips with this, and below is conjecture on my part. Imagine sampling many points from a 3-D Gaussian distribution (with identity covariance), making a nice cloud of points. Next choose any point P. P could be close to the cloud or far away, it doesn't matter. No matter which point P you pick, if you adjust all the points from your cloud of samples in accordance to this James-Stein formula, moving them all towards your chosen point P by various amounts, then, on average they will move closer to the center of your Gaussian distribution. This happens no matter where P is. The cloud is, of course, centered around the center of the Gaussian distribution. As the points are pulled towards this arbitrary point P some will be pulled away from the the center of Gaussian, some are pulled towards the center, and some are squeezed so that they are pulled away from the center in the paralled direction, but squeezed closer in the perpendicular direction. Anyhow, apparently everything ends up, on average, closer to the center of the Gaussian in the end. I'm not entirely sure what to make of this result. Perhaps it means that mean squared error is a silly error metric?
- titanomachy 2y ago
  Your visualization helped me understand this! If the center of the distribution is far from P, then all the lines from P to the points in your cluster are basically parallel, and you just shift your point cluster which doesn’t help your estimate. But if P is close to the mean, then it sits near the middle of your cluster, so pulling all points towards P is “shrinking” the cluster more than “shifting” it.
- toth 2y ago
  You make a valid point, but I feel there is something in the direction the article is gesturing at... The mean of the n-dimensional gaussian is an element of R^n, an unbounded space. There's no uninformed prior over this space, so there is always a choice of origin implicit in some way... As you say, you can shrink towards any point and you get a valid James-Steiner estimator that is strictly better than the naive estimator. But if you send the point you are shrinking towards to infinity you get the naive estimator again. So it feels like the fact you are implicitly selecting a finite chunk of R^n around an origin plays a role in the paradox...
zeroonetwothree 2y ago
I don’t understand the picture with the shaded circle. Sure the area to the left is smaller, but it also is more likely to be chosen because in a Gaussian values closer to the mean are more likely. So the picture alone doesn’t prove anything.
- rssoconnor 2y ago
  In the diagram the mean of the distribution is the center of the circle. Of the set of samples a fixed distance d from the mean of the distribution, strictly less than half of them will be closer to the origin than the mean is, and strictly greater than half of them will be further from the the origin than the mean. This is true for all values of d > 0, so the result holds for all samples.
deleted 2y ago
[deleted]
moi2388 2y ago
What a great read!

103 points by robertvc 2y ago

I'm horrible at stats, but is this saying that if I have 5 jars of pennies, and I guess the amount in each one. Then I find the average of all my guesses, and the variance between the guesses, then I can adjust each guess to a more likely answer with this method?

eru 2y ago
No, I don't think these problems are related.
kgwgk 2y ago
Not necessarily "more likely" but "better" in some "loss" sense. It could be "more likely" in the jars example where estimates may convey some relevant information for each other. But consider this example from wikipedia: "Suppose we are to estimate three unrelated parameters, such as the US wheat yield for 1993, the number of spectators at the Wimbledon tennis tournament in 2001, and the weight of a randomly chosen candy bar from the supermarket. Suppose we have independent Gaussian measurements of each of these quantities. Stein's example now tells us that we can get a better estimate (on average) for the vector of three parameters by simultaneously using the three unrelated measurements." https://en.wikipedia.org/wiki/Stein%27s_example#Example https://en.wikipedia.org/wiki/Stein%27s_example#Example

mitthrowaway2 2y ago

Sorry, I'm siding with the physicists here. If you're going to declare that your seemingly arbitrary choice of coordinate system is actually not arbitrary and part of your prior information about where the mean of the distribution is suspected to be, you have to put that in the initial problem statement.

kgwgk 2y ago
There is nothing magical about the origin, the shrinkage can be done towards any point and in fact when estimating multiple means it's customary to move each point closer to their average. https://www.math.drexel.edu/~tolya/EfronMorris.pdf https://www.math.drexel.edu/~tolya/EfronMorris.pdf
- mitthrowaway2 2y ago
  There is something magical about the origin when the result does not respect translational symmetry. In fact, in a real world setting I would probably use my first measurement to define the origin, having no other reference to reach for.
  - kgwgk 2y ago
    What does not respect translational symmetry? You have an estimator. If you apply shrinkage towards the origin you have another estimator. If you apply shrinkage towards [42, 42, ..., 42] you have yet another estimator. Etc. Is it a problem that different estimators produce different results?
    - mitthrowaway2 2y ago
      The James-Stein estimator does not respect translational symmetry. If I do a change of variables x2 = (x - offset), for an arbitrary offset, it gives me a different result! Whereas an estimator that just says I should guess that the mean is x, is unaffected by a change of coordinate system. This is a big problem if the coordinate system itself is not intended to contain information about the location of the mean. This makes sense if "zero" is physically meaningful, for example if negative values are not allowed in the problem domain (number of spectators at Wimbledon stadium, etc). Although in that case, my distribution probably shouldn't be Gaussian!
      - kgwgk 2y ago
        This is what the original paper from Stein says: "We choose an arbitrary point in the sample space independent of the outcome of the experiment and call it the origin. Of course, in the way we have expressed the problem this choice has already been made, but in a correct coordinate-free presentation, it would appear as an arbitrary choice of one point in an affine space." The James-Stein estimator in its general form is about shrinking towards an arbitrary point (which usually is not the origin). It respects translational symmetry if you transform that arbitrary point like everything else.

pfortuny 2y ago

I do not get it: if variance is too large, a random sample is very little representative of the mean. As simple as that? Now the specific formula may be complicated. But otherwise I do not understand the “paradox”? Or am I missing something?

hcks 2y ago
In the 1D case the single point will be the best estimator for the mean no matter what the variance is
- pfortuny 2y ago
  OK, this is a good reply, and very informative, many thanks. Now I understand why the thing is paradoxical.

credit_guy 2y ago

Stein's paradox is bogus. Somebody needs to say that. Here's one wikipedia example: > Suppose we are to estimate three unrelated parameters, such as the US wheat yield for 1993, the number of spectators at the Wimbledon tennis tournament in 2001, and the weight of a randomly chosen candy bar from the supermarket. Suppose we have independent Gaussian measurements of each of these quantities. Stein's example now tells us that we can get a better estimate (on average) for the vector of three parameters by simultaneously using the three unrelated measurements. Here's what's bogus about this: the "better estimate (on average)" is mathematically true ... for a certain definition of "better estimate". But whatever that definition is, it is irrelevant to the real world. If you believe you get a better estimate of the US wheat yield by estimating also the number of Wimbledon spectators and the weight of a candy bar in a shop, then you probably believe in telepathy and astrology too.

wavemode 2y ago
(Disclaimer, stats noob here) - I thought the point was that, you have a better chance of being -overall- closer to the mean (i.e., the 3D euclidean distance between your guess and the mean would be the smallest, on average), even though you may not necessarily have improved your odds of guessing any of the single individual means. So it's not that "you get a better estimate of the US wheat yield by estimating also the number of Wimbledon spectators and the weight of a candy bar in a shop", it's simply that you get a better estimate for the combined vector of the three means. (Which, in this case, the vector of the three means is probably meaningless, since the three data sets are entirely unrelated. But we could also imagine scenarios where that vector is meaningful.) Am I misunderstanding something?
- credit_guy 2y ago
  You are most likely right. I am personally bothered by the way it is presented as a "paradox", with the implication that it would have real world applications. I have zero doubts that you can't improve the estimate of the US wheat yields by looking at some other unrelated things, like candy bars. Presenting the result as if it a real "improvement" is false advertisement. On the other hand, if we look at related observations, then the improvement is not a paradox at all. Let's say I want to estimate the average temperature in the US and in Europe. They are related, and combining the estimates will result to a better result, to nobody's surprise.

jprete 2y ago

My intuition is that the problem is in using squares for the error. The volume of space available for a given distance of error in 3-space is O(N^3) the magnitude of the error, so an error term of O(N^2) doesn't grow fast enough compared to the volume that can contain that magnitude of error. But I really don't know, it's just an intuition with no formalism behind it.

fromMars 2y ago

Can someone confirm the validity of the section called ”Can we derive the James-Stein estimator rigorously?"? The claim that the best estimator must be smooth seemed surprising to me.

hyperbovine 2y ago

Had a bit of a chuckle at the very-2024 definition of the Stein shrinkage estimator: \hat{mu} = ReLU(…)

toth 2y ago
Ditto. I think that ship has sailed, but I think it's unfortunate that "ReLU(x)" became a popular notation for "max(0,x)". And using the name "rectified linear unit" for basically "positive part" seems like a parody, like insisting on calling water "dihydrogen monoxide".
- BlueTemplar 2y ago
  It hasn't "sailed" as long as they want to communicate with non-machine-learning people.

rssoconnor 2y ago

I think the part on "How arbitrary is the origin, really?" is not correct. The origin is arbitrary. As the Wikipedia article points you you can pick any point, whether or not it is the origin, and use the James-Stein estimator to push your estimate towards that point and it will improve one's mean squared error. If you pick a point to the left of your sample, then moving your estimate to the left will improve your mean squared error on average. If you pick a point to the right of your sample, then moving your estimate to the right will improve your mean squared error as well. I'm still trying to come to grips with this, and below is conjecture on my part. Imagine sampling many points from a 3-D Gaussian distribution (with identity covariance), making a nice cloud of points. Next choose any point P. P could be close to the cloud or far away, it doesn't matter. No matter which point P you pick, if you adjust all the points from your cloud of samples in accordance to this James-Stein formula, moving them all towards your chosen point P by various amounts, then, on average they will move closer to the center of your Gaussian distribution. This happens no matter where P is. The cloud is, of course, centered around the center of the Gaussian distribution. As the points are pulled towards this arbitrary point P some will be pulled away from the the center of Gaussian, some are pulled towards the center, and some are squeezed so that they are pulled away from the center in the paralled direction, but squeezed closer in the perpendicular direction. Anyhow, apparently everything ends up, on average, closer to the center of the Gaussian in the end. I'm not entirely sure what to make of this result. Perhaps it means that mean squared error is a silly error metric?

titanomachy 2y ago
Your visualization helped me understand this! If the center of the distribution is far from P, then all the lines from P to the points in your cluster are basically parallel, and you just shift your point cluster which doesn’t help your estimate. But if P is close to the mean, then it sits near the middle of your cluster, so pulling all points towards P is “shrinking” the cluster more than “shifting” it.
toth 2y ago
You make a valid point, but I feel there is something in the direction the article is gesturing at... The mean of the n-dimensional gaussian is an element of R^n, an unbounded space. There's no uninformed prior over this space, so there is always a choice of origin implicit in some way... As you say, you can shrink towards any point and you get a valid James-Steiner estimator that is strictly better than the naive estimator. But if you send the point you are shrinking towards to infinity you get the naive estimator again. So it feels like the fact you are implicitly selecting a finite chunk of R^n around an origin plays a role in the paradox...

zeroonetwothree 2y ago

I don’t understand the picture with the shaded circle. Sure the area to the left is smaller, but it also is more likely to be chosen because in a Gaussian values closer to the mean are more likely. So the picture alone doesn’t prove anything.

rssoconnor 2y ago
In the diagram the mean of the distribution is the center of the circle. Of the set of samples a fixed distance d from the mean of the distribution, strictly less than half of them will be closer to the origin than the mean is, and strictly greater than half of them will be further from the the origin than the mean. This is true for all values of d > 0, so the result holds for all samples.

deleted 2y ago

[deleted]

moi2388 2y ago

What a great read!