4 ms·
Moments and Hilbert space need have nothing to do with real valued function f of real variable x minimizing E[ (Y - f(X)^2 ] for real valued random varia
by graycat 4y ago
Moments and Hilbert space need have
nothing to do with real valued function f of
real variable x minimizing
E[ (Y - f(X)^2 ]
for real valued random variables X and Y.
For Y it is enough that it has an
expectation. For X, it need only be
measurable, that is, a random variable.
I referenced the Radon-Nikodym theorem.
Assuming that, the definition and basic
properties of
E[Y|X]
are immediate.
Let's back up to something simpler: What
real number a minimizes
E[ (Y - a)^2 ]
= E[ Y^2 - 2aY + a^2 ]
= E[Y^2] - E[2aY] + E[a^2]
= E[Y^2] - 2aE[Y] + E[a^2]
= E[Y^2] + E[Y]^2 - 2aE[Y]
- E[Y]^2 + E[a^2]
= E[Y^2] + E[Y]^2 - 2aE[Y] - E[Y]^2 + a^2
= E[Y^2] + (E[Y] - a)^2 - E[Y]^2
which is minimized by
a = E[Y]
So, if real random variable
X = a
that is, takes on only one value, then
E[ (Y - a)^2 ]
is minimized by
a = E[Y]
To improve on that case, suppose real
random variable X is discrete, takes on
finitely many values
a_i
for positive integer n and
i = 1, 2, ..., n
and suppose random variables
/ X = a_i
1_{X = a_i} = |
\ 0 otherwise
Then
X = sum_{i = 1}^n a_i 1_{X = a_i}
Let
b_i = E[Y 1_{X = a_i}]
Then by what we did above for
a = E[Y]
minimizing
E[ (Y - a)^2 ]
we have that
f(X) = sum_{i = 1}^n b_i 1_{X = a_i}
minimizes
E[(Y - f(X))^2]
And we would have
E[Y|X] = f(X)
Basically we are doing classic cross
tabulation: We collect data on both X
and Y. When X = a_i, we average the
corresponding Y values and get b_i for our
estimate of Y.
The Radon-Nikodym result, among other
things, lets us do the same when X is not
discrete, that is, is any random variable.
We don't even need for X to have an
expectation -- all we want is
measurability so that from X we have a
sigma algebra.
The Radon-Nikodym result is part of
measure theory, with a novel von Neumann
proof in Rudin. The connection with
probability and conditional expectation is
part of graduate probability as in
Neveu.
My all-time favorite course in school was
the graduate probability course I took
from Neveu, etc.
I never wanted to be a college professor,
but for some unusual reasons for some
years I was. Now I'm happy that I'm no
longer a college professor!
I've published some peer-reviewed papers
of my original research in pure/applied
math. I don't like publishing and don't
want to publish. In particular, I don't
want to write a lecture in graduate
probability here on Hacker News.
But you can see above that I made no use
of moments.
This thread started with an observation
that conditional probability is amazing.
Then, easy enough, I mentioned that
conditional probability is a special case
of conditional expectation which has an
amazing minimization property.
In addition, conditional expectation is
the core of both Markov processes and
martingales. The martingale convergence
theorem is astounding.
My real interest now is my Internet
startup -- back to it.
- LudwigNagasena 4y agoI am not sure what you mean by moments, but in probability theory “the n-th moment” means E[X^n]; and you’ve clearly used them. In probability theory the Radon-Nikodym is used to justify existence of random variables. But it cannot be used to show that any variable has an expected value (ie has a moment, ie is integrable). The Cauchy distribution is a traditional counter example to such assertion. > For Y it is enough that it has an expectation That is false. Just look at your “simple example”. You arrive to E[Y^2]. An expectation is not enough. It should be clear intuitively because MSE is a minimization of the second moment of a certain rv, so you are bound to have at least some constraints for second-order relationships for the variables you’ve used to construct that certain rv. I think you may need to re-read Rudin and Neveau and refresh your knowledge. (Though I can say nothing about Neveau, I studied using the textbook by Shao.)
- graycat 4y agoFor any real valued random variable, E[Y^2] always exists although it might be infinity. You mentioned moments -- as I recall some writing in statistics uses that word, but I never have. Your description of the Radon-Nikodym (R-N) result does not look right. Again, I don't want to write a lecture in graduate probability, but from memory the usual description of that result goes: Given real valued, integrable function f and sigma algebra s, there exists real value function g measurable with respect to sigma algebra s so that g integrates like f over all sets in s. So, with real valued random variable Y in place of function f, we let E[Y|s] be the function g. If s is the sigma algebra generated by real random variable X, then we write g(X) = E[Y|X] = E[Y|s] So the R-N result makes immediate the definition of E[Y|X] and shows that it exists. If you want a perfect, polished presentation, see the two references I gave, Rudin and Neveu. My main contribution here remains: f(X) = E[Y|X] minimizes E[ (Y - f(X))^2 ] and is a minimum variance, unbiased estimator of Y. That's a generalization of the start of this thread and a bit amazing.
- LudwigNagasena 4y ago> Given real valued, integrable function f Of course that makes it possible. Note that it doesn’t say that f is simply measurable. By definition being integrable means E[|f|] < infinity. (And being square-integrable means E[f^2] < infinity, an even stronger condition). Consider Y = Z/X where Z, X iid ~ N(0,1). Try to calculate E[Y|X], and try to minimize E[ (Y - f(X))^2 ]. Neither will make sense. I am sure both Rudin and Neveu were aware of this trivial result.