3 ms·

> If you don't assume existence of second moments, it is non-trivial to show that min E[(X-f(Y))^2] makes sense if any. To show that E[Y|X] exists, essential

> If you don't assume existence of second moments, it is non-trivial to show that min E[(X-f(Y))^2] makes sense if any. To show that E[Y|X] exists, essentially I did that. Really, all need from X is its sigmaalgebra. That is, can do the argument conditioning on a sigmaalgrbra instead of a random variable X. For Y, I didn't include the details back to the Radon-Nikodym theorem, but may not even need that E[Y] is finite. Sure don't need finite "second moments".

LudwigNagasena 4y ago
You assumed that E[Y|X] and E[Y^2|X] exist.
- graycat 4y ago
  E[Y^2|X] exists? Doesn't ask for much, merely that Y be measurable, that is, a random variable. We get to f'get about Riemann integration of probability densities. Full details, with some of the most elegant, powerful, and precise work in the history of civilization, are in the references I gave, to Rudin and Neveu.
  - LudwigNagasena 4y ago
    Uhm. Not every random variable has the second moment. And wasn’t that your objection towards the formulation in terms of Hilbert space? I am a bit lost trying to understand what you are trying to argue.
    - graycat 4y ago
      Moments and Hilbert space need have nothing to do with real valued function f of real variable x minimizing E[ (Y - f(X)^2 ] for real valued random variables X and Y. For Y it is enough that it has an expectation. For X, it need only be measurable, that is, a random variable. I referenced the Radon-Nikodym theorem. Assuming that, the definition and basic properties of E[Y|X] are immediate. Let's back up to something simpler: What real number a minimizes E[ (Y - a)^2 ] = E[ Y^2 - 2aY + a^2 ] = E[Y^2] - E[2aY] + E[a^2] = E[Y^2] - 2aE[Y] + E[a^2] = E[Y^2] + E[Y]^2 - 2aE[Y] - E[Y]^2 + E[a^2] = E[Y^2] + E[Y]^2 - 2aE[Y] - E[Y]^2 + a^2 = E[Y^2] + (E[Y] - a)^2 - E[Y]^2 which is minimized by a = E[Y] So, if real random variable X = a that is, takes on only one value, then E[ (Y - a)^2 ] is minimized by a = E[Y] To improve on that case, suppose real random variable X is discrete, takes on finitely many values a_i for positive integer n and i = 1, 2, ..., n and suppose random variables / X = a_i 1_{X = a_i} = | \ 0 otherwise Then X = sum_{i = 1}^n a_i 1_{X = a_i} Let b_i = E[Y 1_{X = a_i}] Then by what we did above for a = E[Y] minimizing E[ (Y - a)^2 ] we have that f(X) = sum_{i = 1}^n b_i 1_{X = a_i} minimizes E[(Y - f(X))^2] And we would have E[Y|X] = f(X) Basically we are doing classic cross tabulation: We collect data on both X and Y. When X = a_i, we average the corresponding Y values and get b_i for our estimate of Y. The Radon-Nikodym result, among other things, lets us do the same when X is not discrete, that is, is any random variable. We don't even need for X to have an expectation -- all we want is measurability so that from X we have a sigma algebra. The Radon-Nikodym result is part of measure theory, with a novel von Neumann proof in Rudin. The connection with probability and conditional expectation is part of graduate probability as in Neveu. My all-time favorite course in school was the graduate probability course I took from Neveu, etc. I never wanted to be a college professor, but for some unusual reasons for some years I was. Now I'm happy that I'm no longer a college professor! I've published some peer-reviewed papers of my original research in pure/applied math. I don't like publishing and don't want to publish. In particular, I don't want to write a lecture in graduate probability here on Hacker News. But you can see above that I made no use of moments. This thread started with an observation that conditional probability is amazing. Then, easy enough, I mentioned that conditional probability is a special case of conditional expectation which has an amazing minimization property. In addition, conditional expectation is the core of both Markov processes and martingales. The martingale convergence theorem is astounding. My real interest now is my Internet startup -- back to it.
      - LudwigNagasena 4y ago
        I am not sure what you mean by moments, but in probability theory “the n-th moment” means E[X^n]; and you’ve clearly used them. In probability theory the Radon-Nikodym is used to justify existence of random variables. But it cannot be used to show that any variable has an expected value (ie has a moment, ie is integrable). The Cauchy distribution is a traditional counter example to such assertion. > For Y it is enough that it has an expectation That is false. Just look at your “simple example”. You arrive to E[Y^2]. An expectation is not enough. It should be clear intuitively because MSE is a minimization of the second moment of a certain rv, so you are bound to have at least some constraints for second-order relationships for the variables you’ve used to construct that certain rv. I think you may need to re-read Rudin and Neveau and refresh your knowledge. (Though I can say nothing about Neveau, I studied using the textbook by Shao.)
        graycat 4y ago
        For any real valued random variable, E[Y^2] always exists although it might be infinity. You mentioned moments -- as I recall some writing in statistics uses that word, but I never have. Your description of the Radon-Nikodym (R-N) result does not look right. Again, I don't want to write a lecture in graduate probability, but from memory the usual description of that result goes: Given real valued, integrable function f and sigma algebra s, there exists real value function g measurable with respect to sigma algebra s so that g integrates like f over all sets in s. So, with real valued random variable Y in place of function f, we let E[Y|s] be the function g. If s is the sigma algebra generated by real random variable X, then we write g(X) = E[Y|X] = E[Y|s] So the R-N result makes immediate the definition of E[Y|X] and shows that it exists. If you want a perfect, polished presentation, see the two references I gave, Rudin and Neveu. My main contribution here remains: f(X) = E[Y|X] minimizes E[ (Y - f(X))^2 ] and is a minimum variance, unbiased estimator of Y. That's a generalization of the start of this thread and a bit amazing.
        LudwigNagasena 4y ago
        > Given real valued, integrable function f Of course that makes it possible. Note that it doesn’t say that f is simply measurable. By definition being integrable means E[|f|] < infinity. (And being square-integrable means E[f^2] < infinity, an even stronger condition). Consider Y = Z/X where Z, X iid ~ N(0,1). Try to calculate E[Y|X], and try to minimize E[ (Y - f(X))^2 ]. Neither will make sense. I am sure both Rudin and Neveu were aware of this trivial result.
    - graycat 4y ago
      For a real random variable Y, E[Y^2] always exists although might be infinity.

LudwigNagasena 4y ago

You assumed that E[Y|X] and E[Y^2|X] exist.

graycat 4y ago
E[Y^2|X] exists? Doesn't ask for much, merely that Y be measurable, that is, a random variable. We get to f'get about Riemann integration of probability densities. Full details, with some of the most elegant, powerful, and precise work in the history of civilization, are in the references I gave, to Rudin and Neveu.
- LudwigNagasena 4y ago
  Uhm. Not every random variable has the second moment. And wasn’t that your objection towards the formulation in terms of Hilbert space? I am a bit lost trying to understand what you are trying to argue.
  - graycat 4y ago
    Moments and Hilbert space need have nothing to do with real valued function f of real variable x minimizing E[ (Y - f(X)^2 ] for real valued random variables X and Y. For Y it is enough that it has an expectation. For X, it need only be measurable, that is, a random variable. I referenced the Radon-Nikodym theorem. Assuming that, the definition and basic properties of E[Y|X] are immediate. Let's back up to something simpler: What real number a minimizes E[ (Y - a)^2 ] = E[ Y^2 - 2aY + a^2 ] = E[Y^2] - E[2aY] + E[a^2] = E[Y^2] - 2aE[Y] + E[a^2] = E[Y^2] + E[Y]^2 - 2aE[Y] - E[Y]^2 + E[a^2] = E[Y^2] + E[Y]^2 - 2aE[Y] - E[Y]^2 + a^2 = E[Y^2] + (E[Y] - a)^2 - E[Y]^2 which is minimized by a = E[Y] So, if real random variable X = a that is, takes on only one value, then E[ (Y - a)^2 ] is minimized by a = E[Y] To improve on that case, suppose real random variable X is discrete, takes on finitely many values a_i for positive integer n and i = 1, 2, ..., n and suppose random variables / X = a_i 1_{X = a_i} = | \ 0 otherwise Then X = sum_{i = 1}^n a_i 1_{X = a_i} Let b_i = E[Y 1_{X = a_i}] Then by what we did above for a = E[Y] minimizing E[ (Y - a)^2 ] we have that f(X) = sum_{i = 1}^n b_i 1_{X = a_i} minimizes E[(Y - f(X))^2] And we would have E[Y|X] = f(X) Basically we are doing classic cross tabulation: We collect data on both X and Y. When X = a_i, we average the corresponding Y values and get b_i for our estimate of Y. The Radon-Nikodym result, among other things, lets us do the same when X is not discrete, that is, is any random variable. We don't even need for X to have an expectation -- all we want is measurability so that from X we have a sigma algebra. The Radon-Nikodym result is part of measure theory, with a novel von Neumann proof in Rudin. The connection with probability and conditional expectation is part of graduate probability as in Neveu. My all-time favorite course in school was the graduate probability course I took from Neveu, etc. I never wanted to be a college professor, but for some unusual reasons for some years I was. Now I'm happy that I'm no longer a college professor! I've published some peer-reviewed papers of my original research in pure/applied math. I don't like publishing and don't want to publish. In particular, I don't want to write a lecture in graduate probability here on Hacker News. But you can see above that I made no use of moments. This thread started with an observation that conditional probability is amazing. Then, easy enough, I mentioned that conditional probability is a special case of conditional expectation which has an amazing minimization property. In addition, conditional expectation is the core of both Markov processes and martingales. The martingale convergence theorem is astounding. My real interest now is my Internet startup -- back to it.
    - LudwigNagasena 4y ago
      I am not sure what you mean by moments, but in probability theory “the n-th moment” means E[X^n]; and you’ve clearly used them. In probability theory the Radon-Nikodym is used to justify existence of random variables. But it cannot be used to show that any variable has an expected value (ie has a moment, ie is integrable). The Cauchy distribution is a traditional counter example to such assertion. > For Y it is enough that it has an expectation That is false. Just look at your “simple example”. You arrive to E[Y^2]. An expectation is not enough. It should be clear intuitively because MSE is a minimization of the second moment of a certain rv, so you are bound to have at least some constraints for second-order relationships for the variables you’ve used to construct that certain rv. I think you may need to re-read Rudin and Neveau and refresh your knowledge. (Though I can say nothing about Neveau, I studied using the textbook by Shao.)
      - graycat 4y ago
        For any real valued random variable, E[Y^2] always exists although it might be infinity. You mentioned moments -- as I recall some writing in statistics uses that word, but I never have. Your description of the Radon-Nikodym (R-N) result does not look right. Again, I don't want to write a lecture in graduate probability, but from memory the usual description of that result goes: Given real valued, integrable function f and sigma algebra s, there exists real value function g measurable with respect to sigma algebra s so that g integrates like f over all sets in s. So, with real valued random variable Y in place of function f, we let E[Y|s] be the function g. If s is the sigma algebra generated by real random variable X, then we write g(X) = E[Y|X] = E[Y|s] So the R-N result makes immediate the definition of E[Y|X] and shows that it exists. If you want a perfect, polished presentation, see the two references I gave, Rudin and Neveu. My main contribution here remains: f(X) = E[Y|X] minimizes E[ (Y - f(X))^2 ] and is a minimum variance, unbiased estimator of Y. That's a generalization of the start of this thread and a bit amazing.
        LudwigNagasena 4y ago
        > Given real valued, integrable function f Of course that makes it possible. Note that it doesn’t say that f is simply measurable. By definition being integrable means E[|f|] < infinity. (And being square-integrable means E[f^2] < infinity, an even stronger condition). Consider Y = Z/X where Z, X iid ~ N(0,1). Try to calculate E[Y|X], and try to minimize E[ (Y - f(X))^2 ]. Neither will make sense. I am sure both Rudin and Neveu were aware of this trivial result.
  - graycat 4y ago
    For a real random variable Y, E[Y^2] always exists although might be infinity.