3 ms·

> I don't think it's obvious why you would want that. To minimize E[(X-f(Y))^2] is just common, traditional, nearly universal least squares with all the powe

> I don't think it's obvious why you would want that. To minimize E[(X-f(Y))^2] is just common, traditional, nearly universal least squares with all the powers, advantages, and attributes there unto pertaining! Yes, can also connect the least squares with Hilbert space geometry, but for that you add an assumption not needed with my approach via the Radon-Nikodym result.

LudwigNagasena 4y ago
> To minimize E[(X-f(Y))^2] is just common, traditional, nearly universal least squares with all the powers, advantages, and attributes there unto pertaining For me that's hardly an intuitive notion. For example, in some econometrics textbooks you can meet the reversed argument: OLS is motivated as a way to estimate conditional expectation function. So one can ask, isn't OLS "just common, traditional, nearly universal least squares with all the powers, advantages, and attributes there unto pertaining" simply because it is a way to estimate conditional expectation? And now you are back to square one. > you add an assumption not needed with my approach via the Radon-Nikodym result Existence of a unique minimizer of E[(X-f(Y))^2] is a property of L2. You may carryover many properties to L1 by continuity but it isn't one of them. If you don't assume existence of second moments, it is non-trivial to show that min E[(X-f(Y))^2] makes sense if any. I think that you have to put specific constraints on X and Y, but I don't know which are the least restrictive.
- graycat 4y ago
  > If you don't assume existence of second moments, it is non-trivial to show that min E[(X-f(Y))^2] makes sense if any. To show that E[Y|X] exists, essentially I did that. Really, all need from X is its sigmaalgebra. That is, can do the argument conditioning on a sigmaalgrbra instead of a random variable X. For Y, I didn't include the details back to the Radon-Nikodym theorem, but may not even need that E[Y] is finite. Sure don't need finite "second moments".
  - LudwigNagasena 4y ago
    You assumed that E[Y|X] and E[Y^2|X] exist.
    - graycat 4y ago
      E[Y^2|X] exists? Doesn't ask for much, merely that Y be measurable, that is, a random variable. We get to f'get about Riemann integration of probability densities. Full details, with some of the most elegant, powerful, and precise work in the history of civilization, are in the references I gave, to Rudin and Neveu.
      - LudwigNagasena 4y ago
        Uhm. Not every random variable has the second moment. And wasn’t that your objection towards the formulation in terms of Hilbert space? I am a bit lost trying to understand what you are trying to argue.
        graycat 4y ago
        Moments and Hilbert space need have nothing to do with real valued function f of real variable x minimizing E[ (Y - f(X)^2 ] for real valued random variables X and Y. For Y it is enough that it has an expectation. For X, it need only be measurable, that is, a random variable. I referenced the Radon-Nikodym theorem. Assuming that, the definition and basic properties of E[Y|X] are immediate. Let's back up to something simpler: What real number a minimizes E[ (Y - a)^2 ] = E[ Y^2 - 2aY + a^2 ] = E[Y^2] - E[2aY] + E[a^2] = E[Y^2] - 2aE[Y] + E[a^2] = E[Y^2] + E[Y]^2 - 2aE[Y] - E[Y]^2 + E[a^2] = E[Y^2] + E[Y]^2 - 2aE[Y] - E[Y]^2 + a^2 = E[Y^2] + (E[Y] - a)^2 - E[Y]^2 which is minimized by a = E[Y] So, if real random variable X = a that is, takes on only one value, then E[ (Y - a)^2 ] is minimized by a = E[Y] To improve on that case, suppose real random variable X is discrete, takes on finitely many values a_i for positive integer n and i = 1, 2, ..., n and suppose random variables / X = a_i 1_{X = a_i} = | \ 0 otherwise Then X = sum_{i = 1}^n a_i 1_{X = a_i} Let b_i = E[Y 1_{X = a_i}] Then by what we did above for a = E[Y] minimizing E[ (Y - a)^2 ] we have that f(X) = sum_{i = 1}^n b_i 1_{X = a_i} minimizes E[(Y - f(X))^2] And we would have E[Y|X] = f(X) Basically we are doing classic cross tabulation: We collect data on both X and Y. When X = a_i, we average the corresponding Y values and get b_i for our estimate of Y. The Radon-Nikodym result, among other things, lets us do the same when X is not discrete, that is, is any random variable. We don't even need for X to have an expectation -- all we want is measurability so that from X we have a sigma algebra. The Radon-Nikodym result is part of measure theory, with a novel von Neumann proof in Rudin. The connection with probability and conditional expectation is part of graduate probability as in Neveu. My all-time favorite course in school was the graduate probability course I took from Neveu, etc. I never wanted to be a college professor, but for some unusual reasons for some years I was. Now I'm happy that I'm no longer a college professor! I've published some peer-reviewed papers of my original research in pure/applied math. I don't like publishing and don't want to publish. In particular, I don't want to write a lecture in graduate probability here on Hacker News. But you can see above that I made no use of moments. This thread started with an observation that conditional probability is amazing. Then, easy enough, I mentioned that conditional probability is a special case of conditional expectation which has an amazing minimization property. In addition, conditional expectation is the core of both Markov processes and martingales. The martingale convergence theorem is astounding. My real interest now is my Internet startup -- back to it.

LudwigNagasena 4y ago

> To minimize E[(X-f(Y))^2] is just common, traditional, nearly universal least squares with all the powers, advantages, and attributes there unto pertaining For me that's hardly an intuitive notion. For example, in some econometrics textbooks you can meet the reversed argument: OLS is motivated as a way to estimate conditional expectation function. So one can ask, isn't OLS "just common, traditional, nearly universal least squares with all the powers, advantages, and attributes there unto pertaining" simply because it is a way to estimate conditional expectation? And now you are back to square one. > you add an assumption not needed with my approach via the Radon-Nikodym result Existence of a unique minimizer of E[(X-f(Y))^2] is a property of L2. You may carryover many properties to L1 by continuity but it isn't one of them. If you don't assume existence of second moments, it is non-trivial to show that min E[(X-f(Y))^2] makes sense if any. I think that you have to put specific constraints on X and Y, but I don't know which are the least restrictive.

graycat 4y ago
> If you don't assume existence of second moments, it is non-trivial to show that min E[(X-f(Y))^2] makes sense if any. To show that E[Y|X] exists, essentially I did that. Really, all need from X is its sigmaalgebra. That is, can do the argument conditioning on a sigmaalgrbra instead of a random variable X. For Y, I didn't include the details back to the Radon-Nikodym theorem, but may not even need that E[Y] is finite. Sure don't need finite "second moments".
- LudwigNagasena 4y ago
  You assumed that E[Y|X] and E[Y^2|X] exist.
  - graycat 4y ago
    E[Y^2|X] exists? Doesn't ask for much, merely that Y be measurable, that is, a random variable. We get to f'get about Riemann integration of probability densities. Full details, with some of the most elegant, powerful, and precise work in the history of civilization, are in the references I gave, to Rudin and Neveu.
    - LudwigNagasena 4y ago
      Uhm. Not every random variable has the second moment. And wasn’t that your objection towards the formulation in terms of Hilbert space? I am a bit lost trying to understand what you are trying to argue.
      - graycat 4y ago
        Moments and Hilbert space need have nothing to do with real valued function f of real variable x minimizing E[ (Y - f(X)^2 ] for real valued random variables X and Y. For Y it is enough that it has an expectation. For X, it need only be measurable, that is, a random variable. I referenced the Radon-Nikodym theorem. Assuming that, the definition and basic properties of E[Y|X] are immediate. Let's back up to something simpler: What real number a minimizes E[ (Y - a)^2 ] = E[ Y^2 - 2aY + a^2 ] = E[Y^2] - E[2aY] + E[a^2] = E[Y^2] - 2aE[Y] + E[a^2] = E[Y^2] + E[Y]^2 - 2aE[Y] - E[Y]^2 + E[a^2] = E[Y^2] + E[Y]^2 - 2aE[Y] - E[Y]^2 + a^2 = E[Y^2] + (E[Y] - a)^2 - E[Y]^2 which is minimized by a = E[Y] So, if real random variable X = a that is, takes on only one value, then E[ (Y - a)^2 ] is minimized by a = E[Y] To improve on that case, suppose real random variable X is discrete, takes on finitely many values a_i for positive integer n and i = 1, 2, ..., n and suppose random variables / X = a_i 1_{X = a_i} = | \ 0 otherwise Then X = sum_{i = 1}^n a_i 1_{X = a_i} Let b_i = E[Y 1_{X = a_i}] Then by what we did above for a = E[Y] minimizing E[ (Y - a)^2 ] we have that f(X) = sum_{i = 1}^n b_i 1_{X = a_i} minimizes E[(Y - f(X))^2] And we would have E[Y|X] = f(X) Basically we are doing classic cross tabulation: We collect data on both X and Y. When X = a_i, we average the corresponding Y values and get b_i for our estimate of Y. The Radon-Nikodym result, among other things, lets us do the same when X is not discrete, that is, is any random variable. We don't even need for X to have an expectation -- all we want is measurability so that from X we have a sigma algebra. The Radon-Nikodym result is part of measure theory, with a novel von Neumann proof in Rudin. The connection with probability and conditional expectation is part of graduate probability as in Neveu. My all-time favorite course in school was the graduate probability course I took from Neveu, etc. I never wanted to be a college professor, but for some unusual reasons for some years I was. Now I'm happy that I'm no longer a college professor! I've published some peer-reviewed papers of my original research in pure/applied math. I don't like publishing and don't want to publish. In particular, I don't want to write a lecture in graduate probability here on Hacker News. But you can see above that I made no use of moments. This thread started with an observation that conditional probability is amazing. Then, easy enough, I mentioned that conditional probability is a special case of conditional expectation which has an amazing minimization property. In addition, conditional expectation is the core of both Markov processes and martingales. The martingale convergence theorem is astounding. My real interest now is my Internet startup -- back to it.