4 ms·

> The unreasonable effectiveness of conditional probabilities Yup. There is a more general result that is more amazing with still more "unreasonable effective

> The unreasonable effectiveness of conditional probabilities Yup. There is a more general result that is more amazing with still more "unreasonable effectiveness": For events A and B, we can consider the conditional probability of A given B: P(A|B) = P(A and B)/P(B) Then we can generalize P(A|B): Suppose we do an experiment and make two observations, X and Y. We regard each of X and Y as a random variable that takes on real number values. We assume that X and Y have averages, that is, expectations, and write those as E[X] for the expectation of X and E[Y] for the expectation of Y. We want to use X to approximate Y. So, we want a real valued function f of one real variable x and use f(X) to approximate Y. For the most accurate approximation, we want to minimize the expectation E[(Y - f(X))^2] Claim: For f(X) we want f(X) = E[Y|X] where E[Y}X} is the conditional expectation of Y given X and a generalization of P(A|B). Proof: We start by using one of the properties of conditional expectation and then continue with just simple algebra: E[(Y - f(X))^2] = E[ E[(Y - f(X))^2|X] ] = E[ E[Y^2 - 2Yf(X) + f(X)^2|X] ] = E[ E[Y^2|X] - 2f(X)E[Y|X] + f(X)^2 ] = E[ E[Y^2|X] + E[Y|X]^2 - 2f(X)E[Y|X] + f(X)^2 - E[Y|X]^2 ] = E[ E[Y^2|X] + ( E[Y|X] - f(X) )^2 - E[Y|X]^2 ] which is minimized with f(X) = E[Y|X] Done. So, the "unreasonable effectiveness" of conditional expectation that it provides the best non-linear least squares approximation. In terms of statistics E[Y|X] is the minimum variance estimator of Y and is also unbiased. Yes, P(A|B), conditional probability, can be seen as a special case of E[Y|X], conditional expectation. The detailed definition of E[Y|X] is based on the Radon-Nikodym theorem. W. Rudin's Real and Complex Analysis gives the novel von Neumann proof. Then in Neveu, Mathematical Foundations of the Calculus of Probability is elegant coverage of E[Y|X].

LudwigNagasena 4y ago
> For the most accurate approximation, we want to minimize the expectation > E[(Y - f(X))^2] I don't think it's obvious why you would want that. It looks like you smuggle in your conclusion with that assumption. I would start with establishing that probability space of variables with second moments is a Hilbert space with the inner product of X and Y defined as E[XY], which lends itself to a nice geometrical interpretation. Then we can project Y onto X, hence Y = Y_∥ + Y_⊥ where Y_∥ is an orthogonal projection and Y_⊥ is the leftover part that is orthogonal to X, ie X can perfectly predict Y_∥ but cannot say anything about Y_⊥. So Y_∥ by construction contains all information we can have about Y if we know X. We can also note that orthogonal projection minimizes distance, i.e. it minimizes E[(X-f(Y))^2].
- graycat 4y ago
  > I don't think it's obvious why you would want that. To minimize E[(X-f(Y))^2] is just common, traditional, nearly universal least squares with all the powers, advantages, and attributes there unto pertaining! Yes, can also connect the least squares with Hilbert space geometry, but for that you add an assumption not needed with my approach via the Radon-Nikodym result.
  - LudwigNagasena 4y ago
    > To minimize E[(X-f(Y))^2] is just common, traditional, nearly universal least squares with all the powers, advantages, and attributes there unto pertaining For me that's hardly an intuitive notion. For example, in some econometrics textbooks you can meet the reversed argument: OLS is motivated as a way to estimate conditional expectation function. So one can ask, isn't OLS "just common, traditional, nearly universal least squares with all the powers, advantages, and attributes there unto pertaining" simply because it is a way to estimate conditional expectation? And now you are back to square one. > you add an assumption not needed with my approach via the Radon-Nikodym result Existence of a unique minimizer of E[(X-f(Y))^2] is a property of L2. You may carryover many properties to L1 by continuity but it isn't one of them. If you don't assume existence of second moments, it is non-trivial to show that min E[(X-f(Y))^2] makes sense if any. I think that you have to put specific constraints on X and Y, but I don't know which are the least restrictive.
    - graycat 4y ago
      > If you don't assume existence of second moments, it is non-trivial to show that min E[(X-f(Y))^2] makes sense if any. To show that E[Y|X] exists, essentially I did that. Really, all need from X is its sigmaalgebra. That is, can do the argument conditioning on a sigmaalgrbra instead of a random variable X. For Y, I didn't include the details back to the Radon-Nikodym theorem, but may not even need that E[Y] is finite. Sure don't need finite "second moments".
      - LudwigNagasena 4y ago

LudwigNagasena 4y ago

> For the most accurate approximation, we want to minimize the expectation > E[(Y - f(X))^2] I don't think it's obvious why you would want that. It looks like you smuggle in your conclusion with that assumption. I would start with establishing that probability space of variables with second moments is a Hilbert space with the inner product of X and Y defined as E[XY], which lends itself to a nice geometrical interpretation. Then we can project Y onto X, hence Y = Y_∥ + Y_⊥ where Y_∥ is an orthogonal projection and Y_⊥ is the leftover part that is orthogonal to X, ie X can perfectly predict Y_∥ but cannot say anything about Y_⊥. So Y_∥ by construction contains all information we can have about Y if we know X. We can also note that orthogonal projection minimizes distance, i.e. it minimizes E[(X-f(Y))^2].

graycat 4y ago
> I don't think it's obvious why you would want that. To minimize E[(X-f(Y))^2] is just common, traditional, nearly universal least squares with all the powers, advantages, and attributes there unto pertaining! Yes, can also connect the least squares with Hilbert space geometry, but for that you add an assumption not needed with my approach via the Radon-Nikodym result.
- LudwigNagasena 4y ago
  > To minimize E[(X-f(Y))^2] is just common, traditional, nearly universal least squares with all the powers, advantages, and attributes there unto pertaining For me that's hardly an intuitive notion. For example, in some econometrics textbooks you can meet the reversed argument: OLS is motivated as a way to estimate conditional expectation function. So one can ask, isn't OLS "just common, traditional, nearly universal least squares with all the powers, advantages, and attributes there unto pertaining" simply because it is a way to estimate conditional expectation? And now you are back to square one. > you add an assumption not needed with my approach via the Radon-Nikodym result Existence of a unique minimizer of E[(X-f(Y))^2] is a property of L2. You may carryover many properties to L1 by continuity but it isn't one of them. If you don't assume existence of second moments, it is non-trivial to show that min E[(X-f(Y))^2] makes sense if any. I think that you have to put specific constraints on X and Y, but I don't know which are the least restrictive.
  - graycat 4y ago
    > If you don't assume existence of second moments, it is non-trivial to show that min E[(X-f(Y))^2] makes sense if any. To show that E[Y|X] exists, essentially I did that. Really, all need from X is its sigmaalgebra. That is, can do the argument conditioning on a sigmaalgrbra instead of a random variable X. For Y, I didn't include the details back to the Radon-Nikodym theorem, but may not even need that E[Y] is finite. Sure don't need finite "second moments".
    - LudwigNagasena 4y ago