"some correlation between x and y that makes us money" Well, there is a relationship between X and Y but it might not be an actual 'correlation'. Why? Becaus

"some correlation between x and y that makes us money" Well, there is a relationship between X and Y but it might not be an actual 'correlation'. Why? Because the relationship might be non-linear. For an easier explanation, consider just real random X and Y. That is, they are just real numbers (but 'random variables'). Okay, time out, what is a 'random variable'? It's general: Go in a lab, measure a number. That's a random variable. The specific number we measured is one of others that we 'might have' measured (observed). So, we did one 'trial' among many other trials we might have done. With the modern view (modern since Kolmogorov's paper in 1933), all we ever observe in all the universe is just some one trial; still we imagine that there are other trials we might have observed. Useless? Not really: It may be that we have other measurements we can make, that is, other random variables. Maybe we have reason to believe that these other random variables have the same distribution (from all those other trials we never see) but are independent of Y. So, we can use these other random variables to say things about Y. That's the 'modern' setup. Can any such mathematical objects actually exist? Yes. Classic measure theory provides the foundation. That's much of what Kolmogorov wrote in 1933. Back to X and Y. Okay, go into Excel and get a 3D graph. Get lots of pairs (x,y) and plot them on the X-Y plane. Then up the Z axis, build a histogram. Have a lot of data so that we get a nice, accurate histogram. So, the histogram will look like a mountain range. That's the joint probability density of X and Y if we scale the mountain range to have total volume 1. Okay, we are told that we just measured X and want to know about Y. So, we go to our Excel 3D graph, find X on its axis (too many X's in this explanation -- there is better notation but it would take more explanation), and then see what the histogram says about Y. So, at X we get the distribution of Y 'given' X (after we scale to 1). That's the 'conditional density distribution' of Y given X. The cumulative version is written P{Y <= y|X) and read the 'conditional probability that Y is <= y given the value of X'. Well, with that conditional distribution, either the density version or the cumulative version, we can find the expectation of Y given X, and that is written E[Y|X]. Then there is a real valued function of a real variable f such that E[Y|X] = f(X). With all the assumptions clear and also all the details, that is the classic Radon-Nikodym theorem. There is a famous proof by von Neumann -- i.e., he took it seriously -- in Rudin's book, and there the theorem is a big thing; in modern probability and stochastic processes, it's a huge thing. E.g., Markov means that the past and future are conditionally independent given the present. A martingale is the present is the same as the conditional expectation of the future given the past. Every stochastic process is the sum of a martingale and a predictable process. Every martingale that doesn't run off to infinity has to converge to a point. Get very deep into modern mathematical finance and get a lot on conditional expectations, Markov processes, and martingales. As in the classic Halmos-Savage paper, sufficient statistics is just an application of the Radon-Nikodym theorem. E.g., for Gaussian data, sample mean and sample standard deviation are sufficient statistics which means that anything that can be done with all the data can also be done just as well starting with just the sample mean and sample standard deviation -- so long a lot of 'big data'! Order statistics are always sufficient. Net, conditional expectation from the Radon-Nikodym theorem is a big thing. So, we can use f(X) = E[Y|X] to approximate Y. Maybe we'd like the least squares approximation. Coming right up: Let our approximating function be g. Then our error is Y - g(X). Then our squared error is (Y - g(X))^2. Then on average our squared error is E[(Y -g(X))^2], and we want to pick g, non-linear, to minimize that. Presto: The g we want is just f. That is g(X) = f(X) = E[Y}X] minimizes E[(Y - g(X))^2], and that's the very best we can do with any function, including non-linear, of X. So, if we want to approximate Y, the very best (least squares) use we can make of X is just f(X). Moreover, our estimate of Y is 'unbiased', that is, E{Y] = E[f(X)] = E[E[Y|X}}. So, we have the best non-linear, minimum variance, unbiased estimator of Y. We should ask for more? Then for f(X), for the discrete approximation, we can get that just from that 3D graph in Excel that we developed from just cross tabulation. So, net, if we have enough data, then cross-tabulation is the discrete approximation of the best estimation we can do, that is, cross tabulation is the one statistical method to rule them all, if we have enough data. So, this is a reason for 'big data'. "exponential cost"? Well, for the average I mentioned, the 100 times n, etc., the cost is just quadratic, but it still gets too large too fast. If all we really want to consider is just correlation, then we can use linear statistical methods that are basically a perpendicular projection where we get to use the Pythagorean theorem.