4 ms·

Here's my non-standard, nutshell, IMHO advice in using probability theory: (1) Random Variables. Go outside. Observe a number. Then that is the value of a r

Here's my non-standard, nutshell, IMHO advice in using probability theory: (1) Random Variables. Go outside. Observe a number. Then that is the value of a random variable. To have a random variable, that the number be random in the sense of unpredictable is not needed. For the phrase and/or criterion "truly random", mostly f'get about it, but we return to that for the subject of random number generation below. So, net, your data, all your data, are the values of random variables. (2) Distributions. Sure, each random variable has a distribution. And there is the Gaussian, uniform, binomial, exponential, Poisson, etc. distributions. Sometimes in practice can use some assumptions to conclude that a random variable has such a known distribution; this is commonly the case for exercises about flipping coins, rolling dice, shuffling cards. For another example, suppose customers are arriving at your Web site. Well maybe the number of arrivals since noon have stationary (over time) independent increments -- maybe you can confirm this just intuitively. Then, presto, bingo, the arrivals are a Poisson process, and the times between arrivals are independent, identically distributed exponential random variables -- see E. Cinlar, Introduction to Stochastic Processes. Further, since might be willing to assume that the arrivals are from many users acting independently, the renewal theorem says that the arrivals will be approximately Poisson, more accurately for more users -- see W. Feller's second volume. Sometimes the central limit theorem can be used to justify a Gaussian assumption. Still, net, in practice, mostly we don't and can't know the distribution. To have much detail on a distribution of one variable takes a lot of data; the joint distribution on several variables takes much more data; the amount of data needed explodes exponentially with the number of joint variables. So, net, don't expect to know or find the distribution. Often you will be able to estimate mean and variance, etc. but not the whole distribution. So, usually need to proceed without knowing distributions. In simple terms: Distributions -- they exist? Yup. We can find them? Nope! (3) Independence. Probability theory is, sure, part of math, but, really, the hugely important, unique feature is the concept of independence. One of the main techniques in applied math is divide and conquer. Well, where you can make an independence assumption lets you so divide. Independence? A simple criterion for practice is, suppose you are given random variables X and Y. You are even given their probability distributions (but NOT their joint probability distribution). Then X and Y are independent if and only if knowing the value of one of them tells you nothing more than you already know about the value of the other one. The hope here is that often in practice you can check this criterion just intuitively from what you know about the real situation. E.g., does a butterfly flapping its wings in Tokyo tell you more about weather tomorrow in NYC? My intuitive guess is that this is a case of independence which means that for predicting weather of NYC tomorrow, we can just f'get about that butterfly. (4) Conditioning. For random variables X and Y, can have the conditional expectation of Y given X, E[Y|X]. Such conditioning is the main way X tells you about Y. Then there is a function f(X) = E[Y|X], and f(X) is the best non-linear least squares estimate of Y. Note that E[E[Y|X]] = E[Y] which means that E[Y|X] is an unbiased estimate of Y. (5) Correlation. If you don't have independence, then likely use the Pearson correlation -- it works like the cosine of an angle. If random variables X and Y are independent, then their Pearson correlation coefficient is 0 -- proof is an easy exercise just from the basic definition and properties of independence. (6) The Classic Limit Theorems. Pay close attention to the central limit theorem (CLT) and the weak and strong laws of large numbers (LLN). The CLT is the main reason we get a Gaussian distribution, and the LLN is the main reason we take averages. (7) Random Number Generation. A sequence of random numbers are to look, for some practical purposes, like a sequence of random variables that are all independent and have uniform distribution on [0,1]. Are they "truly random"? Maybe not. But if they are, then they are independent and identically distributed (i.i.d.) on [0,1] -- and that's all there is to it, and don't have to struggle to say or understand more.

pacala 9y ago
Probability theory expositions, especially for [software] engineers, would be better served if they were well typed. What is the type of a random variable, E[Y|X], E[E[Y|X]]? Hint, a random variable is not a scalar, but rather a function, the probability distribution.
- mturmon 9y ago
  Hmm, a random variable (in the sense of measure theory, as in OP) is indeed a function - but it's not a probability distribution. An R.V. is a measurable function from the sample space into the reals. A probability distribution is a function assigning probabilities to measurable sets, formally, a function from the sigma-algebra into [0,1]. So in particular, a R.V. (like a gaussian) can take on negative values. A probability distribution cannot. Also, the domain of the R.V. is the sample space. But the domain of the probability distribution is the sigma-algebra over that sample space.
  - graycat 9y ago
    A distribution is a real valued function of a real variable. The domain of the function is the whole real line. Note: Below, borrowing from D. Knuth's TeX, we use the underscore character '_' to denote the start of a subscript. Details: For real valued random variable X, probability measure P, and the set of real numbers R, the cumulative distribution of X is the function F_X: R --> R where, for x in R, F_X(x) = P(X <= x). If F_X is differentiable, then the probability density distribution of X is the real valued function of a real variable f_X: R --> R where, for all x in R, f_X(x) = d/dx F_X(x) where d/dx is the calculus derivative. For the connections with sigma algebras, that is more advanced than most engineers care about, but here are some of the details: For real numbers a and b with a < b, there is the open interval (a,b) = {x|a < x < b} A topology on R is a collection of subsets regarded as open and that satisfy the axioms for a topology -- the sets in a topology are closed under finite intersections and arbitrary unions and both R and the empty set are open. The usual topology on R is the smallest (a short argument shows that this "smallest" is well defined) topology that has each open interval an element of the topology -- right, the topology regards the open intervals as open. The usual reason to discuss a topology is to have a means of defining continuous functions, a means more general than from the usual "for each epsilon greater than zero, there exists a delta greater than zero such that ..." or in terms of limits of sequences. Indeed, there are advanced situations where we can use topologies to define continuous functions where epsilon and delta and where converging sequences don't work. If curious, look up Moore-Smith convergence, nets, and filters or just Kelley, General Topology. Well, a sigma algebra is like a topology, that is, is a collection of subsets: A sigma algebra is closed under countable unions and relative complements. Right, we avoid uncountable unions because otherwise we will get stuck in a big mud hole. It is an early exercise that there are no countably infinite sigma algebras. The reason for sigma algebras is to permit defining a measurable function, that is, one where we can apply the Lebesgue integration theory. The integral of calculus is due to B. Riemann and is the Riemann integral. W. Rudin, Principles of Mathematical Analysis shows that for a continuous real valued function with domain a compact set (closed and bounded, where closed is the complement of an open set) has a Riemann integral. Well in this case, the Lebesgue integral gives the same numerical answer -- same thing. The advantage of the Lebesgue approach is that the function can be even bizarre and its domain can be much more general. Indeed, in probability theory, expectation is just the Lebesgue integral. In simple terms, Riemann partitioned on the X axis, and Lebesgue partitioned on the Y axis. Well, given the usual topology on R, we can ask for the smallest sigma algebra on R that has the topology as a subset. That sigma algebra is the Borel sets of R. Uh, Lebesgue was a student of E. Borel. In Rudin will find the Heine-Borel theorem. So, in probability theory, we have a sample space. Each point in the sample space is a trial, i.e., essentially a real world experimental trial (note: really our attitude is that in all the universe we see only one such trial -- if this seems far out, then blame the Russians, e.g., A. Kolmogorov, E. Dynkin, etc.!). Well, an event is a subset of the sample space, that is, a set of trials. So, flip a coin. Let H be the event, the set of all trials where, that the coin comes up heads. Well, to apply Lebesgue's theory of integration, we want the set of all events to be a sigma algebra. Then a probability measure is a measure in the sense of Lebesgue's measure theory, that is, a real value function, in the case of probability taking values in [0,1], and with domain the sigma algebra of events. So, for the event H, we can ask for the probability of H, that is, P(H), which is a number in [0,1]. For a fair coin tossed by an honest member of the FBI we have P(H) = 1/2. Then a real valued random variable X is just a real valued function with domain the sample space and also measurable: This part about being measurable is that for each Borel set A, a subset of R, the set of all trials w so that X(w) is in A is an event, that is, an element of the sigma algebra on the set of trials. That is, the inverse image under X of the Borel sets are events, elements of the sigma algebra on the sample space. So, with X measurable in this way, we have a near perfect shot at defining the expectation of X, E[X]. For this we have a little two step dance: First we look at X^+ ('^' denotes a superscript) where X is >= 0. So, X^+ is the positive part of X. Similarly X^- is the negative part of X. So, X = X^+ + X^-. Uh, I'm working quickly from memory; maybe we want X^- to be -X where X < 0 and 0 otherwise. So both X^+ and X^- are >= 0 and we have X = X*+ - X^-. Either way. Well, we can use Lebesgue's theory to integrate X^+ and X^-. Biggie stuff: The X need only be measurable, and that admits lots of really wildly bizarre functions. We've got great generality, and that's good to have in various limiting arguments. Uh, we like limiting arguments because that is our main way to approximate which our main way to being healthy, wealthy, and wise! So the Lebesgue integral of X^+ we write as E[X^+]. Similarly for X^-. Now no way do we want to be subtracting one infinity from another since permitting that would trash the usual laws of arithmetic. So, for our second step, in the case X^- >= 0, if at least one of E[X^+] and E[X^-] is finite, then we define E[X] = E[X^+] - E[X^-]. Now we've defined expectation ("average") of a real random variable X. Our definition is just the Lebesgue integral. For the Lebesgue integral, we wanted the sigma algebras. On the real line we can consider the sigma algebra of Lebesgue measurable sets; that's larger than the Borel sets. Then we just ask, assume, assert, believe, ..., that our random variables are measurable with respect to the sigma algebra of Lebesgue sets and the sigma algebra of the events. Uh, right, Lebesgue measure on R assigns Lebesgue measure b - a to interval (a,b) and extends from there. Fine details are in various texts by Rudin, Royden, etc. That's the beginnings of the role of sigma algebras in advanced approaches to probability, statistics, and stochastic processes. It turns out, the sigma algebra approach is for several parts of what we want in probability, much nicer, e.g., for defining independence and conditional expectation. E.g., if we want to know that some set of uncountably infinitely many random variables are independent, we can. Same for conditioning on uncountably infinitely many random variables, e.g., the past history of a stochastic process.

pacala 9y ago

Probability theory expositions, especially for [software] engineers, would be better served if they were well typed. What is the type of a random variable, E[Y|X], E[E[Y|X]]? Hint, a random variable is not a scalar, but rather a function, the probability distribution.

mturmon 9y ago
Hmm, a random variable (in the sense of measure theory, as in OP) is indeed a function - but it's not a probability distribution. An R.V. is a measurable function from the sample space into the reals. A probability distribution is a function assigning probabilities to measurable sets, formally, a function from the sigma-algebra into [0,1]. So in particular, a R.V. (like a gaussian) can take on negative values. A probability distribution cannot. Also, the domain of the R.V. is the sample space. But the domain of the probability distribution is the sigma-algebra over that sample space.
- graycat 9y ago
  A distribution is a real valued function of a real variable. The domain of the function is the whole real line. Note: Below, borrowing from D. Knuth's TeX, we use the underscore character '_' to denote the start of a subscript. Details: For real valued random variable X, probability measure P, and the set of real numbers R, the cumulative distribution of X is the function F_X: R --> R where, for x in R, F_X(x) = P(X <= x). If F_X is differentiable, then the probability density distribution of X is the real valued function of a real variable f_X: R --> R where, for all x in R, f_X(x) = d/dx F_X(x) where d/dx is the calculus derivative. For the connections with sigma algebras, that is more advanced than most engineers care about, but here are some of the details: For real numbers a and b with a < b, there is the open interval (a,b) = {x|a < x < b} A topology on R is a collection of subsets regarded as open and that satisfy the axioms for a topology -- the sets in a topology are closed under finite intersections and arbitrary unions and both R and the empty set are open. The usual topology on R is the smallest (a short argument shows that this "smallest" is well defined) topology that has each open interval an element of the topology -- right, the topology regards the open intervals as open. The usual reason to discuss a topology is to have a means of defining continuous functions, a means more general than from the usual "for each epsilon greater than zero, there exists a delta greater than zero such that ..." or in terms of limits of sequences. Indeed, there are advanced situations where we can use topologies to define continuous functions where epsilon and delta and where converging sequences don't work. If curious, look up Moore-Smith convergence, nets, and filters or just Kelley, General Topology. Well, a sigma algebra is like a topology, that is, is a collection of subsets: A sigma algebra is closed under countable unions and relative complements. Right, we avoid uncountable unions because otherwise we will get stuck in a big mud hole. It is an early exercise that there are no countably infinite sigma algebras. The reason for sigma algebras is to permit defining a measurable function, that is, one where we can apply the Lebesgue integration theory. The integral of calculus is due to B. Riemann and is the Riemann integral. W. Rudin, Principles of Mathematical Analysis shows that for a continuous real valued function with domain a compact set (closed and bounded, where closed is the complement of an open set) has a Riemann integral. Well in this case, the Lebesgue integral gives the same numerical answer -- same thing. The advantage of the Lebesgue approach is that the function can be even bizarre and its domain can be much more general. Indeed, in probability theory, expectation is just the Lebesgue integral. In simple terms, Riemann partitioned on the X axis, and Lebesgue partitioned on the Y axis. Well, given the usual topology on R, we can ask for the smallest sigma algebra on R that has the topology as a subset. That sigma algebra is the Borel sets of R. Uh, Lebesgue was a student of E. Borel. In Rudin will find the Heine-Borel theorem. So, in probability theory, we have a sample space. Each point in the sample space is a trial, i.e., essentially a real world experimental trial (note: really our attitude is that in all the universe we see only one such trial -- if this seems far out, then blame the Russians, e.g., A. Kolmogorov, E. Dynkin, etc.!). Well, an event is a subset of the sample space, that is, a set of trials. So, flip a coin. Let H be the event, the set of all trials where, that the coin comes up heads. Well, to apply Lebesgue's theory of integration, we want the set of all events to be a sigma algebra. Then a probability measure is a measure in the sense of Lebesgue's measure theory, that is, a real value function, in the case of probability taking values in [0,1], and with domain the sigma algebra of events. So, for the event H, we can ask for the probability of H, that is, P(H), which is a number in [0,1]. For a fair coin tossed by an honest member of the FBI we have P(H) = 1/2. Then a real valued random variable X is just a real valued function with domain the sample space and also measurable: This part about being measurable is that for each Borel set A, a subset of R, the set of all trials w so that X(w) is in A is an event, that is, an element of the sigma algebra on the set of trials. That is, the inverse image under X of the Borel sets are events, elements of the sigma algebra on the sample space. So, with X measurable in this way, we have a near perfect shot at defining the expectation of X, E[X]. For this we have a little two step dance: First we look at X^+ ('^' denotes a superscript) where X is >= 0. So, X^+ is the positive part of X. Similarly X^- is the negative part of X. So, X = X^+ + X^-. Uh, I'm working quickly from memory; maybe we want X^- to be -X where X < 0 and 0 otherwise. So both X^+ and X^- are >= 0 and we have X = X*+ - X^-. Either way. Well, we can use Lebesgue's theory to integrate X^+ and X^-. Biggie stuff: The X need only be measurable, and that admits lots of really wildly bizarre functions. We've got great generality, and that's good to have in various limiting arguments. Uh, we like limiting arguments because that is our main way to approximate which our main way to being healthy, wealthy, and wise! So the Lebesgue integral of X^+ we write as E[X^+]. Similarly for X^-. Now no way do we want to be subtracting one infinity from another since permitting that would trash the usual laws of arithmetic. So, for our second step, in the case X^- >= 0, if at least one of E[X^+] and E[X^-] is finite, then we define E[X] = E[X^+] - E[X^-]. Now we've defined expectation ("average") of a real random variable X. Our definition is just the Lebesgue integral. For the Lebesgue integral, we wanted the sigma algebras. On the real line we can consider the sigma algebra of Lebesgue measurable sets; that's larger than the Borel sets. Then we just ask, assume, assert, believe, ..., that our random variables are measurable with respect to the sigma algebra of Lebesgue sets and the sigma algebra of the events. Uh, right, Lebesgue measure on R assigns Lebesgue measure b - a to interval (a,b) and extends from there. Fine details are in various texts by Rudin, Royden, etc. That's the beginnings of the role of sigma algebras in advanced approaches to probability, statistics, and stochastic processes. It turns out, the sigma algebra approach is for several parts of what we want in probability, much nicer, e.g., for defining independence and conditional expectation. E.g., if we want to know that some set of uncountably infinitely many random variables are independent, we can. Same for conditioning on uncountably infinitely many random variables, e.g., the past history of a stochastic process.