5 ms·

>> In statistics, σ refers to an area under the normal distribution defined in terms of standard deviations (1σ = 1 standard deviation). > No. Yes. The reason

• graycat 13y ago
  > Yes. The reason the various sigma values have the numerical values they have is because they represent integrals under the normal distribution, either one-tailed or two-tailed. No. I defined sigma, that is, standard deviation, fully precisely and correctly. The normal distribution has nothing to do with that definition. And the standard deviation is a number, just a number, just as I defined it as σ = E[(X - E[X])^2]^(1/2) which clearly is just a number and not an area. Or, for random variable X with cumulative distribution F_X, that is, for real number x, P(X <= x) = F_X(x) we have, with notation from D. Knuth's TeX, that σ^2 = \int (x - E[X])^2 dF_X(x) This integral need not be in the sense of Riemann (i.e., freshman calculus) because dF_X is a measure on the real line; so, the integral is in the sense of measure theory (see any of Rudin, Real and Complex Analysis; Royden, Real Analysis; Halmos, Measure Theory; Loève, Probability Theory). Sigma is defined for any random variable X or its distribution provided that E[X] exists and is finite. Again, a "normal" or Gaussian assumption is not necessary. So, sigma is defined for discrete distributions, the uniform distribution, the Poisson distribution, the exponential distribution, etc. For a random variable X, if don't know its distribution, then can't say what the numerical value of its standard distribution is. Moreover if for some random variable X that has a standard deviation want to know, say, the probability P( -5σ <= X <= 5σ) then that is an area and need the distribution of X to find the numerical value. All that is 100%, completely, totally, absolutely true. That's what σ or standard deviation is. In particular, the statement "In statistics, σ refers to an area under the normal distribution defined in terms of standard deviations (1σ = 1 standard deviation)." is flatly false. The field of statistics has no such statement or convention. Yes, if want to be really sloppy, make some assumptions not clearly stated, and have some conventions for identifying some things in special ways, e.g., that sigma is an area, then can do so. Maybe some parts of physics do this. I do remember when I was studying physics the prof handed out a little book on how errors were handled in physics. The book was a sloppy mess and one of the reasons I lost respect for accuracy and precision in physics and majored in math instead. It is true that about 100 years ago some fields of study, especially parts of psychology and much of education, concluded that the Gaussian distribution was some universal law of data handed down by God. Well, God did no such thing. Still, some people in educational statistics believe that student test scores should have a Gaussian distribution and, if the scores do not have such a distribution, will, from many such scores, find the empirical distribution and, then, transform the scores so that the distribution is closely Gaussian. Maybe physics drank that Kool Aid that all experimental errors of course, as given by God, have a Gaussian distribution, that is "a perfect bell curve", and, then, yes, can get "68% of the data is within one standard deviation of the mean, 95% is within two, and so on.", and that σ has a particular numerical value and regard standard deviation as an area. Yes, maybe this is physics but it is very sloppy thinking and not mathematics, probability, statistics, or anything from God. Of course, even with a Gaussian assumption, standard deviation does not have a particular numerical value. Instead, for Gaussian random variable X with E[X} = 0 each of P( -σ <= X <= σ) P( -2σ <= X <= 2σ) P( -5σ <= X <= 5σ) has a particular numerical value. Sorry 'bout that.
  • sillysaurus3 13y ago
    Maybe physics drank that Kool Aid that all experimental errors of course, as given by God, have a Gaussian distribution, that is "a perfect bell curve", and, then, yes, can get "68% of the data is within one standard deviation of the mean, 95% is within two, and so on.", and that σ has a particular numerical value and regard standard deviation as an area. Yes, maybe this is physics but it is very sloppy thinking That's very interesting! What are some practical or pragmatic advantages of your way of thinking compared to the standard (sloppy) way of thinking? I don't mean to put you on the spot; I genuinely would love to know. For example, what are some statistical problems which the standard (sloppy) mental model would have a tough time solving, but which your mental model would be able to yield tools for?
    • lutusp 13y ago
      The OP's argument may be that the normal distribution is often applied to datasets with which it has no innate connection and that can only mislead the applier. This is correct -- there are any number of examples where the statistical reasoning behind the normal distribution is wildly inappropriate to the data set being analyzed. The classic example is a dataset with anomalous outliers, say, a cohort in which one person died in childbirth and another who lived to the age of 120. Average age: 60. But the physicists who apply this method to their LHC data, and those who apply it to the newer finding of gravitational waves, know exactly what they're doing. The data being analyzed are entirely appropriate to this method, and the conclusions being drawn are sound.
    • graycat 13y ago
      Mary teaches calculus to 20 students, and Bob also teaches calculus to 15 students. A standardized test is given to all the students, all 35. So, suppose we assume a null hypothesis that all 35 student scores are independent random variables with the same distribution, that is, (1) Mary and Bob are equally good as teachers and (2) their students are equally well qualified. If the students had been appropriately randomly assigned to Mary and Bob, then maybe we can believe (2) so that only (1) is in question. So, we are going to test (1), that is, that Mary and Bob are equally good as teachers. This hypothesis, that is, that Mary and Bob are equally good is called a null hypothesis since it assumes that there is no effect between Mary's class and Bob's class (even though Mary is teaching more students than Bob). So, here is how we do our test: We throw all 35 scores into a pot, stir the pot energetically, pull out 20 in one bowl for Mary and put the other 15 scores into a bowl for Bob. Then we average the scores in each bowl and take the difference in the averages, say, Mary's average minus Bob's average. Then we repeat this many times -- for this might want to use a computer with a good random number generator. This process is sometimes called 'resampling'. It's also possible to argue that what is going on is a finite group of measure preserving transformations that, thus, can yield what we are doing more intuitively. Physicists like symmetries that result in conservation laws, but here we have symmetries resulting in an hypothesis tests. So, we get the empirical distribution of the differences in the averages. Then we look at the difference in the actual averages, that is, from the actual students of Mary and Bob. Call this difference X. Now we see where X is in the empirical distribution of differences we found. If X is out in the tails with probability, say, 1%, then either (A) Mary and Bob are equally good as teachers, that is, we accept the null hypothesis, and we have observed something that should happen 1% of the time or less or (B) Mary and Bob are not equally good as teachers and we reject the null hypothesis and conclude that there is a difference in teaching between Mary and Bob. If the 1% is too small to believe in, then we accept (B) and pop a Champaign cork for the better teacher. Here we made no assumptions at all about the probability distributions of the scores. So, our hypothesis test does not assume a distribution and is distribution free or, as is sometimes said, non-parametric (because we did not assume a distribution, say, Gaussian with parameters, e.g., mean and variance). Look, Ma, no standard deviations! The 1% is he significance level of our hypothesis test and is the probability of rejecting the null hypothesis when it is true and, thus, is the probability of Type I error. Oh, consider a large server farm or network. Suppose we identify 10,000 systems we want to monitor for health and wellness and detect problems never seen before. Suppose from one of the 10,000 systems, we consider one. Suppose from this system we receive data 100 times a second on each of 12 numerical variables. Suppose the server farm is supposed to be fairly stable and we collect such data for, say, 3 months. Call this history data. Maybe the machine learning people would call this training data. Whatever. Now we can construct a 12 dimensional, distribution-free hypothesis test where the null hypothesis is that the system is healthy and also select our false alarm rate (probability of Type I error) in small steps over a wide range. So, we have a multi-dimensional, distribution-free hypothesis test. Such are rare, but, really now we have a large class of them. Yes, we use a group of measure preserving transformations. As I recall, back in 1999 there was a paper on such things in Information Sciences. I wouldn't call that paper machine learning, but maybe some people would. Some of what is interesting in the paper is how the heck to know and adjust the false alarm rate, that is, the probability of Type I error. Again, look, Ma, no standard deviations or Gaussian assumptions.