5 ms·

Each experimental outcome has an associated probability that the outcome resulted from chance, not the thing being measured. This arises from the default scient

by lutusp 13y ago

Each experimental outcome has an associated probability that the outcome resulted from chance, not the thing being measured. This arises from the default scientific precept that is called the "null hypothesis", the assumption that a given measurement was produced by chance, not nature. Scientists use a value designated by p to describe the probability that a result arose by chance rather than design. In the social sciences, a p value of 0.01 - 0.05 is common, which means a result can be explained by chance with a 1% to 5% probability. As one moves through fields of greater rigor and seriousness, the p value required to declare a discovery becomes smaller. In experimental physics, 5σ (five sigma) has become the standard. In statistics, σ refers to an area under the normal distribution defined in terms of standard deviations (1σ = 1 standard deviation). Experimental physics uses a one-tailed 5σ value, which is quite strict -- it has a numerical value of about 3×10-7. What this means is that an experimental result must be solid enough (and/or be repeated often enough) that the probability that it arose from chance is equal to or less than 5σ. More here: http://blogs.scientificamerican.com/observations/2012/07/17/five-sigmawhats-that/ http://blogs.scientificamerican.com/observations/2012/07/17/...

graycat 13y ago
> In statistics, σ refers to an area under the normal distribution defined in terms of standard deviations (1σ = 1 standard deviation). No. The Gaussian (normal distribution) is not involved. Sigma is just the symbol for standard deviation, that is, the square root of the variance. So, for random variable X where with expectation E[X] that exists and is finite (in practice a weak assumption), the standard deviation is square root of E[(X - E[X])^2]. So, the standard deviation is just a number, a measure of how 'spread out' the distribution is, and not an "area". Then 5σ is just 5 times the standard deviation.
- sigterm 13y ago
  Are you sure? I'm pretty confident in this case the assumption is that the distribution is normal. With no knowledge of the shape of distribution, the 5 sigma bound is extremely weak (one given by Chebyshev's inequality [0], in this case, ~4%). Prove me wrong though, as I'm not a physicist or mathematician. [0] http://en.wikipedia.org/wiki/Chebyshev's_inequality#Sharpness_of_bounds http://en.wikipedia.org/wiki/Chebyshev's_inequality#Sharpnes...
  - lutusp 13y ago
    You're correct. Here's a short article that makes the point that what's being discussed are areas of the normal distribution: http://blogs.scientificamerican.com/observations/2012/07/17/five-sigmawhats-that/ http://blogs.scientificamerican.com/observations/2012/07/17/...
    - Myrmornis 13y ago
      No, I think you're wrong. See my comment above. Note that nowhere in the article you link does it say there's an assumption of normality. Doubtless the journalist or whoever chose the picture of th normal distribution was a bit confused also. Edit: oh, no, actually I think you're right :). I thought it was referring to the location of the test statistic in its null distribution. But it seems it's a scale for measuring p-values. This explains it clearly: What does "five sigma" mean? It means that the results would occur by chance alone as rarely as a value sampled from a Gaussian distribution would be five standard deviations from the mean. http://www.graphpad.com/www/data-analysis-resource-center/blog/statistical-significance-defined-using-the-five-sigma-standard/ http://www.graphpad.com/www/data-analysis-resource-center/bl...
      - lutusp 13y ago
        All that aside, remember that the context is a nontechnical person asking what 5σ means in statistical analysis in physics. That greatly narrows the possible interpretations. By the way, this same conversation took place after the LHC Higgs anouncement -- the same five-sigma standard for discovery, and the same detailed discussion of what that means. > But it seems it's a scale for measuring p-values. Yes, and that was a point I made in my reply to the OP -- that the context assumed an association with p-values, which in turn assume a normal distribution.
        Myrmornis 13y ago
        Sorry, I'm either not understanding or I still am not convinced you are correct. p-values don't assume a normal distribution. A p-value is the probability of generating an equal or more extreme value of your test statistic under the null hypothesis. Normal distributions are nowhere in sight. The only way that the Normal distribution comes into play is that physicists are measuring the smallness of their p-values by stating a number of standard deviations from the Normal mean that would have the same p-value. I'm finding this discussion helpful by the way. I have worked in applied statistics but not in any fields which use this "sigma" scale or would talk about "sigma values".
- lutusp 13y ago
  >> In statistics, σ refers to an area under the normal distribution defined in terms of standard deviations (1σ = 1 standard deviation). > No. Yes. The reason the various sigma values have the numerical values they have is because they represent integrals under the normal distribution, either one-tailed or two-tailed. Therefore in the present context sigma values represent definite integrals of the normal distribution. > Then 5σ is just 5 times the standard deviation. Yes, correct, except that the present conversation is about p-values and the meaning of sigma in this specific context. Five sigma has the value it does because it represents one minus the area between 0 and five sigma of the normal distribution, i.e. the one-tailed meaning of five sigma. Link: http://blogs.scientificamerican.com/observations/2012/07/17/five-sigmawhats-that/ http://blogs.scientificamerican.com/observations/2012/07/17/... Quote: "A graph of the normal distribution, showing 3 standard deviations on either side of the mean µ. A five-sigma observation corresponds to data even further from the mean." Quote: "For particle physics, the sigma used is the standard deviation arising from a normal distribution of data, familiar to us as a bell curve. In a perfect bell curve, 68% of the data is within one standard deviation of the mean, 95% is within two, and so on." Picture http://blogs.scientificamerican.com/observations/files/2012/07/400px-Standard_deviation_diagram.svg_.png http://blogs.scientificamerican.com/observations/files/2012/...
  - graycat 13y ago
    > Yes. The reason the various sigma values have the numerical values they have is because they represent integrals under the normal distribution, either one-tailed or two-tailed. No. I defined sigma, that is, standard deviation, fully precisely and correctly. The normal distribution has nothing to do with that definition. And the standard deviation is a number, just a number, just as I defined it as σ = E[(X - E[X])^2]^(1/2) which clearly is just a number and not an area. Or, for random variable X with cumulative distribution F_X, that is, for real number x, P(X <= x) = F_X(x) we have, with notation from D. Knuth's TeX, that σ^2 = \int (x - E[X])^2 dF_X(x) This integral need not be in the sense of Riemann (i.e., freshman calculus) because dF_X is a measure on the real line; so, the integral is in the sense of measure theory (see any of Rudin, Real and Complex Analysis; Royden, Real Analysis; Halmos, Measure Theory; Loève, Probability Theory). Sigma is defined for any random variable X or its distribution provided that E[X] exists and is finite. Again, a "normal" or Gaussian assumption is not necessary. So, sigma is defined for discrete distributions, the uniform distribution, the Poisson distribution, the exponential distribution, etc. For a random variable X, if don't know its distribution, then can't say what the numerical value of its standard distribution is. Moreover if for some random variable X that has a standard deviation want to know, say, the probability P( -5σ <= X <= 5σ) then that is an area and need the distribution of X to find the numerical value. All that is 100%, completely, totally, absolutely true. That's what σ or standard deviation is. In particular, the statement "In statistics, σ refers to an area under the normal distribution defined in terms of standard deviations (1σ = 1 standard deviation)." is flatly false. The field of statistics has no such statement or convention. Yes, if want to be really sloppy, make some assumptions not clearly stated, and have some conventions for identifying some things in special ways, e.g., that sigma is an area, then can do so. Maybe some parts of physics do this. I do remember when I was studying physics the prof handed out a little book on how errors were handled in physics. The book was a sloppy mess and one of the reasons I lost respect for accuracy and precision in physics and majored in math instead. It is true that about 100 years ago some fields of study, especially parts of psychology and much of education, concluded that the Gaussian distribution was some universal law of data handed down by God. Well, God did no such thing. Still, some people in educational statistics believe that student test scores should have a Gaussian distribution and, if the scores do not have such a distribution, will, from many such scores, find the empirical distribution and, then, transform the scores so that the distribution is closely Gaussian. Maybe physics drank that Kool Aid that all experimental errors of course, as given by God, have a Gaussian distribution, that is "a perfect bell curve", and, then, yes, can get "68% of the data is within one standard deviation of the mean, 95% is within two, and so on.", and that σ has a particular numerical value and regard standard deviation as an area. Yes, maybe this is physics but it is very sloppy thinking and not mathematics, probability, statistics, or anything from God. Of course, even with a Gaussian assumption, standard deviation does not have a particular numerical value. Instead, for Gaussian random variable X with E[X} = 0 each of P( -σ <= X <= σ) P( -2σ <= X <= 2σ) P( -5σ <= X <= 5σ) has a particular numerical value. Sorry 'bout that.
ghshephard 13y ago
And here is an entertaining cartoon about p > 0.05 http://xkcd.com/882/ http://xkcd.com/882/
judk 13y ago
> The probability that it arose from chance is equal to or less than 5σ That's not what p value means. The p value is the probability that a high-variance random effect (centered around an average behavior of "nothing interesting") would yield a result as extreme as the observation, assuming that random distribution. You need Bayes theorem and highly subjective assumptons if you want to derive a posterior probability that an observed result was drawn by chance from a sample with a boring/interesting mean.
- lutusp 13y ago
  >> The probability that it arose from chance is equal to or less than 5σ > That's not what p value means. Yes, I know. Verbal shorthand and some associated risk of being misinterpreted.

by lutusp 13y ago

graycat 13y ago

> In statistics, σ refers to an area under the normal distribution defined in terms of standard deviations (1σ = 1 standard deviation). No. The Gaussian (normal distribution) is not involved. Sigma is just the symbol for standard deviation, that is, the square root of the variance. So, for random variable X where with expectation E[X] that exists and is finite (in practice a weak assumption), the standard deviation is square root of E[(X - E[X])^2]. So, the standard deviation is just a number, a measure of how 'spread out' the distribution is, and not an "area". Then 5σ is just 5 times the standard deviation.

sigterm 13y ago
Are you sure? I'm pretty confident in this case the assumption is that the distribution is normal. With no knowledge of the shape of distribution, the 5 sigma bound is extremely weak (one given by Chebyshev's inequality [0], in this case, ~4%). Prove me wrong though, as I'm not a physicist or mathematician. [0] http://en.wikipedia.org/wiki/Chebyshev's_inequality#Sharpness_of_bounds http://en.wikipedia.org/wiki/Chebyshev's_inequality#Sharpnes...
- lutusp 13y ago
  You're correct. Here's a short article that makes the point that what's being discussed are areas of the normal distribution: http://blogs.scientificamerican.com/observations/2012/07/17/five-sigmawhats-that/ http://blogs.scientificamerican.com/observations/2012/07/17/...
  - Myrmornis 13y ago
    No, I think you're wrong. See my comment above. Note that nowhere in the article you link does it say there's an assumption of normality. Doubtless the journalist or whoever chose the picture of th normal distribution was a bit confused also. Edit: oh, no, actually I think you're right :). I thought it was referring to the location of the test statistic in its null distribution. But it seems it's a scale for measuring p-values. This explains it clearly: What does "five sigma" mean? It means that the results would occur by chance alone as rarely as a value sampled from a Gaussian distribution would be five standard deviations from the mean. http://www.graphpad.com/www/data-analysis-resource-center/blog/statistical-significance-defined-using-the-five-sigma-standard/ http://www.graphpad.com/www/data-analysis-resource-center/bl...
    - lutusp 13y ago
      All that aside, remember that the context is a nontechnical person asking what 5σ means in statistical analysis in physics. That greatly narrows the possible interpretations. By the way, this same conversation took place after the LHC Higgs anouncement -- the same five-sigma standard for discovery, and the same detailed discussion of what that means. > But it seems it's a scale for measuring p-values. Yes, and that was a point I made in my reply to the OP -- that the context assumed an association with p-values, which in turn assume a normal distribution.
      - Myrmornis 13y ago
        Sorry, I'm either not understanding or I still am not convinced you are correct. p-values don't assume a normal distribution. A p-value is the probability of generating an equal or more extreme value of your test statistic under the null hypothesis. Normal distributions are nowhere in sight. The only way that the Normal distribution comes into play is that physicists are measuring the smallness of their p-values by stating a number of standard deviations from the Normal mean that would have the same p-value. I'm finding this discussion helpful by the way. I have worked in applied statistics but not in any fields which use this "sigma" scale or would talk about "sigma values".
lutusp 13y ago
>> In statistics, σ refers to an area under the normal distribution defined in terms of standard deviations (1σ = 1 standard deviation). > No. Yes. The reason the various sigma values have the numerical values they have is because they represent integrals under the normal distribution, either one-tailed or two-tailed. Therefore in the present context sigma values represent definite integrals of the normal distribution. > Then 5σ is just 5 times the standard deviation. Yes, correct, except that the present conversation is about p-values and the meaning of sigma in this specific context. Five sigma has the value it does because it represents one minus the area between 0 and five sigma of the normal distribution, i.e. the one-tailed meaning of five sigma. Link: http://blogs.scientificamerican.com/observations/2012/07/17/five-sigmawhats-that/ http://blogs.scientificamerican.com/observations/2012/07/17/... Quote: "A graph of the normal distribution, showing 3 standard deviations on either side of the mean µ. A five-sigma observation corresponds to data even further from the mean." Quote: "For particle physics, the sigma used is the standard deviation arising from a normal distribution of data, familiar to us as a bell curve. In a perfect bell curve, 68% of the data is within one standard deviation of the mean, 95% is within two, and so on." Picture http://blogs.scientificamerican.com/observations/files/2012/07/400px-Standard_deviation_diagram.svg_.png http://blogs.scientificamerican.com/observations/files/2012/...
- graycat 13y ago
  > Yes. The reason the various sigma values have the numerical values they have is because they represent integrals under the normal distribution, either one-tailed or two-tailed. No. I defined sigma, that is, standard deviation, fully precisely and correctly. The normal distribution has nothing to do with that definition. And the standard deviation is a number, just a number, just as I defined it as σ = E[(X - E[X])^2]^(1/2) which clearly is just a number and not an area. Or, for random variable X with cumulative distribution F_X, that is, for real number x, P(X <= x) = F_X(x) we have, with notation from D. Knuth's TeX, that σ^2 = \int (x - E[X])^2 dF_X(x) This integral need not be in the sense of Riemann (i.e., freshman calculus) because dF_X is a measure on the real line; so, the integral is in the sense of measure theory (see any of Rudin, Real and Complex Analysis; Royden, Real Analysis; Halmos, Measure Theory; Loève, Probability Theory). Sigma is defined for any random variable X or its distribution provided that E[X] exists and is finite. Again, a "normal" or Gaussian assumption is not necessary. So, sigma is defined for discrete distributions, the uniform distribution, the Poisson distribution, the exponential distribution, etc. For a random variable X, if don't know its distribution, then can't say what the numerical value of its standard distribution is. Moreover if for some random variable X that has a standard deviation want to know, say, the probability P( -5σ <= X <= 5σ) then that is an area and need the distribution of X to find the numerical value. All that is 100%, completely, totally, absolutely true. That's what σ or standard deviation is. In particular, the statement "In statistics, σ refers to an area under the normal distribution defined in terms of standard deviations (1σ = 1 standard deviation)." is flatly false. The field of statistics has no such statement or convention. Yes, if want to be really sloppy, make some assumptions not clearly stated, and have some conventions for identifying some things in special ways, e.g., that sigma is an area, then can do so. Maybe some parts of physics do this. I do remember when I was studying physics the prof handed out a little book on how errors were handled in physics. The book was a sloppy mess and one of the reasons I lost respect for accuracy and precision in physics and majored in math instead. It is true that about 100 years ago some fields of study, especially parts of psychology and much of education, concluded that the Gaussian distribution was some universal law of data handed down by God. Well, God did no such thing. Still, some people in educational statistics believe that student test scores should have a Gaussian distribution and, if the scores do not have such a distribution, will, from many such scores, find the empirical distribution and, then, transform the scores so that the distribution is closely Gaussian. Maybe physics drank that Kool Aid that all experimental errors of course, as given by God, have a Gaussian distribution, that is "a perfect bell curve", and, then, yes, can get "68% of the data is within one standard deviation of the mean, 95% is within two, and so on.", and that σ has a particular numerical value and regard standard deviation as an area. Yes, maybe this is physics but it is very sloppy thinking and not mathematics, probability, statistics, or anything from God. Of course, even with a Gaussian assumption, standard deviation does not have a particular numerical value. Instead, for Gaussian random variable X with E[X} = 0 each of P( -σ <= X <= σ) P( -2σ <= X <= 2σ) P( -5σ <= X <= 5σ) has a particular numerical value. Sorry 'bout that.

ghshephard 13y ago

And here is an entertaining cartoon about p > 0.05 http://xkcd.com/882/ http://xkcd.com/882/

judk 13y ago

> The probability that it arose from chance is equal to or less than 5σ That's not what p value means. The p value is the probability that a high-variance random effect (centered around an average behavior of "nothing interesting") would yield a result as extreme as the observation, assuming that random distribution. You need Bayes theorem and highly subjective assumptons if you want to derive a posterior probability that an observed result was drawn by chance from a sample with a boring/interesting mean.

lutusp 13y ago
>> The probability that it arose from chance is equal to or less than 5σ > That's not what p value means. Yes, I know. Verbal shorthand and some associated risk of being misinterpreted.