5 ms·

Nice, really nice, introduction to statistical hypothesis testing, nicer, sorry 'bout that, than is common in teaching of statistics. Nice work by physics.

nonbel 8y ago
What are you saying is nice?
- graycat 8y ago
  Some unusual clarity on how hypothesis testing works or should work. For one more point, we pick the null hypothesis so that we can when we collect data we can calculate probabilities of that data assuming the null hypothesis is true. So, what null hypotheses we can consider are both enabled and constrained by the assumptions to calculate the probabilities given the data.
  - nonbel 8y ago
    >"we pick the null hypothesis so that we can when we collect data we can calculate probabilities of that data assuming the null hypothesis is true." That is an awful reason to choose a null hypothesis. You should choose a null hypothesis you derived from your theory so you can see if your theory is capable of making accurate predictions.
    - beojan 8y ago
      Your null hypothesis shouldn't have anything to do with your theory. It should always be "nothing interesting (or new) is happening". Your alternative hypothesis is derived from your theory.
      - nonbel 8y ago
        Yep, that's the "switcheroo" that reverses the long successful logic of science. I'd suggest starting with Meehl 1967: >"In the physical sciences, the usual result of an improvement in experimental design, instrumentation, or numerical mass of data, is to increase the difficulty of the “observational hurdle” which the physical theory of interest must successfully surmount; whereas, in psychology and some of the allied behavior sciences, the usual effect of such improvement in experimental precision is to provide an easier hurdle for the theory to surmount." https://meehl.dl.umn.edu/sites/g/files/pua1696/f/074theorytestingparadox.pdf https://meehl.dl.umn.edu/sites/g/files/pua1696/f/074theoryte...
        beojan 8y ago
        But if you didn't do that, you would be assuming any crazy theory you think of is true, and require overwhelming evidence to refute it. > I'd suggest starting with Meehl 1967 Your very quote states that this whole idea works for physics, and is describing that it doesn't work in psychology and social sciences. I'm confused by this: To take the IQ example, if you take the set of all people in a country, for instance, and randomly select two sets of 5 people, you will find that their average IQ differs by some amount. If you instead select two sets of 50000 people, the average IQs should be a lot closer (to each other, and to the true average). This is by definition true, since IQ is by definition normally distributed. If that's not what you see, you have a methodological problem.
        nonbel 8y ago
        By "physics" he clearly is referring to testing the quantitative predictions of models. The first sentence of the abstract: >"Because physical theories typically predict numerical values, an improvement in ex-perimental precision reduces the tolerance range and hence increases corroborability." I don't know the historical details but it would appear that since 1967 the NHST virus has proceeded to begin its infestation of physics. Now physicists are acting more and more like psychologists (focusing on testing a null model rather than the predictions of their models; which is the first phase of the illness) and will reap the same consequences of stagnation, etc. Lowering the threshold to 3e-7 instead of 5e-2 just makes this slower and more expensive. >"To take the IQ example, if you take the set of all people in a country, for instance, and randomly select two sets of 5 people, you will find that their average IQ differs by some amount. If you instead select two sets of 50000 people, the average IQs should be a lot closer (to each other, and to the true average). This is by definition true, since IQ is by definition normally distributed. If that's not what you see, you have a methodological problem." I'm not sure why you are talking about standard errors of random samples from a population... He was just giving stats 101 background for philosophers so he could get to this point: "While no competent psychologist is unaware of this obvious distinction between a substantive psychological theory T and a statistical hypothesis H implied by it, in practice there is a tendency to conflate the substan- tive theory with the statistical hypothesis, thereby illicitly conferring upon T some-what the same degree of support given H by a successful refutation of the null hypothesis. Hence the investigator, upon finding an observed difference which has an extremely small probability of occurring on the null hypothesis, gleefully records the tiny probability number “p < .001,” and there is a tendency to feel that the extreme smallness of this probability of a Type I error is somehow transferable to a small probability of “making a theoretical mistake.”
    - graycat 8y ago
      You are at least partly correct. My statement was correct, but it was intended only as an explanation of some of the mechanics of the applied math and applied probability of getting a doable hypothesis test. The role of the test in a scientific theory was deeper than my statement! In the main, novel hypothesis test I invented and published, there was more than just what I outlined and you criticized, more along the lines you mentioned: That "more" was something like "if the next observation is a long way from anything we've seen before, then maybe it is an anomaly?". The actual work of the math in the paper showed, intuitively, what "a long way" meant and, in particular, for the math, how actually to calculate, along the lines of my statement, the hypothesis test false alarm rate. So, in short, in nearest neighbors anomaly detection, with meager assumptions, we actually can calculate false alarm rate and, thus, get an hypothesis test. And, then, we are not limited strictly to the single nearest but can do some weighted sum over several of the relatively near. Right, such things appear to be necessarily close to multi-variate probability density estimation. But such estimation commonly requires so much data that it is not a good, direct way to calculate or estimate probabilities; curiously we can do well with the calculations without actually getting a local density estimate although in a sense we are close to doing that. The other remarks here are even better, e.g., get to how to do science, maybe at the LHC or some such. Good and beyond what I tried to say.

nonbel 8y ago

What are you saying is nice?

graycat 8y ago
Some unusual clarity on how hypothesis testing works or should work. For one more point, we pick the null hypothesis so that we can when we collect data we can calculate probabilities of that data assuming the null hypothesis is true. So, what null hypotheses we can consider are both enabled and constrained by the assumptions to calculate the probabilities given the data.
- nonbel 8y ago
  >"we pick the null hypothesis so that we can when we collect data we can calculate probabilities of that data assuming the null hypothesis is true." That is an awful reason to choose a null hypothesis. You should choose a null hypothesis you derived from your theory so you can see if your theory is capable of making accurate predictions.
  - beojan 8y ago
    Your null hypothesis shouldn't have anything to do with your theory. It should always be "nothing interesting (or new) is happening". Your alternative hypothesis is derived from your theory.
    - nonbel 8y ago
      Yep, that's the "switcheroo" that reverses the long successful logic of science. I'd suggest starting with Meehl 1967: >"In the physical sciences, the usual result of an improvement in experimental design, instrumentation, or numerical mass of data, is to increase the difficulty of the “observational hurdle” which the physical theory of interest must successfully surmount; whereas, in psychology and some of the allied behavior sciences, the usual effect of such improvement in experimental precision is to provide an easier hurdle for the theory to surmount." https://meehl.dl.umn.edu/sites/g/files/pua1696/f/074theorytestingparadox.pdf https://meehl.dl.umn.edu/sites/g/files/pua1696/f/074theoryte...
      - beojan 8y ago
        But if you didn't do that, you would be assuming any crazy theory you think of is true, and require overwhelming evidence to refute it. > I'd suggest starting with Meehl 1967 Your very quote states that this whole idea works for physics, and is describing that it doesn't work in psychology and social sciences. I'm confused by this: To take the IQ example, if you take the set of all people in a country, for instance, and randomly select two sets of 5 people, you will find that their average IQ differs by some amount. If you instead select two sets of 50000 people, the average IQs should be a lot closer (to each other, and to the true average). This is by definition true, since IQ is by definition normally distributed. If that's not what you see, you have a methodological problem.
        nonbel 8y ago
        By "physics" he clearly is referring to testing the quantitative predictions of models. The first sentence of the abstract: >"Because physical theories typically predict numerical values, an improvement in ex-perimental precision reduces the tolerance range and hence increases corroborability." I don't know the historical details but it would appear that since 1967 the NHST virus has proceeded to begin its infestation of physics. Now physicists are acting more and more like psychologists (focusing on testing a null model rather than the predictions of their models; which is the first phase of the illness) and will reap the same consequences of stagnation, etc. Lowering the threshold to 3e-7 instead of 5e-2 just makes this slower and more expensive. >"To take the IQ example, if you take the set of all people in a country, for instance, and randomly select two sets of 5 people, you will find that their average IQ differs by some amount. If you instead select two sets of 50000 people, the average IQs should be a lot closer (to each other, and to the true average). This is by definition true, since IQ is by definition normally distributed. If that's not what you see, you have a methodological problem." I'm not sure why you are talking about standard errors of random samples from a population... He was just giving stats 101 background for philosophers so he could get to this point: "While no competent psychologist is unaware of this obvious distinction between a substantive psychological theory T and a statistical hypothesis H implied by it, in practice there is a tendency to conflate the substan- tive theory with the statistical hypothesis, thereby illicitly conferring upon T some-what the same degree of support given H by a successful refutation of the null hypothesis. Hence the investigator, upon finding an observed difference which has an extremely small probability of occurring on the null hypothesis, gleefully records the tiny probability number “p < .001,” and there is a tendency to feel that the extreme smallness of this probability of a Type I error is somehow transferable to a small probability of “making a theoretical mistake.”
  - graycat 8y ago
    You are at least partly correct. My statement was correct, but it was intended only as an explanation of some of the mechanics of the applied math and applied probability of getting a doable hypothesis test. The role of the test in a scientific theory was deeper than my statement! In the main, novel hypothesis test I invented and published, there was more than just what I outlined and you criticized, more along the lines you mentioned: That "more" was something like "if the next observation is a long way from anything we've seen before, then maybe it is an anomaly?". The actual work of the math in the paper showed, intuitively, what "a long way" meant and, in particular, for the math, how actually to calculate, along the lines of my statement, the hypothesis test false alarm rate. So, in short, in nearest neighbors anomaly detection, with meager assumptions, we actually can calculate false alarm rate and, thus, get an hypothesis test. And, then, we are not limited strictly to the single nearest but can do some weighted sum over several of the relatively near. Right, such things appear to be necessarily close to multi-variate probability density estimation. But such estimation commonly requires so much data that it is not a good, direct way to calculate or estimate probabilities; curiously we can do well with the calculations without actually getting a local density estimate although in a sense we are close to doing that. The other remarks here are even better, e.g., get to how to do science, maybe at the LHC or some such. Good and beyond what I tried to say.