4 ms·

Let me replay it to you and see if I understand it, because I don't know if I do: You assume a null hypothesis that (usually) represents the status quo of no i

• Tloewald 12y ago
  No there's a 5% chance the null hypothesis is correct. the probability your theory is correct is unknowable.
  • SpaceManNabs 12y ago
    This is false. The null hypothesis is either correct or false. The veracity of the null hypothesis does not change as long as the experiments are repeated the same.
    • Tloewald 12y ago
      How does this contradict what I said? Enlighten me? I said the theory -- by which I meant the non-null hypothesis -- is not endorsed by a low p value (indeed, this is a major point of the article). A low p-value says "hey they doesn't look like random data" not "your brilliant hypothesis is probably true". The data might not look random because of a methodological error, outright fraud, or a confound. This is particularly important when you consider people looking at data over and over again trying to find "an effect". Theoretically, the tests are supposed to get tougher and tougher each time you examine the data (add one degree of freedom) but in practice this doesn't happen. It doesn't matter much with large data sets, but the social sciences often use datasets where n is roughly 100, and you might only have 20 subjects in a cell.
• yummyfajitas 12y ago
  Not quite. The 5% represents the chance that if the null hypothesis is true, you would draw data at least as extreme as the data you just saw in a repeated experiment. Computing the probability that the data came from the theory stated in the null hypothesis would require a (Baysian) prior. Also, Tloewald's reply is completely and inexorably wrong. Tloewald seems to want a Bayesian answer, which frequentist statistics can't give you.
  • NamTaf 12y ago
    Thanks, that small distinction does make sense to me. I'm surprised I had it as close as I did.
    • qznc 12y ago
      Since I just learned yesterday that statisticians precisely distinguish between "probability" and "confidence", kudos for using it correctly. At least I believe you used them correctly.
  • GhotiFish 12y ago
    could you explain the distinction between saying: "Given the evidence, there is a >=5% probability of the null hypothesis being true" and "There is a >=5% probability that if the null hypothesis were true, that your data would be at least as extreme" The only difference I see is how you avoided saying anything about the null hypothesis, but I don't see how you can avoid saying anything about it. if the h0 were true, then the probability of the result is unlikely, how can you not conclude that h0 is unlikely? What step are you missing other than collecting a preponderance of evidence against it? The article never enters into this distinction. It makes it clear that people misinterpret evidence against the null hypothesis as evidence of the alternative, which is a false dichotomy. I am confused. I also have sympathy for Tloewald at this point.
    • yummyfajitas 12y ago
      Sure. Let H0 be the null hypothesis and D be the data you observed. The first statement is P(H0|D) = 0.05. The second is P(D|H0) = 0.05. The two quantities are related to each other via Bayes rule: P(H0|D)=P(D|H0)P(H0)/P(D) So indeed, as P(D|H0) goes down, so does P(H0|D). But if P(H0)/P(D) is sufficiently large, you can easily have P(H0|D) high while P(D|H0) is low. I too have sympathy for everyone confused by frequentist stats - they tend to answer the exact opposite question that one really wants answered. In contrast, Bayesian stats tend to answer the question that most people ask.
• SapphireSun 12y ago
  Just to add to what the others say, rejecting the null hypothesis is also not evidence that your alternative hypothesis is correct. Often the hypothesis testing framework is stated something like: H0: µ = 0 (null hypothesis) Ha: µ ≠ 0 (alternative hypothesis) When you reject H0, it means that you can be somewhat confident that there was was some kind of distortion in your data that moved the mean away from (in this case) 0. You can create a number of theories that purport to explain this mechanistically, but you'll often need particular setups like a randomized controlled trial that can eliminate alternative explanations. When you've eliminated all the competing reasonable hypotheses, there can only be one. If you can use that hypothesis to make non-obvious predictions, that's further proof that it's right as well. Hypothesis testing is there to tell you when to take an effect seriously, it doesn't tell you whether your explanation is right outside of very carefully constructed circumstances (i.e. where if you see a particular effect, only one theory can explain it).
  • Paradigma11 12y ago
    But the additional theories that you will disprove in further experiments are a subset of Ha so how can disproving H0 not be seen as evidence that Ha is correct. How can H0: the 2 means are equal. and Ha: The 2 means are not equal not encompass the whole universe. For practical purposes NHST is a function that returns either H0 or Ha.
• bartkappenburg 12y ago
  We were forced not to think in probabilities during my econometrics education if we talked about confidence intervals. Say you have a confidence interval of 95% or higher (p value <= 5%), then the right thing to say is: 95% of the variance within the data is explained by the model (you've come up with).
  • Paradigma11 12y ago
    In my opinion the most useful answer would be, for example a mean: The mean is within this interval. We used a model that has this properties, 95% of the... I think it is ridiculous that the main focus always seems to be arcane properties of the statistical algorithm and not the answer that it delivers.
• rcthompson 12y ago
  Others have already explained it well enough, but I'll just add a few important concepts. First, if you remember nothing else, remember this: the p-value has absolutely nothing to do with the alternative hypothesis. In the context of p-values, there is no alternative hypothesis; it doesn't exist. The p-value only says anything about the relationship of your data to the null hypothesis. The second thing to keep in mind is that any null hypothesis likely has lots of assumptions built in to it, and if any of those is violated, the associated p-value is invalid. So you should always be aware of what those assumptions are, and try to verify that your data satisfied then if possible. (Almost no one does this, of course.) For example, the standard t-test assumes, among other things, that your data is normally distributed and that all groups have equal variance. If either of these assumptions is violated, the test is invalid. Lastly, if you're in a situation where you're performing a large number of tests, always correct for multiple testing (i.e. false discovery rate or some similar method). Also, you can take advantage of your large data set to construct a negative control (e.g. by shuffling samples, the exact method will vary) and verify that your chosen statistical test gives a flat distribution of p-values, which is the expected result when all of the null hypotheses are true. If you have an excess of small p-values in the null data set, this indicates your test is producing false positives and is not reliable (presumably because one or more assumptions have been violated).