3 ms·

Your examples weren't so much contrived as incoherent, to the point where I begin to suspect you don't really understand NHST. "if I try to decide that native

by Keysh 9y ago

Your examples weren't so much contrived as incoherent, to the point where I begin to suspect you don't really understand NHST. "if I try to decide that native Hawaiians are US citizens, and the null hypothesis is that they are, but since only ~0.2% of total US population is native Hawaiian, NHST would conclude that native Hawaiians aren't US citizens." So what is the observation in this case, and what would the corresponding prediction from the null hypothesis be? The observation that "0.2% of the US population is native Hawaiian" has no relation to your claimed null hypothesis at all. The rest of your objection seems like one of those confused arguments trying to rule out basic reductio ad absurdam ("but if X really isn't true, then your arguments about seeing or not seeing the consequences of X have no basis!"). (And the central limit theorem has nothing to do with null hypothesis testing: you can do NHST with completely non-Gaussian statistics.)

justwantaccount 9y ago
Yes, you can conduct NHST without the central limit theorem. However, it's used very widely in NHST. Was there anything wrong with what I said about what a typical t-test usually looked like? My lab would use that approach to do molecular biology. You don't seem to understand my argument, so let me rephrase: The example about Native Hawaiians was meant to highlight the nature of conditional probabilities, and p-values are conditional probabilities. Just because p-values are below some threshold doesn't necessarily mean that the null hypothesis is incorrect and therefore should be rejected. Just because p-values values are high doesn't mean that the null hypothesis should fail to be rejected. P-values do not theoretically give that information. It doesn't even represent the probability of observing that value, since it's a conditional probability - as in, the probability of observing that value given that the null hypothesis is true, not the probability of observing that value. If the p-value is below 0.005, can you scientifically, theoretically conclude that the null hypothesis should be rejected? The probability of seeing a Native Hawaiian person given that the person is a US citizen is below the threshold of 0.005, but does that mean the conditioned part (the US citizen thing) should be rejected? Granted, it's hard to relate that example to actual experiments, but my argument is that p-values don't theoretically give any conclusions either way, and trying to make it "scientific" to draw conclusions by introducing thresholds to a conditional probability, no matter how strict, seems inherently flawed. Using it as a single metric among many, to use it as a tool for exploration makes sense to me. Even to make strong suggestions, sure, especially with all the controls RCTs put in. But to make hard conclusions, as in NHST? The approach itself doesn't have the theoretical power to do so.

by Keysh 9y ago

justwantaccount 9y ago

Yes, you can conduct NHST without the central limit theorem. However, it's used very widely in NHST. Was there anything wrong with what I said about what a typical t-test usually looked like? My lab would use that approach to do molecular biology. You don't seem to understand my argument, so let me rephrase: The example about Native Hawaiians was meant to highlight the nature of conditional probabilities, and p-values are conditional probabilities. Just because p-values are below some threshold doesn't necessarily mean that the null hypothesis is incorrect and therefore should be rejected. Just because p-values values are high doesn't mean that the null hypothesis should fail to be rejected. P-values do not theoretically give that information. It doesn't even represent the probability of observing that value, since it's a conditional probability - as in, the probability of observing that value given that the null hypothesis is true, not the probability of observing that value. If the p-value is below 0.005, can you scientifically, theoretically conclude that the null hypothesis should be rejected? The probability of seeing a Native Hawaiian person given that the person is a US citizen is below the threshold of 0.005, but does that mean the conditioned part (the US citizen thing) should be rejected? Granted, it's hard to relate that example to actual experiments, but my argument is that p-values don't theoretically give any conclusions either way, and trying to make it "scientific" to draw conclusions by introducing thresholds to a conditional probability, no matter how strict, seems inherently flawed. Using it as a single metric among many, to use it as a tool for exploration makes sense to me. Even to make strong suggestions, sure, especially with all the controls RCTs put in. But to make hard conclusions, as in NHST? The approach itself doesn't have the theoretical power to do so.