3 ms·

> perhaps saying in some cases "you don't have enough data to determine validity yet Contrary to popular belief, that's actually exactly what a p-value is. A p

> perhaps saying in some cases "you don't have enough data to determine validity yet Contrary to popular belief, that's actually exactly what a p-value is. A p-value determines whether your sample size is large enough to reject the null hypothesis, given the observed variance in the sample data. Introductory statistics classes don't really explain this properly, because it would require knowledge of continuity theory that most don't have at that stage, but that's really what a p-value is already telling you.

jrochkind1 9y ago
I admit I'm not a statistician (or someone that regularly does quantitative research) and don't really understand what a p-value says (without refreshing myself on it for reading for 30 minutes each time heh). So thanks for correction. But I think many of the researchers applying it don't either, heh. The real point is just that there are other statistical tests that might be appropriate here, and perhaps would be more reliable at catching this. And that there's actually widespread agreement among experts that p-value testing is widely wrongly used. > large enough to reject the null hypothesis Or at any rate, that there's a calculated max 5% chance of the null hypothesis being true, if the assumptions/pre-requisites of the p-value test were actually met. Which is still actually kinda large, 5%, depending. And as I try to think about and confuse myself, may actually be entirely the wrong test as applied here, not sure.
- chimeracoder 9y ago
  > But I think many of the researchers applying it don't either, heh. As a statistician myself, I'd agree with that statement. I've actually said previously - and only half-joking - that p-values should be banned from research journals. The problem with p-values is that they don't mean what people generally assume they do, and because they look so close to what people want them to mean ("the probability that my conclusions are wrong, given my assumptions and data"), it's very easy to project spurious meaning onto them. In reality, p-values actually provide very little information, and the information they do provide is generally not of relevance. But they're so commonly-used that it's very hard to convince people to use more sophisticated techniques for reporting and modeling information.
  - jrochkind1 9y ago
    I guess it won't change until the 'peer-review' process for every article includes a statistician, and papers actually get rejected for improper or insufficient use of statistics. I think academia these days forces researchers to really care about little except getting grants and getting published (with the former effected by the latter) -- caring about using statistics properly (let alone the actual validity or usefulness of their findings) will hurt rather than help their careers unless it effects one of those two things positively.

jrochkind1 9y ago

I admit I'm not a statistician (or someone that regularly does quantitative research) and don't really understand what a p-value says (without refreshing myself on it for reading for 30 minutes each time heh). So thanks for correction. But I think many of the researchers applying it don't either, heh. The real point is just that there are other statistical tests that might be appropriate here, and perhaps would be more reliable at catching this. And that there's actually widespread agreement among experts that p-value testing is widely wrongly used. > large enough to reject the null hypothesis Or at any rate, that there's a calculated max 5% chance of the null hypothesis being true, if the assumptions/pre-requisites of the p-value test were actually met. Which is still actually kinda large, 5%, depending. And as I try to think about and confuse myself, may actually be entirely the wrong test as applied here, not sure.

chimeracoder 9y ago
> But I think many of the researchers applying it don't either, heh. As a statistician myself, I'd agree with that statement. I've actually said previously - and only half-joking - that p-values should be banned from research journals. The problem with p-values is that they don't mean what people generally assume they do, and because they look so close to what people want them to mean ("the probability that my conclusions are wrong, given my assumptions and data"), it's very easy to project spurious meaning onto them. In reality, p-values actually provide very little information, and the information they do provide is generally not of relevance. But they're so commonly-used that it's very hard to convince people to use more sophisticated techniques for reporting and modeling information.
- jrochkind1 9y ago
  I guess it won't change until the 'peer-review' process for every article includes a statistician, and papers actually get rejected for improper or insufficient use of statistics. I think academia these days forces researchers to really care about little except getting grants and getting published (with the former effected by the latter) -- caring about using statistics properly (let alone the actual validity or usefulness of their findings) will hurt rather than help their careers unless it effects one of those two things positively.