4 ms·
I think there are other statistical methods that would take account of sample size in different ways -- perhaps saying in some cases "you don't have enough data
by jrochkind1 9y ago
I think there are other statistical methods that would take account of sample size in different ways -- perhaps saying in some cases "you don't have enough data to determine validity yet". Among other things that different tests would reveal. p-value is not the only way to analyze your results statistically.
p-values say a certain thing, and are not appropriate to be used as the single universal statistical test for validity, and may likely not even be the best 'default' choice.
I think the message here is indeed that statistical verification/validation is _hard_, and requires actual statistical expertise to know what the tests you are applying mean (and what the assumed pre-requisites of the test are) and apply the right ones. The current practice of "p-value and only p-value, always" is flawed.
(But it's also because the influences of current scientific/academic practice are to motivate people to find the _quickest and cheapest way to get published_, rather than to give yourself additional barriers to publication in pursuit of truth!)
There has been a lot of writing about this in more formal and less cutesy settlers-example ways, this is not an original observation to this blog post. It's in fact been a growing concern, including leading to a recent statement of caution on over- and mis-use of p-values by the American Statistical Association.
http://www.nature.com/news/statisticians-issue-warning-over-misuse-of-p-values-1.19503 http://www.nature.com/news/statisticians-issue-warning-over-...
https://www.sciencedirect.com/science/article/pii/S1063458412007789 https://www.sciencedirect.com/science/article/pii/S106345841...
http://blogs.plos.org/publichealth/2015/06/24/p-values/ http://blogs.plos.org/publichealth/2015/06/24/p-values/
http://www.ejves.com/article/S1078-5884(15)00585-7/pdf http://www.ejves.com/article/S1078-5884(15)00585-7/pdf
http://sphweb.bumc.bu.edu/otlt/mph-modules/ep/ep713_randomerror/EP713_RandomError7.html http://sphweb.bumc.bu.edu/otlt/mph-modules/ep/ep713_randomer...
I like this as a demonstration of one way p-values can be insufficient or lead you wrong, as people have been increasingly discussing.
It would be interesting to apply some of the other suggested statistical tests (confidence intervals?) to the settlers experiment results.
- chimeracoder 9y ago> perhaps saying in some cases "you don't have enough data to determine validity yet Contrary to popular belief, that's actually exactly what a p-value is. A p-value determines whether your sample size is large enough to reject the null hypothesis, given the observed variance in the sample data. Introductory statistics classes don't really explain this properly, because it would require knowledge of continuity theory that most don't have at that stage, but that's really what a p-value is already telling you.
- jrochkind1 9y agoI admit I'm not a statistician (or someone that regularly does quantitative research) and don't really understand what a p-value says (without refreshing myself on it for reading for 30 minutes each time heh). So thanks for correction. But I think many of the researchers applying it don't either, heh. The real point is just that there are other statistical tests that might be appropriate here, and perhaps would be more reliable at catching this. And that there's actually widespread agreement among experts that p-value testing is widely wrongly used. > large enough to reject the null hypothesis Or at any rate, that there's a calculated max 5% chance of the null hypothesis being true, if the assumptions/pre-requisites of the p-value test were actually met. Which is still actually kinda large, 5%, depending. And as I try to think about and confuse myself, may actually be entirely the wrong test as applied here, not sure.
- chimeracoder 9y ago> But I think many of the researchers applying it don't either, heh. As a statistician myself, I'd agree with that statement. I've actually said previously - and only half-joking - that p-values should be banned from research journals. The problem with p-values is that they don't mean what people generally assume they do, and because they look so close to what people want them to mean ("the probability that my conclusions are wrong, given my assumptions and data"), it's very easy to project spurious meaning onto them. In reality, p-values actually provide very little information, and the information they do provide is generally not of relevance. But they're so commonly-used that it's very hard to convince people to use more sophisticated techniques for reporting and modeling information.
- jrochkind1 9y agoI guess it won't change until the 'peer-review' process for every article includes a statistician, and papers actually get rejected for improper or insufficient use of statistics. I think academia these days forces researchers to really care about little except getting grants and getting published (with the former effected by the latter) -- caring about using statistics properly (let alone the actual validity or usefulness of their findings) will hurt rather than help their careers unless it effects one of those two things positively.