4 ms·

This is worth reading: The "Unreliable Small-Sample" Misconception https://www.dailykos.com/stories/2008/11/9/656465/- https://www.dailykos.com/stories/2008/1

by arikr 8y ago

This is worth reading: The "Unreliable Small-Sample" Misconception https://www.dailykos.com/stories/2008/11/9/656465/- https://www.dailykos.com/stories/2008/11/9/656465/- > My point here is that arguing that the sample is inaccurate simply because of how large or small it is is not correct. The size of the sample actually makes very little difference in how applicable the results are to the population. It simply makes the range of possible results smaller as the size of the sample increases. I've noticed a trend on reddit and HN where commenters will point out small sample sizes, often with the direct or indirect implication that this makes the study mostly invalid. I believe that's false - studies with small sample sizes can be very valid. The smaller the sample size the larger the effect size needs to be for it to be a statistically significant result, but a small sample size by itself isn't bad. And I think that the internet would be able to better appreciate research if more people were aware of the small sample size misconception.

SamReidHughes 8y ago
There's also temporal sample size: you can infer that A causes B if B happens very soon after A, and nothing else around that time might cause B. For example, if A is exposure to a chemical and B is an allergic reaction. Edit: A great example is that stomach ulcer guy.
stewbrew 8y ago
Since the article doesn't mention test power (and should it be for the reason of rebuttal of that Pearson-Neyman concept) I don't think it's a worth read. Sorry. "studies with small sample sizes can be very valid." They could be with a certain probability but (unless you believe in a very strong prior) you can't know with sufficient certainty.
devwastaken 8y ago
Sample size is related to the conclusions you can draw. If their goal is that ibuprofen has this effect, then 15 can be enough given the specific effects they are getting. If their goal is more complex, say that ibuprofen reduces fertility, that's be a lot harder if the effect wasn't massive and well reproduceable.
- yomly 8y ago
  I find this topic hard to comment on specifically, but I think you are probably on the right lines about specificity of the observed phenomenon being of key here. Using a ridiculous example "for a sample set of 10, the author found that exposure to fire causes burns" would hopefully not draw similar sampling concerns but as I don't know this subject matter deeply, I can't comment whether it is true for this case
drb91 8y ago
Yes, this is why my anecdotes are proof only I see reality accurately. In seriousness, this is a lazy claim when you could have actually addressed the study in question. Since you did not, I assume you have no supporting observations.
celrod 8y ago
Andrew Gelman put it very well here: https://andrewgelman.com/2017/02/06/not-kill-statistical-significance-makes-stronger-fallacy/ https://andrewgelman.com/2017/02/06/not-kill-statistical-sig... Or, if you want to some it all up in a single graph: https://andrewgelman.com/2014/11/17/power-06-looks-like-get-used/ https://andrewgelman.com/2014/11/17/power-06-looks-like-get-... (although your experiments hopefully have power much greater than 0.06) The gist of the point: If your sample size is small, significant results are likely to (sometimes vastly) overestimate the true effect size, and there is a non-negligible chance that the effect size is actually estimated in the wrong direction.
opportune 8y ago
Small sample size makes publication bias much more likely as well

by arikr 8y ago

SamReidHughes 8y ago

There's also temporal sample size: you can infer that A causes B if B happens very soon after A, and nothing else around that time might cause B. For example, if A is exposure to a chemical and B is an allergic reaction. Edit: A great example is that stomach ulcer guy.

stewbrew 8y ago

Since the article doesn't mention test power (and should it be for the reason of rebuttal of that Pearson-Neyman concept) I don't think it's a worth read. Sorry. "studies with small sample sizes can be very valid." They could be with a certain probability but (unless you believe in a very strong prior) you can't know with sufficient certainty.

devwastaken 8y ago

Sample size is related to the conclusions you can draw. If their goal is that ibuprofen has this effect, then 15 can be enough given the specific effects they are getting. If their goal is more complex, say that ibuprofen reduces fertility, that's be a lot harder if the effect wasn't massive and well reproduceable.

yomly 8y ago
I find this topic hard to comment on specifically, but I think you are probably on the right lines about specificity of the observed phenomenon being of key here. Using a ridiculous example "for a sample set of 10, the author found that exposure to fire causes burns" would hopefully not draw similar sampling concerns but as I don't know this subject matter deeply, I can't comment whether it is true for this case

drb91 8y ago

Yes, this is why my anecdotes are proof only I see reality accurately. In seriousness, this is a lazy claim when you could have actually addressed the study in question. Since you did not, I assume you have no supporting observations.

celrod 8y ago

Andrew Gelman put it very well here: https://andrewgelman.com/2017/02/06/not-kill-statistical-significance-makes-stronger-fallacy/ https://andrewgelman.com/2017/02/06/not-kill-statistical-sig... Or, if you want to some it all up in a single graph: https://andrewgelman.com/2014/11/17/power-06-looks-like-get-used/ https://andrewgelman.com/2014/11/17/power-06-looks-like-get-... (although your experiments hopefully have power much greater than 0.06) The gist of the point: If your sample size is small, significant results are likely to (sometimes vastly) overestimate the true effect size, and there is a non-negligible chance that the effect size is actually estimated in the wrong direction.

opportune 8y ago

Small sample size makes publication bias much more likely as well