6 ms·
A lot of the COVID19 stuff only has a population size (NOT EVEN sample size) of like, 15. Ex: J&J's blood clot issue. People make conclusions off of tinier sam
by dragontamer 5y ago
A lot of the COVID19 stuff only has a population size (NOT EVEN sample size) of like, 15. Ex: J&J's blood clot issue.
People make conclusions off of tinier sample sizes. There are plenty of scientific studies with just 50 or 100 people, but you need to keep the small numbers in mind.
As long as sampling was done correctly, then small sample size won't matter much !!
- bumby 5y ago>There are plenty of scientific studies with just 50 or 100 people, but you need to keep the small numbers in mind. The converse also needs to be held in mind as well. Very large studies are more likely to display statistically significant findings. Your p-value is really just a test of your sample size, so as the study grows your p-value tends to shrink. Get a large enough sample and you'll almost always find statistical significance. Just referencing a p-value isn't enough, it needs to be read in the context of the study size (whether large or small). Edit: For those downvoting, here is a much better and eloquent explanation: https://twitter.com/daniela_witten/status/1312180959609974784 https://twitter.com/daniela_witten/status/131218095960997478...
- loonster 5y agoThe size of the sample depends on how rare of an event that is being studied. For something that happens 80% of the time, 300 is a very reasonable number. Something that happens 1/1000 or 999/1000 would need a much larger sample size.
- bumby 5y agoYes, I agree. But that’s to the same point. The p-value is still a measure of whether your sample size is large enough to capture a small probability event in the context of assessing the null hypothesis
- spywaregorilla 5y agoThis is a bad explanation of p values. Of the start: > If you’re testing whether mean=0 and actually the truth is that mean=0.000000001, and if you have a large enough sample size, then YOU WILL GET A TINY P-VALUE. Think about this for a second. Does it make sense that the results of my statistical test would change if I measured in terragrams vs. micrograms? No, it doesn't, because it's not the numerical difference between null hypothesis and sample mean, it's the difference relative to standard deviation. If the effect size is 0.000000001 and the standard deviation is 0.0000000000000001 then this is an enormous, important effect. If the standard deviation is 0.1, then this is a really small and unimportant effect. It can still be observed correctly with a sufficient sample size though because you've stated it's a real effect to begin with. You will absolutely not get a tiny p value just because your sample size is large. That is wrong. You will be able to detect very small effects, but if there is no difference, you are more likely to draw the correct conclusion with 1,000,000 samples than with 100 samples. The point the author here is trying to convey but is not getting clearly, is that a large sample size can detect an effect that is practically useless and call it significant. p-values tell you if it seems probable, given your data, if there is a difference between the two datasets. They make no claims about whether the difference is material. Example python code to do prove p values are not likelier to be smaller just due to large sample sizes, given no actual difference: import random from scipy.stats import ttest_ind num_tests = 10000 res = [] for test in range(num_tests): n = 100 a = [random.random() for _ in range(n)] b = [random.random() for _ in range(n)] res = res + [ttest_ind(a,b)[1] <= 0.05] print(f"10,000 tests with n=200 {sum(res)/ num_tests}") num_tests = 1000 res = [] for test in range(num_tests): n = 100000 a = [random.random() for _ in range(n)] b = [random.random() for _ in range(n)] res = res + [ttest_ind(a,b)[1] <= 0.05] print(f"10,000 tests with n=20,000 {sum(res)/ num_tests}")
- bumby 5y agoThis is a good explanation, thank you. >You will be able to detect very small effects, but if there is no difference, you are more likely to draw the correct conclusion with 1,000,000 samples than with 100 samples. I believe that this is to the tweet author's point where they stated that in the real-world a null hypothesis never exactly holds true. So you can detect even the tiniest of variance with a large enough sample. >given no actual difference I think the error here is that there is a priori knowledge of no difference, where the author is stating that in real-life scenarios there will always be some differences even if being too minute to be of practical significance. We can fabricate "no difference" in simulations but in real experimental design there will almost always be variance, even if it's just an artifact of the measurement process rather than being causal from the independent variable(s). Whether or not the differences statistically significant depends on the experiment design, to include sample size. So while I understand your valid point I think the author's claim was more about the practical application of statistics rather than the mathematical precision as it relates with simulated examples. But I could be misinterpreting.
- arcticbull 5y agoI agree with you broadly, though would the J&J sample size not be about 21.4 million? 15 is just the number of adverse events right?
- dragontamer 5y agoWhen we get into "total population" statistics, we kinda leave the realm of statistics lol. There's 15 documented adverse events, but we don't really know how many adverse events there were in total. If we simply study those 15 adverse events, then your population study is only those 15 events (with a hell-of-a-lot of selection bias). ---------- But I'm mostly thinking about how one person I know thinks: "Did you hear about these 15 cases and even one of them died! Clearly J&J is unsafe to take!!!!" For that kind of conclusion, its like they've focused their mind entirely on 15 cases, as if their "population of study" was 15. Its obviously bad logic, but maybe I wasn't very clear in my original post that I was referencing some bad logic there. --------- But there's plenty of 50-person or 200-person studies that ARE properly done, and the science is solid. Especially when we're looking at say Phase 1 / Phase 2 style trials of safety and/or efficacy (the earlier stage trials before a Phase 3 trial is conducted). Its not a big enough sample to be "certain" of the results, but its a big enough sample to "provide evidence", possibly in conjuction with other studies. ---------- In many cases: having 5 or 6 studies with 50-random people each and performing a meta-study across them (especially if all 5 of those studies were of high quality sampling) would lead to a more robust conclusion than just one 1000-person study done slightly incorrectly (ex: the 1000-person "study" is a voluntary telephone survey or some other low-quality study)
- NotSammyHagar 5y agoSorry but I disagree. The key point is that over a very large population, millions, only 15 people got it. It's just wrong to say it was computed over only 15 people. This study referenced up above had the ~285 or whatever people who answered the survey, not that much info. ~285/350 is not that much data. Anecdotally and based on my own experience I'd agree that the burnout is high, probably over 50%.
- danenania 5y agoPeople make conclusions based on tiny sample sizes when the results fit their biases. Most studies are worthless due to small sample sizes and confounding variables, and most “conclusions” are based on cherry-picked outliers. The fact that it’s common to do this doesn’t make it responsible or correct.
- anthonypasq 5y agohave you taken intro statistics? all small sample sizes do is require your effect to be more pronounced to still be statistically significant.
- danenania 5y agoAh, then I guess that's why so few "statistically significant" studies end up being contradicted by further research...
- myfavoritedog 5y agoYeah, given the track record of surveys/studies, you’re better off assuming the opposite conclusions that they claim to validate.
- mactrey 5y agoYou make it sound like those studies were bad because the sample size was too small. In reality bad study design or having an unrepresentative sample are more common issues.
- danenania 5y agoI think they go hand-in-hand. People naively apply intro-level statistics to messy real-world problems and then think they need an orders of magnitude smaller sample than they would need to even hint at the possibility of an interesting effect.
- 5y ago