5 ms·

I'm not a scientist, but I'm curious if any of you know: why do these important studies use such small N? I think its pretty obvious that a small N is not stat

I'm not a scientist, but I'm curious if any of you know: why do these important studies use such small N? I think its pretty obvious that a small N is not statistically significant, yet for this study, or the myriad of COVID-19 studies, the N is usually < 20. It's just so frustrating that professional scientists just completely ignore this statistical fact. Why not use use N >= 100 or not at all. It just confuses things in my opinion.

jdm2212 6y ago
The paper doesn't claim to be proving efficacy, just safety and feasibility. But university PR office gotta PR.
superhuzza 6y ago
>why do these important studies use such small N Generally because bigger N means the study is more complicated logistically ( cost, finding participants, whatever). And if the results are promising you can always get more funding to move on to bigger trials. Nobody is going to pay for you to recruit 1000 participants if you can't show an effect size with a smaller number first. > I think its pretty obvious that a small N is not statistically significant Well this totally depends on the effect size. You can't just dismiss results because N is small, that ignores how probability works. >Why not use use N >= 100 or not at all. Because N > 100 is a totally arbitrary value. The size of N that you need is totally dependent on the effect size, not an arbitrary value. You'd miss lots of real but small effects by having an N of only 100. You'd miss out on lots of real but large effects which can be observed at N < 100. Basically this approach would severely limit the branches of research we can do, for a questionable gain in statistical reliability.
- grugagag 6y ago
  > And if the results are promising you can always get more funding to move on to bigger trials. This seems to be such a case. I wonder whether it was set from the beginning with more funding in mind.
gshdg 6y ago
Because you start with a small inexpensive trial first in order to determine whether the larger, more expensive trial is a) safe, and b) worth investing in. What's unfortunate is that the research gets published at this stage beyond the scientific community that understands how inconclusive it is. Because in the long run that just contributes to the erosion of public trust in the scientific method.
renewiltord 6y ago
Is it really so obvious that small N is statistically insignificant? Stat sig is a method to empirically improve knowledge. Well, if it's intuitive that you need high N to get empirical knowledge, let's try an intuitive example. Say you're a tribal chieftain. One of your men (one among thousands) finds a berry bush. Another eats a berry from the bush. He promptly dies. You're a scientific tribe, though, and you know N must be high before you conclude the berries are poisonous. So, one of the other guys eats another berry. He, too, dies immediately. The process repeats until it's you considering the bush. How does the situation change if the number of people before you was 19 vs 21?
nabla9 6y ago
You issue is with people upvoting things into HN frontpage, not with N. There are multiple reasons why small N is not bad thing. Firstly, multi year research projects are not silent for years or decades before they deliver final paper. They publish interim research, technical reports, exploratory research or preliminary research to clarify issues and fine tune the direction of research. You do something and you publish it. Secondly, multiple studies where N is small can be more reliable than one big study with large N. There are errors other than statistical errors. Thirdly. Effect size is much more important than N. N=1 can win you a Nobel price if the effect size is big enough. Grow a mice that weights 10 kg and N being 1 or very small number is good enough. There are many studies where N is tens of thousands (p < 0.001) but the effect size is so small that the result is not scientifically interesting. Finally, statistics is not the final arbiter of importance of the study. It has become standard to add it into every publication, no matter how small. But researchers don't focus on that. --- Everyone wants to participate in the discussion and most people in comment sections know something about statistics but little to nothing about the subject, so you always get the 'correlation is not causation', 'small N' so it's irrelevant talking points.
markroseman 6y ago
You're probably confusing science as a top-down enterprise rather than a field that works bottom-up (where "bottom" would be grad students). Evidence emerges and builds from below.
mcnamaratw 6y ago
I'm not a med sci person, but I bet if we look at the algorithms for getting to large N, all of them are expensive and all of them visit small N first. Anyway the results for any N seem very likely to be misleading to laypeople. (Laypeople means "almost everyone." For example I'm in photonics. In psychiatry I'm a layperson.) I agree with nabla's comment that the real problem is our inability as laypeople to properly understand (or even properly upvote) scientific work.
mcguire 6y ago
I'm sorry your text is grey; this is a valid question. These are preliminary studies: someone says, "Let me try something," it appears to work, and they report what they saw, with the resources they had. As a result, someone organizes follow-on studies with more resources, which costs much more. At that point, the effect usually disappears, due to the greater power of the more careful study.

jdm2212 6y ago

The paper doesn't claim to be proving efficacy, just safety and feasibility. But university PR office gotta PR.

superhuzza 6y ago

>why do these important studies use such small N Generally because bigger N means the study is more complicated logistically ( cost, finding participants, whatever). And if the results are promising you can always get more funding to move on to bigger trials. Nobody is going to pay for you to recruit 1000 participants if you can't show an effect size with a smaller number first. > I think its pretty obvious that a small N is not statistically significant Well this totally depends on the effect size. You can't just dismiss results because N is small, that ignores how probability works. >Why not use use N >= 100 or not at all. Because N > 100 is a totally arbitrary value. The size of N that you need is totally dependent on the effect size, not an arbitrary value. You'd miss lots of real but small effects by having an N of only 100. You'd miss out on lots of real but large effects which can be observed at N < 100. Basically this approach would severely limit the branches of research we can do, for a questionable gain in statistical reliability.

grugagag 6y ago
> And if the results are promising you can always get more funding to move on to bigger trials. This seems to be such a case. I wonder whether it was set from the beginning with more funding in mind.

gshdg 6y ago

Because you start with a small inexpensive trial first in order to determine whether the larger, more expensive trial is a) safe, and b) worth investing in. What's unfortunate is that the research gets published at this stage beyond the scientific community that understands how inconclusive it is. Because in the long run that just contributes to the erosion of public trust in the scientific method.

renewiltord 6y ago

Is it really so obvious that small N is statistically insignificant? Stat sig is a method to empirically improve knowledge. Well, if it's intuitive that you need high N to get empirical knowledge, let's try an intuitive example. Say you're a tribal chieftain. One of your men (one among thousands) finds a berry bush. Another eats a berry from the bush. He promptly dies. You're a scientific tribe, though, and you know N must be high before you conclude the berries are poisonous. So, one of the other guys eats another berry. He, too, dies immediately. The process repeats until it's you considering the bush. How does the situation change if the number of people before you was 19 vs 21?

nabla9 6y ago

You issue is with people upvoting things into HN frontpage, not with N. There are multiple reasons why small N is not bad thing. Firstly, multi year research projects are not silent for years or decades before they deliver final paper. They publish interim research, technical reports, exploratory research or preliminary research to clarify issues and fine tune the direction of research. You do something and you publish it. Secondly, multiple studies where N is small can be more reliable than one big study with large N. There are errors other than statistical errors. Thirdly. Effect size is much more important than N. N=1 can win you a Nobel price if the effect size is big enough. Grow a mice that weights 10 kg and N being 1 or very small number is good enough. There are many studies where N is tens of thousands (p < 0.001) but the effect size is so small that the result is not scientifically interesting. Finally, statistics is not the final arbiter of importance of the study. It has become standard to add it into every publication, no matter how small. But researchers don't focus on that. --- Everyone wants to participate in the discussion and most people in comment sections know something about statistics but little to nothing about the subject, so you always get the 'correlation is not causation', 'small N' so it's irrelevant talking points.

markroseman 6y ago

You're probably confusing science as a top-down enterprise rather than a field that works bottom-up (where "bottom" would be grad students). Evidence emerges and builds from below.

mcnamaratw 6y ago

I'm not a med sci person, but I bet if we look at the algorithms for getting to large N, all of them are expensive and all of them visit small N first. Anyway the results for any N seem very likely to be misleading to laypeople. (Laypeople means "almost everyone." For example I'm in photonics. In psychiatry I'm a layperson.) I agree with nabla's comment that the real problem is our inability as laypeople to properly understand (or even properly upvote) scientific work.

mcguire 6y ago

I'm sorry your text is grey; this is a valid question. These are preliminary studies: someone says, "Let me try something," it appears to work, and they report what they saw, with the resources they had. As a result, someone organizes follow-on studies with more resources, which costs much more. At that point, the effect usually disappears, due to the greater power of the more careful study.