3 ms·

> It's my understanding that when designing a study, you look to strengthen statistical power, which is determined by a combination of a) significance criterion

by Osmium 11y ago

> It's my understanding that when designing a study, you look to strengthen statistical power, which is determined by a combination of a) significance criterion (p values, confidence intervals) b) magnitude of the effect and c) sample size. Increasing the sample size decreases sampling error. I don't disagree with this at all. You always want to increase your sample size if possible. But as you also say, the magnitude of the effect you're observing is also important; if it's very strong, then it can be revealed by a small sample size. An absurd example could be "cyanide capsules kill (n=2)". Obviously, the smaller the sample size, the harder it is to draw reliable conclusions, but I wouldn't think n=19 is necessarily too small. I think many of phase I clinical trials aren't much more than that. More importantly, mentioning a "small" sample size makes a nice soundbite, and is often used to discredit studies. These studies might be good or they might not be, but talking about sample size can be used as a "thought-terminating cliche". Perhaps a person is looking for evidence that reinforces their own beliefs (we all do it), and they read a criticism of a study, it mentions "small sample sizes", and they can walk away happy in a belief the study was flawed. But all that's really happened is that it's prevented a proper discussion of the study taking place. I'm not sure how much of a problem this is, but I've seen it happen frequently.

roymurdock 11y ago
Got it, thanks for the explanation. I generally trust that the authors of such studies know what they're doing when they design an experiment with a certain n value in mind. Agreed that the insufficient n argument does crop up a lot around HN.
sliverstorm 11y ago
Also important, I recall a note from one of my statistics classes to the effect of, strong sample sizes are often counter-intuitively small. I want to say the example was something like polling in a political race, and the math worked out that you could get reliable results polling only a few tens of people so long as you had random sampling.
- roymurdock 11y ago
  One of my professors had a rule of thumb of discounting anything below n=25, but you absolutely need to ensure random sampling as its one of the fundamental assumptions of regression - making sure the probability of being part of the sample is uniform across the entire population.

by Osmium 11y ago

roymurdock 11y ago

Got it, thanks for the explanation. I generally trust that the authors of such studies know what they're doing when they design an experiment with a certain n value in mind. Agreed that the insufficient n argument does crop up a lot around HN.

sliverstorm 11y ago

Also important, I recall a note from one of my statistics classes to the effect of, strong sample sizes are often counter-intuitively small. I want to say the example was something like polling in a political race, and the math worked out that you could get reliable results polling only a few tens of people so long as you had random sampling.

roymurdock 11y ago
One of my professors had a rule of thumb of discounting anything below n=25, but you absolutely need to ensure random sampling as its one of the fundamental assumptions of regression - making sure the probability of being part of the sample is uniform across the entire population.