3 ms·

I appreciate your comments but as the biologist hopefully you can understand that it's not like we are trying to keep our sample sizes small. We understand stat

I appreciate your comments but as the biologist hopefully you can understand that it's not like we are trying to keep our sample sizes small. We understand statistical power. It's simply not feasible logistically and financially to have the ideal numbers a lot of the time. Real life intervenes, so we do our best. For math/computer people which is probably many here in HN, I think they have a hard time with real-world messiness that is biomedical research. It's the same mentality of "oh I'll just apply my algorithms to this..." which works for them but not so much for us.

capnrefsmmat 10y ago
Given how infrequently we see power calculations in published papers, and the results of typical surveys of scientists, I don't think the typical scientist does understand statistical power or other statistical concepts. There are plenty of surveys (e.g. http://www.tandfonline.com/doi/abs/10.1080/17470218.2014.885986 http://www.tandfonline.com/doi/abs/10.1080/17470218.2014.885...) showing that scientists just pick sample sizes based on what's usually done in their field, rather than working out what's necessary. There are plenty of other surveys showing practicing scientists don't understand the concepts of significance and power. Some of the authors of those studies don't understand them either -- I've seen at least one study with questions with no correct answer... I know large sample sizes are often impractical, and you have to make do. But given that, results must be presented with all sorts of disclaimers, since results from underpowered studies are frequently wrong or exaggerated (https://www.ncbi.nlm.nih.gov/pubmed/18633328 https://www.ncbi.nlm.nih.gov/pubmed/18633328). That's not what happens -- scientists who should be aware that their studies have very little value as evidence instead present them as groundbreaking definitive results. I would much rather see slow, difficult, tentative studies instead of the high-speed spray of nearly meaningless results we see every day. But that will take a dramatic change in career and funding incentives.
- toufka 10y ago
  >I would much rather see slow, difficult, tentative studies instead of the high-speed spray of nearly meaningless results we see every day. A major hurdle is the experiments do keep getting slower and slower. As a recently graduated biomedical grad student I spent literally 6.5 years of my life trying to get an experiment to work enough times to glean useful data out of it. I knew exactly how much power my data had (I had literally years to think and rethink about it). The number of stars that had to align to get equipment, protocols, controls, materials, animals, sleep-schedules, etc. all aligned to get data out of an incredibly complicated system meant that there is no fast data. Useful biological data is hard-won. And it's getting harder and harder (read, more expensive and labor-intensive). You could say, well, then just don't expect every scientist to do their own (independent) research (see the author lists on CERN publications...). That indeed would change incentives a lot in the biomedical fields - and I actually do think for the better. The last point is I think there's a big difference between how a practicing scientist sees a 'published paper' in their field, and how everyone else sees it. A published paper really is just a mark of what you did, and what happened, and maybe a basic interpretation. It really does not claim to be 'true' in a strong sense. And there are entirely reasonable situations where different papers come to opposite (justifiable) conclusions. This is itself weighted data to be used in the next motions of the field precisely because sample size and (more often than not) experimental complexity makes clarity hard to come by for a single paper.

capnrefsmmat 10y ago

Given how infrequently we see power calculations in published papers, and the results of typical surveys of scientists, I don't think the typical scientist does understand statistical power or other statistical concepts. There are plenty of surveys (e.g. http://www.tandfonline.com/doi/abs/10.1080/17470218.2014.885986 http://www.tandfonline.com/doi/abs/10.1080/17470218.2014.885...) showing that scientists just pick sample sizes based on what's usually done in their field, rather than working out what's necessary. There are plenty of other surveys showing practicing scientists don't understand the concepts of significance and power. Some of the authors of those studies don't understand them either -- I've seen at least one study with questions with no correct answer... I know large sample sizes are often impractical, and you have to make do. But given that, results must be presented with all sorts of disclaimers, since results from underpowered studies are frequently wrong or exaggerated (https://www.ncbi.nlm.nih.gov/pubmed/18633328 https://www.ncbi.nlm.nih.gov/pubmed/18633328). That's not what happens -- scientists who should be aware that their studies have very little value as evidence instead present them as groundbreaking definitive results. I would much rather see slow, difficult, tentative studies instead of the high-speed spray of nearly meaningless results we see every day. But that will take a dramatic change in career and funding incentives.

toufka 10y ago
>I would much rather see slow, difficult, tentative studies instead of the high-speed spray of nearly meaningless results we see every day. A major hurdle is the experiments do keep getting slower and slower. As a recently graduated biomedical grad student I spent literally 6.5 years of my life trying to get an experiment to work enough times to glean useful data out of it. I knew exactly how much power my data had (I had literally years to think and rethink about it). The number of stars that had to align to get equipment, protocols, controls, materials, animals, sleep-schedules, etc. all aligned to get data out of an incredibly complicated system meant that there is no fast data. Useful biological data is hard-won. And it's getting harder and harder (read, more expensive and labor-intensive). You could say, well, then just don't expect every scientist to do their own (independent) research (see the author lists on CERN publications...). That indeed would change incentives a lot in the biomedical fields - and I actually do think for the better. The last point is I think there's a big difference between how a practicing scientist sees a 'published paper' in their field, and how everyone else sees it. A published paper really is just a mark of what you did, and what happened, and maybe a basic interpretation. It really does not claim to be 'true' in a strong sense. And there are entirely reasonable situations where different papers come to opposite (justifiable) conclusions. This is itself weighted data to be used in the next motions of the field precisely because sample size and (more often than not) experimental complexity makes clarity hard to come by for a single paper.