3 ms·

This is an important counterargument. Effect size absolutely matters.

• Bartweiss 8y ago
  For that matter, very small studies are safer from certain types of false positive. The file-drawer problem with small lab studies is serious, and can't be solved with effect sizes; it's much easier to run a few of these and publish the hits than to do that with large-population studies. But small lab studies provide a lot of value alongside population studies. They help reveal issues with bad self-report data, which are major concerns for sleep and weight questions. They also offer an efficient way to replicate and extend large studies. If you're spending a decade with 10k+ subjects, gathering a huge range of data is understandable (and I would argue, preferable). But it leaves you with a choice between salami slicing across all your variables, or applying multiple-comparisons corrections that destroy your sensitivity. Using large multi-factor studies as exploratory work, then validating effects with small focused studies is actually a pretty healthy outcome. A final thought: I have a lot of problems with this study, but file-drawer issues aren't one of them. Looking at a bunch of correlated variables (insulin sensitivity, weight gain, outside-meal eating, etc) and reporting them all is a solid way to mitigate that - we know those things should correlate for basically any real effect, which makes it much harder to stumble into a false positive and publish.