4 ms·

The problem with upping the standard is, as the PNAS article acknowledges, that you need a larger sample size to produce any given result. Unfortunately, many

The problem with upping the standard is, as the PNAS article acknowledges, that you need a larger sample size to produce any given result. Unfortunately, many studies -- particularly those in medicine -- are already conducted with samples that are too small to detect any effect you'd reasonably expect to see, because many researchers do not calculate in advance what sample size would be required. This has interesting paradoxical effects: the only published studies are those that overestimate the size of the true effect. http://www.refsmmat.com/statistics/power.html http://www.refsmmat.com/statistics/power.html http://www.refsmmat.com/statistics/regression.html#truth-inflation http://www.refsmmat.com/statistics/regression.html#truth-inf... So there's a tradeoff. Do you want to eliminate false positives at the cost of more false negatives? It's a difficult balance. I suspect there are many areas where poor statistical practice can be remedied to produce better results without greater expense.

toufka 13y ago
I still think the answer is not in any particular standard in any particular field, rather it's outputting the final, worked datasets. Making the numbers themselves available and usable to the 'reader' or scientist following up will immediately clear up issues of reproducibility or not meeting statistical thresholds. If I see the same data, run it through my own processes and it looks like noise I'm likely to discount the results moving forward. There's nothing inherently wrong with saying 'look, I did this once, and this is what I saw'. And many times it's useful. It's less useful, but still reasonable in certain circumstances to say, 'look, I did this a hundred times and I saw this once'. There are occasions when the experiment cannot reasonably have more than a statistically insignificant 'n' - but it still might be useful to see what happens. What is not reasonable is to say, 'this is what I (sometimes) see always (trust me)' and hide your data behind a 'representative' jpg and a 'statistically significant' p-value. As I scientist who has worked with both rich, and poor datasets from standard and invented datatypes, I'd just say, "let me see the data".

toufka 13y ago

I still think the answer is not in any particular standard in any particular field, rather it's outputting the final, worked datasets. Making the numbers themselves available and usable to the 'reader' or scientist following up will immediately clear up issues of reproducibility or not meeting statistical thresholds. If I see the same data, run it through my own processes and it looks like noise I'm likely to discount the results moving forward. There's nothing inherently wrong with saying 'look, I did this once, and this is what I saw'. And many times it's useful. It's less useful, but still reasonable in certain circumstances to say, 'look, I did this a hundred times and I saw this once'. There are occasions when the experiment cannot reasonably have more than a statistically insignificant 'n' - but it still might be useful to see what happens. What is not reasonable is to say, 'this is what I (sometimes) see always (trust me)' and hide your data behind a 'representative' jpg and a 'statistically significant' p-value. As I scientist who has worked with both rich, and poor datasets from standard and invented datatypes, I'd just say, "let me see the data".