3 ms·

> That [the method of statistical analysis] could be a problem in our particular context didn’t dawn on me and my colleagues — nor on anyone else in the field

> That [the method of statistical analysis] could be a problem in our particular context didn’t dawn on me and my colleagues — nor on anyone else in the field — before [whoever]'s discovery. This is troubling to me. It does not seem like the bias was due to something arcane, some finer point in advanced statistics, something hidden from view etc. The poster says: > It involved regression towards the mean — when noisy data are measured repeatedly, values that at first look extreme become less so. This may not be 100% straightforward when it's buried in the middle of a paper, but if you actually consider the methodolgy you are likely to notice this happening. So here's what _I_ learn from this case: * It is possible that the reviewers at Nature don't properly scrutinize the methodological soundness of some submissions (I say "possible" since this is a single example, not a pattern) * PhD avisors, like the author's, may not be exercising due dilligence on statistical research done with their PhD candidates. The author's advisor had this to say: > "It's great that we’ve persisted in attempting to understand our methodology and findings!” so he says it's "great" that they did not fully understand their methodology before submitting a paper using it. Maybe that's not exactly what he meant, but still, pretty worrying.

bachmeier 6y ago
You present a view of journals as an outlet for correct results. Realistically, that's not possible, and I wish more people accepted that it's not possible. If a result is confirmed in many studies by many authors using many datasets and many methodologies, that's how we know we can trust a result. I personally do not put much weight on a single paper's results unless there's something special about it. The most important thing is that everything about the investigation is open: the methodology (including everything not found in the paper), the programs, the data. It's more like posting code for a large project on Github and then new researchers make PRs to correct bugs or extend the program in useful ways.
- einpoklum 6y ago
  I think you're conflating correctness and degrees of validity. Yes, I expect journals to have to have correct results - not in the sense that the generalizations from their findings are universally valid, but in that the statements of fact and of logical implications are valid. To be more explicit: * "Events X, Y, Z occurred" <- expect this kind of sentences to be correct * "We did X, Y, Z" <- expect this kind of sentences to be correct * "A and A->B, so B" <- expect this kind of sentences to be correct * "We therefore conclude that X" <- Don't expect X to necessarily be correct.
  - kansface 6y ago
    > in that the statements of fact and of logical implications are valid. See for example https://slatestarcodex.com/2019/05/07/5-httlpr-a-pointed-review/ https://slatestarcodex.com/2019/05/07/5-httlpr-a-pointed-rev... for a slightly different take.
    - einpoklum 6y ago
      I don't see how that disagrees with what I wrote.
foobeloo 6y ago
Pay attention to the context of that quote about regression to the mean. The problem in this case manifested as the opposite: 'egression from the mean', which can happen due to the combined effects of binning, cross-thresholding and heteroscedasticity (see https://www.biorxiv.org/content/10.1101/2020.12.15.422942v1 https://www.biorxiv.org/content/10.1101/2020.12.15.422942v1). I think that _is_ a fairly arcane point, at least to most life scientists.

bachmeier 6y ago

You present a view of journals as an outlet for correct results. Realistically, that's not possible, and I wish more people accepted that it's not possible. If a result is confirmed in many studies by many authors using many datasets and many methodologies, that's how we know we can trust a result. I personally do not put much weight on a single paper's results unless there's something special about it. The most important thing is that everything about the investigation is open: the methodology (including everything not found in the paper), the programs, the data. It's more like posting code for a large project on Github and then new researchers make PRs to correct bugs or extend the program in useful ways.

einpoklum 6y ago
I think you're conflating correctness and degrees of validity. Yes, I expect journals to have to have correct results - not in the sense that the generalizations from their findings are universally valid, but in that the statements of fact and of logical implications are valid. To be more explicit: * "Events X, Y, Z occurred" <- expect this kind of sentences to be correct * "We did X, Y, Z" <- expect this kind of sentences to be correct * "A and A->B, so B" <- expect this kind of sentences to be correct * "We therefore conclude that X" <- Don't expect X to necessarily be correct.
- kansface 6y ago
  > in that the statements of fact and of logical implications are valid. See for example https://slatestarcodex.com/2019/05/07/5-httlpr-a-pointed-review/ https://slatestarcodex.com/2019/05/07/5-httlpr-a-pointed-rev... for a slightly different take.
  - einpoklum 6y ago
    I don't see how that disagrees with what I wrote.

foobeloo 6y ago

Pay attention to the context of that quote about regression to the mean. The problem in this case manifested as the opposite: 'egression from the mean', which can happen due to the combined effects of binning, cross-thresholding and heteroscedasticity (see https://www.biorxiv.org/content/10.1101/2020.12.15.422942v1 https://www.biorxiv.org/content/10.1101/2020.12.15.422942v1). I think that _is_ a fairly arcane point, at least to most life scientists.