3 ms·
I think there are probably three things I want to follow up on: 1. I do not believe sexual dimorphism is necessarily irrelevant here. I do believe that any sub
by notafraudster 7y ago
I think there are probably three things I want to follow up on:
1. I do not believe sexual dimorphism is necessarily irrelevant here. I do believe that any subgroup analysis is a researcher degree of freedom and an avenue for p-hacking or fishing. Ideally subgroups would be well specified in advance: particularly the expectation of an effect in male rats and a null in female rats. Given that this did not happen, I would expect in-text justification for the subgroup analysis. I assume there are many brain development processes which are sexually dimorphic, and many which are not. To trust the study I need to know that this is one such case without reference to the study's results. Your end-of-post comment suggests that dimorphism is valid to explore, but the dimorphism should favour effects in the opposite direction, which causes me to further doubt this study's results (in this statement only I am introducing a Bayesian notion of belief)
2. By "low a priori power" I mean that the number of units they chose would typically not be well-powered for modest effects in the subgroup. I am not talking about anything Bayesian. If I knew nothing about this subject, but picked a modest effect size on the quantity of interest and did a power analysis, I would find the study is well under 80% powered to detect the effect. That the effect was detected in the actual study should not lead us to say "wow, if it was underpowered but still detected, the effect must be huge", it should lead us to say "wow, this effect is very sensitive to any kind of confounding". This raises additional study integrity concerns and replicability concerns.
3. There are different points of view about multiple comparison adjustment. In the absence of pre-registration, I am inclined to believe that even closely related hypotheses that are all well justified theoretically should be adjusted for multiple comparisons. The authors report some null results and some significant results, and we are made to infer the null results are true nulls and the significant results are truly significant. The error rate for that joint set of inferences is much higher than the nominal error rate.
To reiterate, my concerns here are exacerbated by the fact that this is a topic likely to get public traction, and which speaks directly to political agendas on both sides of the marijuana policy aisle.
I have limited interest in following up on the neuroscience/biology stuff, since that is not my field. I want to speak only to responsible scientific practice. If neuroscience routinely publishes these kinds low-power multiple-comparison subgroup-analyses, I suspect there will be a replicability crisis in the near future.