4 ms·

You think that sexual dimorphism doesn't apply to .. brain development? Frankly, it's absurd to claim that's an inappropriate subgroup. There are certainly va

You think that sexual dimorphism doesn't apply to .. brain development? Frankly, it's absurd to claim that's an inappropriate subgroup. There are certainly valid criticisms to this particular paper, but this sort of analysis is standard and eminently justified for mammalian development work. I'm also not sure what 'low a priori power' means. Statistical power is well defined and depends only on the sampling involved. You don't need to make any assumptions about it, Bayesian or otherwise. As for the statistics, if anything the low p-values they get are even more impressive given the modest power of these analyses. The effect size must be enormous. Granted the gross PPI response didn't give a dazzling p-value, the followup is quite convincing, showing obvious quantitative changes by a number of metrics, particularly the NAcS Dopamine time-response. I'm curious where you think a multiple-comparison adjustment needs to be be made. The authors are quite focused on a narrow analysis of the systems implicated in observational PCE studies. Where I agree is that I would like the authors to have discussed the sexual dimorphism to some degree. It would be handy to know if there is any pre-existing literature that suggests that PCE affects males more than females; all that I could find was one that found effects in females, not males. Someone in the field likely has some reasonable ideas about what might be going on. However, I'd hesitate to say the issue is avoided; Nature has strict length requirements and such a discussion would necessarily be somewhat meandering.

notafraudster 7y ago
I think there are probably three things I want to follow up on: 1. I do not believe sexual dimorphism is necessarily irrelevant here. I do believe that any subgroup analysis is a researcher degree of freedom and an avenue for p-hacking or fishing. Ideally subgroups would be well specified in advance: particularly the expectation of an effect in male rats and a null in female rats. Given that this did not happen, I would expect in-text justification for the subgroup analysis. I assume there are many brain development processes which are sexually dimorphic, and many which are not. To trust the study I need to know that this is one such case without reference to the study's results. Your end-of-post comment suggests that dimorphism is valid to explore, but the dimorphism should favour effects in the opposite direction, which causes me to further doubt this study's results (in this statement only I am introducing a Bayesian notion of belief) 2. By "low a priori power" I mean that the number of units they chose would typically not be well-powered for modest effects in the subgroup. I am not talking about anything Bayesian. If I knew nothing about this subject, but picked a modest effect size on the quantity of interest and did a power analysis, I would find the study is well under 80% powered to detect the effect. That the effect was detected in the actual study should not lead us to say "wow, if it was underpowered but still detected, the effect must be huge", it should lead us to say "wow, this effect is very sensitive to any kind of confounding". This raises additional study integrity concerns and replicability concerns. 3. There are different points of view about multiple comparison adjustment. In the absence of pre-registration, I am inclined to believe that even closely related hypotheses that are all well justified theoretically should be adjusted for multiple comparisons. The authors report some null results and some significant results, and we are made to infer the null results are true nulls and the significant results are truly significant. The error rate for that joint set of inferences is much higher than the nominal error rate. To reiterate, my concerns here are exacerbated by the fact that this is a topic likely to get public traction, and which speaks directly to political agendas on both sides of the marijuana policy aisle. I have limited interest in following up on the neuroscience/biology stuff, since that is not my field. I want to speak only to responsible scientific practice. If neuroscience routinely publishes these kinds low-power multiple-comparison subgroup-analyses, I suspect there will be a replicability crisis in the near future.

notafraudster 7y ago

I think there are probably three things I want to follow up on: 1. I do not believe sexual dimorphism is necessarily irrelevant here. I do believe that any subgroup analysis is a researcher degree of freedom and an avenue for p-hacking or fishing. Ideally subgroups would be well specified in advance: particularly the expectation of an effect in male rats and a null in female rats. Given that this did not happen, I would expect in-text justification for the subgroup analysis. I assume there are many brain development processes which are sexually dimorphic, and many which are not. To trust the study I need to know that this is one such case without reference to the study's results. Your end-of-post comment suggests that dimorphism is valid to explore, but the dimorphism should favour effects in the opposite direction, which causes me to further doubt this study's results (in this statement only I am introducing a Bayesian notion of belief) 2. By "low a priori power" I mean that the number of units they chose would typically not be well-powered for modest effects in the subgroup. I am not talking about anything Bayesian. If I knew nothing about this subject, but picked a modest effect size on the quantity of interest and did a power analysis, I would find the study is well under 80% powered to detect the effect. That the effect was detected in the actual study should not lead us to say "wow, if it was underpowered but still detected, the effect must be huge", it should lead us to say "wow, this effect is very sensitive to any kind of confounding". This raises additional study integrity concerns and replicability concerns. 3. There are different points of view about multiple comparison adjustment. In the absence of pre-registration, I am inclined to believe that even closely related hypotheses that are all well justified theoretically should be adjusted for multiple comparisons. The authors report some null results and some significant results, and we are made to infer the null results are true nulls and the significant results are truly significant. The error rate for that joint set of inferences is much higher than the nominal error rate. To reiterate, my concerns here are exacerbated by the fact that this is a topic likely to get public traction, and which speaks directly to political agendas on both sides of the marijuana policy aisle. I have limited interest in following up on the neuroscience/biology stuff, since that is not my field. I want to speak only to responsible scientific practice. If neuroscience routinely publishes these kinds low-power multiple-comparison subgroup-analyses, I suspect there will be a replicability crisis in the near future.