2 ms·

So there is a clear result found and then post hoc there is an explanation made for why this makes sense. That's a bit of a red flag since you preferably hypoth

So there is a clear result found and then post hoc there is an explanation made for why this makes sense. That's a bit of a red flag since you preferably hypothesize first, but let's look at the analysis. If I'm not mistaken, then these are the most important parts: - 467 measured metabolites (features) - 136 males and 69 females = 205 (data points) - "To minimize overfitting, only correlations with q values < 0.05 were tallied." Then the predictive performance in Fig. 3 looks very promising with a 0.88 AUC score. However, according to the methods: "Classifiers of 4–7 metabolites were selected and tested for diagnostic accuracy using area under the receiver operator characteristic curve and random forest analysis." Then, those AUC scores do not provide so much value since they first used the data to select the best features and then used the same data to fit the model on the best features. It also looks a bit like the authors have thrown the kitchen sink at these 200 samples since I see R, GraphPad Prism, Python, CIRCOS, Cytoscape, and MetaboAnalyst being used. And the statement "Fifty (50) random samples at each subsample size were taken to estimate the population statistics based on the central limit theorem." is a bit weird because the central limit theorem shows up throughout many analyses including confidence intervals. So all in all I'm not 100% sure, but I am skeptical. Anyone here who knows more about "Metabolic network and hub-and-spoke analysis" or "network growth" analysis? Or why it is reasonable to use so many metrics instead of being more selective?

elcritch 2y ago
> So there is a clear result found and then post hoc there is an explanation made for why this makes sense. That's a bit of a red flag since you preferably hypothesize first, but let's look at the analysis. Eh, this doesn’t look to be a P-hacking attempt. Scientific exploration doesn’t always hit the hypothesis first model. Ideally this paper should be a source of future specific hypothesis and experiments. Though, IMHO, their implicit hypothesis is that metabolism is a core feature of ASD. The points about the predictive power in fig 3 are interesting.
- immibis 2y ago
  Getting 80000 measurements and then accepting a 1-in-20 false positive rate, however, sounds like P-hacking.

elcritch 2y ago

> So there is a clear result found and then post hoc there is an explanation made for why this makes sense. That's a bit of a red flag since you preferably hypothesize first, but let's look at the analysis. Eh, this doesn’t look to be a P-hacking attempt. Scientific exploration doesn’t always hit the hypothesis first model. Ideally this paper should be a source of future specific hypothesis and experiments. Though, IMHO, their implicit hypothesis is that metabolism is a core feature of ASD. The points about the predictive power in fig 3 are interesting.

immibis 2y ago
Getting 80000 measurements and then accepting a 1-in-20 false positive rate, however, sounds like P-hacking.