> What does that mean in practice? Imagine a hypothetical scientist that is fundamentally confused about something important, so all hypotheses they generate ar

> What does that mean in practice? Imagine a hypothetical scientist that is fundamentally confused about something important, so all hypotheses they generate are false. Yet, using p=0.05, 5% of those hypotheses will be "confirmed experimentally". In that case, it is not 5% of the "experimentally confirmed" hypotheses that are wrong -- it is full 100%. Even without any cherry-picking. Well, that's example is also introducing dependence, which is a tricky thing of course whenever we talk about chance and stats. But there's also another issue - a statement like "5% of positive published results are by chance since we have a p<=0.05 standard" treats every set of results as if p=0.05, whereas some of them are considerably lower anyway. Though the point of bad actors cherry-picking to screw up the data also comes into play here. (And of course, fully independent things in life are much harder to find than one might think at first.)