Andrew Gelman put it very well here: https://andrewgelman.com/2017/02/06/not-kill-statistical-significance-makes-stronger-fallacy/ https://andrewgelman.com/201

Andrew Gelman put it very well here: https://andrewgelman.com/2017/02/06/not-kill-statistical-significance-makes-stronger-fallacy/ https://andrewgelman.com/2017/02/06/not-kill-statistical-sig... Or, if you want to some it all up in a single graph: https://andrewgelman.com/2014/11/17/power-06-looks-like-get-used/ https://andrewgelman.com/2014/11/17/power-06-looks-like-get-... (although your experiments hopefully have power much greater than 0.06) The gist of the point: If your sample size is small, significant results are likely to (sometimes vastly) overestimate the true effect size, and there is a non-negligible chance that the effect size is actually estimated in the wrong direction.