I recall having read this piece before, and it is well written; but for a non-statistician, there's really just one thing you need to remember: Extraordinary c

I recall having read this piece before, and it is well written; but for a non-statistician, there's really just one thing you need to remember: Extraordinary claims require extraordinary evidence. If you're testing 500,000 hypotheses and you find that one of them is significant, it better be extremely significant, and it had better survive independent replication. Otherwise... well, let's not belabor the point. Just keep in mind that a wild and wacky theory needs some heavy-duty experimental evidence (i.e. replication) before you, or the PI, or the referees for Ye Olde Journal, have any reason to believe it. If everyone involved would keep this in mind, it would cut down hugely on publication bias and popular confirmation bias as well.