6 ms·

Accept your data. According to one PI, success depends more on the people doing a project than on the topic of study; some people make their project work no mat

by gone35 8y ago

Accept your data. According to one PI, success depends more on the people doing a project than on the topic of study; some people make their project work no matter what the topic is. Others block their own way to success because they require too much evidence. I, for one, have always been excessively critical of my own data. Although I am a meticulous experimenter, whenever my results seemed to confirm my hypothesis, I was afraid to accept them because I knew I was biased (it’s funny, I know). I was spending too much time looking for alternative explanations instead of accepting that I was on the right track and moving on to the next step. We have all heard about the reproducibility crisis — and I am not advocating the reckless publication of unverified data, or recommending being so careful that you end up publishing nothing — but there is a difference between being reckless and shooting yourself in the foot. And this, fellows, is how you end up with the reproducibility crisis. Incentives are just all messed up.

prestonh 8y ago
What's good for the field (we've decided) is when we only allow a 5% Type I error rate (95% of published positive results are correct). What's good for the individual is to drive their Type I error rate up just enough so that they can publish as much as possible without getting caught.
- radford-neal 8y ago
  Umm... That's not actually what only publishing a result when the p-value is less than 0.05 accomplishes. Such a publishing policy instead guarantees (or would if there were no other problems) that __only 5% of the FALSE ideas that researchers test get published as being "confirmed"__ (i.e., when the null hypothesis is true, only 5% of the time does it get "rejected" at the 0.05 level). It guarantees nothing about what fraction of these supposedly-confirmed ideas are actually true. That fraction could vary from zero (if researchers never think of true ideas) to 100% (if researchers never think of false ideas).
  - prestonh 8y ago
    You're right, I was trying to dumb down p-values to make a point about perverse incentives in academic publishing. What's worse is that other factors that increase the Type I error rate (only positive results are published) only compound the effect.
- Will_Parker 8y ago
  A 0.05 p value very much does not mean there's a 95% likelihood of the result being true. It's a fallacy that is far too common.
gshulegaard 8y ago
You are reading that completely differently than how I read it. I personally think that the author is reflecting on the fact that he may have been not focused enough on completing a single task before starting another. It sounds (and this is speculative, I don't want to put words in the author's mouth) like the author would start with a hypothesis, run an experiment that generates data that appears to support that hypothesis, but then start running tangential experiments disproving alternative hypotheses to further strengthening the original hypothesis. Hence his refelection: > I am not advocating the reckless publication of unverified data, or recommending being so careful that you end up publishing nothing — but there is a difference between being reckless and shooting yourself in the foot. To me, the author is looking back and now realizes that he may have been too cautious in that he was running too many experiments to publish one result and that there is a spectrum of caution and that he may have missed the sweet spot. As others have pointed out ~19 papers (and only 5 first author) over 23 years seems to be a low publication rate, which I think the author is reflecting on. I agree that the incentives are messed up and these upside down incentives are a driving force behind the reproducibility crisis, but I wouldn't take this quote from the author and say it is an illustration of the reproducibility crisis.
- retsibsi 8y ago
  > As others have pointed out ~19 papers (and only 5 first author) over 23 years seems to be a low publication rate, which I think the author is reflecting on. Low compared to what? Compared to the expectations set by his fellow scientists in a field shaped by the norms and incentives in question. There's a circularity here (and in the earlier comment by Obi_Juan_Kenobi): apparently this guy's real problem was his own low productivity, which we know because he published less than his colleagues and competitors -- therefore his case can't tell us anything about the warped incentive structure that favours publication maximization over intellectual integrity.
  - gshulegaard 8y ago
    By the author's own admission, he thinks he could have focused himself a little better. He isn't saying he would have had to cut corners and do poor science, just not "shoot himself in the foot" to use his own phrasing. In reflection it sounds like he thinks that there was an opportunity for him to publish more frequently without doing bad science. This is why I don't read his self-reflection as necessarily being an illustration of the type of mindset that leads to the reproducibility crisis. It sounds like a reasonable scientist reflecting on things he might have reasonably done better with the benefit of hindsight. No where do I read the implication that he is lamenting his choice to not do shoddy/quick science for the sake of securing a tenure track position. Moreover this is self reflection so a lot the things he is reflecting on are speculative in that he thinks they might have helped him find a tenured position, but he can't be sure. He even has this disclaimer at the beginning of his piece: > I’m not claiming to know the formula for how to get tenured (if I had that, I would have used it for myself). Instead, I would like to offer the advice I’ve gathered.
refurb 8y ago
Like it or not, being a good scientist (read: getting tenure track and funding) is not just about good science, but also exciting, cutting-edge science. If a scientist runs a couple experiments and find something truly remarkable, the guy who rushes to publish is going to win out over the guy that says “this needs another year of work before it’s perfect”.

by gone35 8y ago

prestonh 8y ago

What's good for the field (we've decided) is when we only allow a 5% Type I error rate (95% of published positive results are correct). What's good for the individual is to drive their Type I error rate up just enough so that they can publish as much as possible without getting caught.

radford-neal 8y ago
Umm... That's not actually what only publishing a result when the p-value is less than 0.05 accomplishes. Such a publishing policy instead guarantees (or would if there were no other problems) that __only 5% of the FALSE ideas that researchers test get published as being "confirmed"__ (i.e., when the null hypothesis is true, only 5% of the time does it get "rejected" at the 0.05 level). It guarantees nothing about what fraction of these supposedly-confirmed ideas are actually true. That fraction could vary from zero (if researchers never think of true ideas) to 100% (if researchers never think of false ideas).
- prestonh 8y ago
  You're right, I was trying to dumb down p-values to make a point about perverse incentives in academic publishing. What's worse is that other factors that increase the Type I error rate (only positive results are published) only compound the effect.
Will_Parker 8y ago
A 0.05 p value very much does not mean there's a 95% likelihood of the result being true. It's a fallacy that is far too common.

gshulegaard 8y ago

You are reading that completely differently than how I read it. I personally think that the author is reflecting on the fact that he may have been not focused enough on completing a single task before starting another. It sounds (and this is speculative, I don't want to put words in the author's mouth) like the author would start with a hypothesis, run an experiment that generates data that appears to support that hypothesis, but then start running tangential experiments disproving alternative hypotheses to further strengthening the original hypothesis. Hence his refelection: > I am not advocating the reckless publication of unverified data, or recommending being so careful that you end up publishing nothing — but there is a difference between being reckless and shooting yourself in the foot. To me, the author is looking back and now realizes that he may have been too cautious in that he was running too many experiments to publish one result and that there is a spectrum of caution and that he may have missed the sweet spot. As others have pointed out ~19 papers (and only 5 first author) over 23 years seems to be a low publication rate, which I think the author is reflecting on. I agree that the incentives are messed up and these upside down incentives are a driving force behind the reproducibility crisis, but I wouldn't take this quote from the author and say it is an illustration of the reproducibility crisis.

retsibsi 8y ago
> As others have pointed out ~19 papers (and only 5 first author) over 23 years seems to be a low publication rate, which I think the author is reflecting on. Low compared to what? Compared to the expectations set by his fellow scientists in a field shaped by the norms and incentives in question. There's a circularity here (and in the earlier comment by Obi_Juan_Kenobi): apparently this guy's real problem was his own low productivity, which we know because he published less than his colleagues and competitors -- therefore his case can't tell us anything about the warped incentive structure that favours publication maximization over intellectual integrity.
- gshulegaard 8y ago
  By the author's own admission, he thinks he could have focused himself a little better. He isn't saying he would have had to cut corners and do poor science, just not "shoot himself in the foot" to use his own phrasing. In reflection it sounds like he thinks that there was an opportunity for him to publish more frequently without doing bad science. This is why I don't read his self-reflection as necessarily being an illustration of the type of mindset that leads to the reproducibility crisis. It sounds like a reasonable scientist reflecting on things he might have reasonably done better with the benefit of hindsight. No where do I read the implication that he is lamenting his choice to not do shoddy/quick science for the sake of securing a tenure track position. Moreover this is self reflection so a lot the things he is reflecting on are speculative in that he thinks they might have helped him find a tenured position, but he can't be sure. He even has this disclaimer at the beginning of his piece: > I’m not claiming to know the formula for how to get tenured (if I had that, I would have used it for myself). Instead, I would like to offer the advice I’ve gathered.

refurb 8y ago

Like it or not, being a good scientist (read: getting tenure track and funding) is not just about good science, but also exciting, cutting-edge science. If a scientist runs a couple experiments and find something truly remarkable, the guy who rushes to publish is going to win out over the guy that says “this needs another year of work before it’s perfect”.