3 ms·

While there are problems with the way statistics are used by scientists and others, the elephant in room is the incentive system. p-hacking and the like result

While there are problems with the way statistics are used by scientists and others, the elephant in room is the incentive system. p-hacking and the like result from the need to get “exciting and publishable” results on a consistent basis or effectively be fired (the simple path is no “sexy” results -> no “high quality” papers -> no grant funding -> no job). A possible solution that I have not seen proposed (this is mostly likely my ignorance) is for journals to only accept/reject research proposals, not finished papers. The journal would publish the resulting paper no matter the outcome of the research.

Vinnl 9y ago
The problem is that journals have incentives too. And these unfortunately often align: sexy results -> more income.
- danieltillett 9y ago
  This is true, but it is surprisingly weak as the income of a journal is not too closely related to its impact factor. My proposal would still mean that Nature would get the most sexy papers because it would get the most sexy proposals. A bigger problem would be journals choosing proposals from authors with a history of getting sexy results rather than on the quality of the proposal. This reliance on "track record" is why the grant system is so broken.
  - Vinnl 9y ago
    Isn't it? I mean, I know there's no hard data on it since most journal subscriptions are bought in bulk, but the journals are using their impact factor to sell those subscriptions (case in point: [1]). Likewise, if they charge publication fees, authors will be most willing to pay large sums of (their funders') money for journals with high impact factors. And do you really think Nature would still get sexy results if there are sexy proposals? Or would your proposal (which I do agree with, for the same reason the traditional publishers won't do it:) lead to better research but fewer "sexy" results? After all, journals with a high impact factor are also known to have more retractions. (The traditional) publishers are going to keep doing what is bringing in the most money - or at least, what they believe what will. At this point, they don't believe that your system would lead to higher income. What could change that if authors are no longer incentivised to chase high impact factor journals. But that means there will have to be better ways to evaluate them that will actually get adopted (i.e. not require evaluators to comb through every applicant's research [2]). This is a difficult problem. And yes, I realise I'm not coming up with a solution. The only ideas I have on that front have a really small chance of success, but it's a bit too much to elaborate on that here. (Although in time I will write about them at [3].) [1] https://mobile.twitter.com/simoxenham/status/235698457124425728/photo/1 https://mobile.twitter.com/simoxenham/status/235698457124425... [2] https://theconversation.com/why-i-disagree-with-nobel-laureates-when-it-comes-to-career-advice-for-scientists-80079 https://theconversation.com/why-i-disagree-with-nobel-laurea... [3] https://medium.com/flockademic https://medium.com/flockademic

Vinnl 9y ago

The problem is that journals have incentives too. And these unfortunately often align: sexy results -> more income.

danieltillett 9y ago
This is true, but it is surprisingly weak as the income of a journal is not too closely related to its impact factor. My proposal would still mean that Nature would get the most sexy papers because it would get the most sexy proposals. A bigger problem would be journals choosing proposals from authors with a history of getting sexy results rather than on the quality of the proposal. This reliance on "track record" is why the grant system is so broken.
- Vinnl 9y ago
  Isn't it? I mean, I know there's no hard data on it since most journal subscriptions are bought in bulk, but the journals are using their impact factor to sell those subscriptions (case in point: [1]). Likewise, if they charge publication fees, authors will be most willing to pay large sums of (their funders') money for journals with high impact factors. And do you really think Nature would still get sexy results if there are sexy proposals? Or would your proposal (which I do agree with, for the same reason the traditional publishers won't do it:) lead to better research but fewer "sexy" results? After all, journals with a high impact factor are also known to have more retractions. (The traditional) publishers are going to keep doing what is bringing in the most money - or at least, what they believe what will. At this point, they don't believe that your system would lead to higher income. What could change that if authors are no longer incentivised to chase high impact factor journals. But that means there will have to be better ways to evaluate them that will actually get adopted (i.e. not require evaluators to comb through every applicant's research [2]). This is a difficult problem. And yes, I realise I'm not coming up with a solution. The only ideas I have on that front have a really small chance of success, but it's a bit too much to elaborate on that here. (Although in time I will write about them at [3].) [1] https://mobile.twitter.com/simoxenham/status/235698457124425728/photo/1 https://mobile.twitter.com/simoxenham/status/235698457124425... [2] https://theconversation.com/why-i-disagree-with-nobel-laureates-when-it-comes-to-career-advice-for-scientists-80079 https://theconversation.com/why-i-disagree-with-nobel-laurea... [3] https://medium.com/flockademic https://medium.com/flockademic