4 ms·

If the NIH grant scheme was not a zero sum game then this would be reasonable. A scientist with real results missed out on a grant that they would have got if J

If the NIH grant scheme was not a zero sum game then this would be reasonable. A scientist with real results missed out on a grant that they would have got if Julia had not made this error. There is a fundamental issue here that is not being discussed is that is the science funding system incentivises people rushing results out and not check if they are real. If Julia had been more careful she would have found the results were false, not published, and not received a grant. So much of the problems with a lack of reproducibility in science is the direct result of people rushing to get out “novel” data and not checking if it is real because if they do they don’t get funded.

renewiltord 6y ago
This is just an optimization problem with where you place the incentives. The follow-up study funding is not exclusively for false null hypothesis rejection due to chance. It's also to allow for some amount of error in experimental design or practice. No one wants 0% experimental design / practice error because that will harm total experiments done per unit time which is also a thing we want to go up. No one wants only certain results because we want novel results / unit time to go up too. It's just multivariate non-linear optimization. Yes, someone whose conclusion was not due to error in programming may have gone without a grant. That's perfectly all right. We optimize policy in the aggregate. You can make an argument that says that we're not correctly placed on the manifold but no single situation will be convincing for that and you'll need to make some case that a policy change to move to some other point in this space will yield a positive total improvement.
- danieltillett 6y ago
  True, but I would personally place more emphasis on avoiding false data being published and less on novelty. If the consequence of publishing something wrong were greater, and the costs of taking the time to do things right less, then we might see more science we can trust published.
  - cycomanic 6y ago
    Not when people's livelihoods are on the line. You would simply get people to not report. What you are asking for is the equivalent of saying "firing programmers when a bug is found will result in better software" In science it will result in much less interesting science being done, because this will desincentivize risk. The big problem in science funding is that it already incentivizes low risk projects (you should have preliminary results that proof it works...) so let's not make it worse
    - danieltillett 6y ago
      I am arguing that the current funding structure encourages poor quality science. Right now it is better to pump out papers as quickly as possible rather than taking the time to check the data is not flawed. From a software perspective it is like paying developers by the number of lines of code delivered and telling them to not worry about bugs. Not an approach that is likely to deliver quality software. What we need in science is to find a way that people have the time and incentives to publish results that are as accurate as possible. Somewhat off topic I do wonder if we would get better software if we fired programmers if bugs were found. I suspect we would not get much code written, but the code that was written would be very high quality.

renewiltord 6y ago

This is just an optimization problem with where you place the incentives. The follow-up study funding is not exclusively for false null hypothesis rejection due to chance. It's also to allow for some amount of error in experimental design or practice. No one wants 0% experimental design / practice error because that will harm total experiments done per unit time which is also a thing we want to go up. No one wants only certain results because we want novel results / unit time to go up too. It's just multivariate non-linear optimization. Yes, someone whose conclusion was not due to error in programming may have gone without a grant. That's perfectly all right. We optimize policy in the aggregate. You can make an argument that says that we're not correctly placed on the manifold but no single situation will be convincing for that and you'll need to make some case that a policy change to move to some other point in this space will yield a positive total improvement.

danieltillett 6y ago
True, but I would personally place more emphasis on avoiding false data being published and less on novelty. If the consequence of publishing something wrong were greater, and the costs of taking the time to do things right less, then we might see more science we can trust published.
- cycomanic 6y ago
  Not when people's livelihoods are on the line. You would simply get people to not report. What you are asking for is the equivalent of saying "firing programmers when a bug is found will result in better software" In science it will result in much less interesting science being done, because this will desincentivize risk. The big problem in science funding is that it already incentivizes low risk projects (you should have preliminary results that proof it works...) so let's not make it worse
  - danieltillett 6y ago
    I am arguing that the current funding structure encourages poor quality science. Right now it is better to pump out papers as quickly as possible rather than taking the time to check the data is not flawed. From a software perspective it is like paying developers by the number of lines of code delivered and telling them to not worry about bugs. Not an approach that is likely to deliver quality software. What we need in science is to find a way that people have the time and incentives to publish results that are as accurate as possible. Somewhat off topic I do wonder if we would get better software if we fired programmers if bugs were found. I suspect we would not get much code written, but the code that was written would be very high quality.