24 ms·

While I applaud Julia for retracting the paper, I do wonder if she handed back the NIH grant she was given off the back of this false result. Some researcher ju

While I applaud Julia for retracting the paper, I do wonder if she handed back the NIH grant she was given off the back of this false result. Some researcher just missed out on a NIH grant that would have been funded if not for this error.

jhauris 6y ago
Every indication is this was an honest mistake that was discovered due to the grant. The costs associated with the experiment where still there. Taking back grants in this situation would disincentivize scientific honesty.
- danieltillett 6y ago
  I wasn’t suggesting the grant be taken back, but that she voluntarily hand it back.
  - Ididntdothis 6y ago
    Unless she is independently wealthy there would be a strong incentive for her to not admit the error if there was an expectation to hand the money back. Where is it supposed to come from?
    - danieltillett 6y ago
      I am not suggesting she hand back the money already spent, just that she might think about handing what is left. I would hand back a grant that was given due to a false result that occurred because of my error. Julia should be praised for doing the right thing with the paper, I just think she should think about going further.
      - Ididntdothis 6y ago
        I don't think so. I am not returning my salary when I screw up at work and I have made bad decisions that cost 100s of thousands. As long it comes from an honest effort mistakes are part of the game.
        unishark 6y ago
        I think you missed the above poster's point. Big grants provide funding for multiple years. It can get cancelled rather than continued. I'm rather surprised it wasn't. They might also cease making payments that haven't been made yet in the current year. I know people this has happened to, purely due to govt shutdowns and such unrelated to anyone screwing up. It would be more like losing the remaining payments of a multi-year contract because it gets broken. Or in your analogy, like losing your job.
    - unishark 6y ago
      There's no way to avoid such incentives here. Being allowed to keep the grant creates an incentive to publish incorrect results in the first place. "Gosh this seems too good to be true, but let's wait till the system has invested another million dollars in my career before investigating the process too closely".
      - danieltillett 6y ago
        This is exactly what the current funding system encourages. If you want to be funded you are almost forced to pump out the papers as fast as you can with the bare minimum of checking.
  - nuclearnice1 6y ago
    Some days at work, if I wrote buggy code, I go to my boss and return my paycheck. No! Wait, that never happens. Sometimes there are going to be mistakes.
    - danieltillett 6y ago
      It is not quite the same thing. If your boss had sold your code to a customer who paid based on the buggy result then would the customer be unreasonable seeking a refund? Would your boss be ethical offering the customer a refund?
      - nuclearnice1 6y ago
        Most software has some bugs. No one asks for a refund. They apply the same reasonable expectation that the people who funded the study did. Whoops mistakes happen.
        searchableguy 6y ago
        No way in related to the mistake in the article. Mistakes can and do happen. But if you sold me software to solve a problem and I paid you for it. You wouldn't refund me if the software didn't solve my original problem? I feel like that's scam but I could be wrong. I didn't get what I was promised and hence I am entitled to a refund. If you got a laptop with broken keyboard or the privacy friendly app that you paid extra for turned out to be facebook, would you not want your money back? You can't just use mistake as an excuse if you promise "specific" things like written in your implicit/explicit contract.
        not2b 6y ago
        You sound like someone who never bought software before. You don't get your money back. You might get a bug fix, but the contract probably says it is sold as is.
        danieltillett 6y ago
renewiltord 6y ago
The purpose of this grant was to run follow-up studies and to work on productionizing the intervention. The purpose of follow-up studies is to detect if result extends. The follow-up work detected result was due to experiment setup error. Grant worked. Actually, encountering this I think I suddenly understand why people get upset about Kickstarter projects not delivering. It is a misunderstanding of a probabilistic situation.
- danieltillett 6y ago
  If the NIH grant scheme was not a zero sum game then this would be reasonable. A scientist with real results missed out on a grant that they would have got if Julia had not made this error. There is a fundamental issue here that is not being discussed is that is the science funding system incentivises people rushing results out and not check if they are real. If Julia had been more careful she would have found the results were false, not published, and not received a grant. So much of the problems with a lack of reproducibility in science is the direct result of people rushing to get out “novel” data and not checking if it is real because if they do they don’t get funded.
  - renewiltord 6y ago
    This is just an optimization problem with where you place the incentives. The follow-up study funding is not exclusively for false null hypothesis rejection due to chance. It's also to allow for some amount of error in experimental design or practice. No one wants 0% experimental design / practice error because that will harm total experiments done per unit time which is also a thing we want to go up. No one wants only certain results because we want novel results / unit time to go up too. It's just multivariate non-linear optimization. Yes, someone whose conclusion was not due to error in programming may have gone without a grant. That's perfectly all right. We optimize policy in the aggregate. You can make an argument that says that we're not correctly placed on the manifold but no single situation will be convincing for that and you'll need to make some case that a policy change to move to some other point in this space will yield a positive total improvement.
    - 6y ago

jhauris 6y ago

Every indication is this was an honest mistake that was discovered due to the grant. The costs associated with the experiment where still there. Taking back grants in this situation would disincentivize scientific honesty.

danieltillett 6y ago
I wasn’t suggesting the grant be taken back, but that she voluntarily hand it back.
- Ididntdothis 6y ago
  Unless she is independently wealthy there would be a strong incentive for her to not admit the error if there was an expectation to hand the money back. Where is it supposed to come from?
  - danieltillett 6y ago
    I am not suggesting she hand back the money already spent, just that she might think about handing what is left. I would hand back a grant that was given due to a false result that occurred because of my error. Julia should be praised for doing the right thing with the paper, I just think she should think about going further.
    - Ididntdothis 6y ago
      I don't think so. I am not returning my salary when I screw up at work and I have made bad decisions that cost 100s of thousands. As long it comes from an honest effort mistakes are part of the game.
      - unishark 6y ago
        I think you missed the above poster's point. Big grants provide funding for multiple years. It can get cancelled rather than continued. I'm rather surprised it wasn't. They might also cease making payments that haven't been made yet in the current year. I know people this has happened to, purely due to govt shutdowns and such unrelated to anyone screwing up. It would be more like losing the remaining payments of a multi-year contract because it gets broken. Or in your analogy, like losing your job.
  - unishark 6y ago
    There's no way to avoid such incentives here. Being allowed to keep the grant creates an incentive to publish incorrect results in the first place. "Gosh this seems too good to be true, but let's wait till the system has invested another million dollars in my career before investigating the process too closely".
    - danieltillett 6y ago
      This is exactly what the current funding system encourages. If you want to be funded you are almost forced to pump out the papers as fast as you can with the bare minimum of checking.
- nuclearnice1 6y ago
  Some days at work, if I wrote buggy code, I go to my boss and return my paycheck. No! Wait, that never happens. Sometimes there are going to be mistakes.
  - danieltillett 6y ago
    It is not quite the same thing. If your boss had sold your code to a customer who paid based on the buggy result then would the customer be unreasonable seeking a refund? Would your boss be ethical offering the customer a refund?
    - nuclearnice1 6y ago
      Most software has some bugs. No one asks for a refund. They apply the same reasonable expectation that the people who funded the study did. Whoops mistakes happen.
      - searchableguy 6y ago
        No way in related to the mistake in the article. Mistakes can and do happen. But if you sold me software to solve a problem and I paid you for it. You wouldn't refund me if the software didn't solve my original problem? I feel like that's scam but I could be wrong. I didn't get what I was promised and hence I am entitled to a refund. If you got a laptop with broken keyboard or the privacy friendly app that you paid extra for turned out to be facebook, would you not want your money back? You can't just use mistake as an excuse if you promise "specific" things like written in your implicit/explicit contract.
        not2b 6y ago
        You sound like someone who never bought software before. You don't get your money back. You might get a bug fix, but the contract probably says it is sold as is.
        danieltillett 6y ago

renewiltord 6y ago

The purpose of this grant was to run follow-up studies and to work on productionizing the intervention. The purpose of follow-up studies is to detect if result extends. The follow-up work detected result was due to experiment setup error. Grant worked. Actually, encountering this I think I suddenly understand why people get upset about Kickstarter projects not delivering. It is a misunderstanding of a probabilistic situation.

danieltillett 6y ago
If the NIH grant scheme was not a zero sum game then this would be reasonable. A scientist with real results missed out on a grant that they would have got if Julia had not made this error. There is a fundamental issue here that is not being discussed is that is the science funding system incentivises people rushing results out and not check if they are real. If Julia had been more careful she would have found the results were false, not published, and not received a grant. So much of the problems with a lack of reproducibility in science is the direct result of people rushing to get out “novel” data and not checking if it is real because if they do they don’t get funded.
- renewiltord 6y ago
  This is just an optimization problem with where you place the incentives. The follow-up study funding is not exclusively for false null hypothesis rejection due to chance. It's also to allow for some amount of error in experimental design or practice. No one wants 0% experimental design / practice error because that will harm total experiments done per unit time which is also a thing we want to go up. No one wants only certain results because we want novel results / unit time to go up too. It's just multivariate non-linear optimization. Yes, someone whose conclusion was not due to error in programming may have gone without a grant. That's perfectly all right. We optimize policy in the aggregate. You can make an argument that says that we're not correctly placed on the manifold but no single situation will be convincing for that and you'll need to make some case that a policy change to move to some other point in this space will yield a positive total improvement.
  - 6y ago