6 ms·

I have long maintained that the NIH should set aside 25% of its budget to fund spot checking of the research output of the remaining 75% of funded research. Eve

I have long maintained that the NIH should set aside 25% of its budget to fund spot checking of the research output of the remaining 75% of funded research. Even if this funding is only sufficient to review a handful of papers each year, it stands to have a transformative effect on the level of hype and p-hacking in many fields, and could nearly eliminate the rare cases of actual fraud. It would also improve the overall quality and reliability of the literature too.

epgui 2y ago
I think if we devoted only 1% to this it would be a huge improvement.
- ta988 2y ago
  Agreed, once you know what to look for and how to reproduce it. What do you do if you can't reproduce? That may mean the original research paper doesn't disclose everything (malicious or not, but malicious is REALLY frequent) or missed an important factor (sometimes doing a reaction in a slightly scratched glass container will change the outcome entirely). To come back to the malicious part, for many researchers, not publishing the exact way they do things is part of how they protect themselves from people reproducing their work. Some do it for money (they want to start a business from that research), others to avoid competition, others because they believe they own the publicly funded research...
  - jltsiren 2y ago
    And sometimes you fail to reproduce something because you failed to do it properly. I don't know how often that happens in the field on in the lab, but it's extremely common on the computational side. Very often, the thing you are trying to reproduce isn't exactly the same that was published. You have to adapt the instructions to your specific case, which can easily go wrong. Or maybe you did a mistake in following the instructions. Or maybe you mixed the instructions for two different cases, because you didn't fully understand the subtleties of the topic. Or maybe you made a mistake in translating the provided scripts to your institute's computational environment.
    - subroutine 2y ago
      Part of the problem is that methods sections in contemporary journals do not provide enough information for exact replication, and in the most egregious cases let authors write stuff like "cultured cells prepared according to prevailing standards".
      - fhsm 2y ago
        Or it’s exactly what was published but the replicator lacks the ability , eg very famously https://www.nature.com/articles/nm1002-1052 https://www.nature.com/articles/nm1002-1052
        pas 2y ago
        From the site: "One of the problems was room temperature was too low elsewhere. Tumors don't grow when mice are chilly because the vessels vasocontrict, it cuts off the blood supply and drugs don't circulate.” That means there are important validation/verification steps left out of the whole process. Sure, it's impossible to give every detail, and naturally there's always a time constraint, but if there's a hypothesis of action it needs to be verified. (Again easier said than done.)
      - epgui 2y ago
        In biochem sometimes there are a lot of skills involved, to the point where it's almost magic intangible qualities making an experiment succeed. Especially for more manual work. For my masters' research I spent 6 years refining a super niche technique until I was able to reproduce my own work.
        ta988 2y ago
        Did you end up publishing the details on how you did it?
      - ta988 2y ago
        No if you want you have virtually unlimited supplementary information you can attach to your publication. It is really about a mix of doing as little as possible and hiding details so competitors can't do it.
faeriechangling 2y ago
It could also stand to go out of its way to try and give more prestige to people who manage to get influential papers retracted.
- spamizbad 2y ago
  This is already happening in academia.
  - passwordoops 2y ago
    You're going to have to provide evidence, as in actual bona-fide promotions handed out, or Tenure committees expressly spelling out and counting this as a factor in their decision
- HPsquared 2y ago
  Unfortunately there isn't a neat mechanism like short-selling.
  - supriyo-biswas 2y ago
    Income tax departments in my country give a small reward (3-5% of the amount in question) if you report tax fraud. Maybe we could have something like that, where the vigilance department receives a small amount of money paid off from the penalty imposed on the researcher with bad/fraudulent research.
    - thechao 2y ago
      See, that’s weird to me? Wouldn’t the best reward be more like 99 or 100%? The gov’t’s already out the money; the incentive is to prevent tax fraud.
      - supriyo-biswas 2y ago
        The tax department is, in fact, not "out of money" in that sense, and tries to recover the money first with some penalties added. They're incredibly empowered to conduct raids and seize property and other belongings, if it comes down to that and the sum involved is large enough.
ThrowawayTestr 2y ago
If I was rich I'd start a journal solely dedicated to publishing papers trying to recreate other papers.
- deleted 2y ago
  [deleted]
- 317070 2y ago
  Wait, you don't need to be rich. With pay-to-publish (also known as free access), it is an absolute goldmine. People generally don't want to do the work of editing and publishing, or lack the academic knowhow to do it. But if that is not an issue, I don't think money will be an issue either.
  - caddemon 2y ago
    There isn't enough incentive currently to publish reproductions, starting a new journal using the same general publishing model isn't going to change that. But with money to burn you could add some incentive, or you could at least do things to improve publication quality like actually paying for good peer reviewers.
- burkaman 2y ago
  Here's one effort, unfortunately it looks like it never got off the ground: http://rescience.org/x http://rescience.org/x. Here's their motivation in a Nature letter to the editor: https://media.nature.com/original/magazine-assets/d41586-020-01328-2/d41586-020-01328-2.pdf https://media.nature.com/original/magazine-assets/d41586-020...
  - nextos 2y ago
    As I said in another comment, I think it'd be much easier to convince Nature to create a new article type, Replication Study. Nature, as well as other top journals, do not publish results that report already published findings. Replication Studies would be the exception. These would provide independent validation of a recent prominent article. People would think twice before misrepresenting findings in top journals, and good ideas would spread out more quickly.
_DeadFred_ 2y ago
Better yet have bounties like how the tech world has bug bounties.
epistasis 2y ago
Do you really think that the current situation poses a >25% cost on scientific productivity? Do you think your system would be able to recapture that? That assessment does not match up with what any practicing scientist thinks is even within the realm of possibility for harm to science. Reading these conversations is like listening to C-suite execs at big companies talk about what employees are getting away with via work at home policies.
- lastiteration 2y ago
  Most stuff in papers can't be replicated so you can't really trust anything and are forced to see what actually works and is worth building upon. This is very expensive both in time and money.
- j-wags 2y ago
  > Do you really think that the current situation poses a >25% cost on scientific productivity? Do you think your system would be able to recapture that? Yes and yes. I'm 6 years past defending my PhD and I have low confidence in being able to reproduce results from papers in my field (computational biophysics). I was recently at an industry-heavy biophysics conference that ran a speed dating event, and my conversation starter was "what fraction of papers in our field do you trust?". I probably talked to ~20 people, with a median response of ~25%. Even a tiny amount invested in reproduction studies and accountability would go a long way. Most papers in _computational_ biophysics still don't publish usable code and data.
  - BenFranklin100 2y ago
    It’s bad enough that too often I trust companies over academics nowadays. At the end of the day, a company has to answer to the customer. If what they offer doesn’t actually work, they go out of business. Academics often don’t have to answer to anyone. Just be smart and make the paper look good while being careful not to do something that could get you nailed for outright fraud.
    - SJC_Hacker 2y ago
moneywoes 2y ago
what’s the incentive for them?
- AlexCoventry 2y ago
  Perhaps it could be mandated by legislation or executive order.
- ptero 2y ago
  NIH (as many other grant giving sources) is a government agency, they will do this if told by the government.
  - SJC_Hacker 2y ago
    The incentive to the PI (professor). "I replicated X's work" or even "I was unable to replicate X's work" isn't exactly a career maker Although I guess you could get the few staff scientists at the NIH to handle it. Some older professors who already have tenure might be willing to help out, if they don't already have much else on their plate
    - hoseja 2y ago
      There are oodles of people doing rote labwork/innovation-less but nonetheless skilled scientific-like work. How are they motivated? Academia's perverse reputational economy is the aberration, not the norm.
- yencabulator 2y ago
  Improving quality of health research seems pretty well aligned with their mission. https://www.nih.gov/about-nih/what-we-do/mission-goals https://www.nih.gov/about-nih/what-we-do/mission-goals
throwaway984393 2y ago
[dead]
ptero 2y ago
Yes, and also require publishing the full methodology for any article funded by public money. This should make verification easier and cheaper. Not necessarily published as part of the paper, a link to the separate document is fine.
idkdotcom 2y ago
[dead]
Helmut10001 2y ago
But who could do the spot-checking? In many fields, there is only one team or a few authors capable of understanding, reproducing, or verifying certain research. In a two-party situation, if you pay the opposing team, they have incentives to exaggerate doubts. Often research papers are written in such a confusing way that it is impossible or very expensive to reproduce or verify the results (e.g. repeat a year-long study). I think it would be better if there were incentives that rewarded quality over quantity. At the moment, my university always says that quality is of the utmost importance, but then threatens to terminate my job if I cannot publish x number of papers in a given year.
- JPLeRouzic 2y ago
  > Often research papers are written in such a confusing way that it is impossible or very expensive to reproduce or verify the results (e.g. repeat a year-long study) From an economic perspective, is this a very desirable situation?
- logicchains 2y ago
  >, there is only one team or a few authors capable of understanding, reproducing, or verifying certain research If the research isn't documented well enough to reproduce/verify, then the paper shouldn't pass review in the first place. The NIH could make it a condition of funding that papers are detailed enough to be reproducible.
  - Helmut10001 2y ago
    My sense is that this would prevent 98% of papers in the social sciences from ever being published. How do you decide whether to renew a researcher's position when the average time to get a single paper through peer review is 5+ years (my average is 2-3 contract renewals per year)? This is not compatible with today's pace. I am not saying that all this is a desirable situation. It is very unfortunate, and I wish there was an easy solution. My first research paper took 5 major revisions and 6 years to get through peer review. All the reviewers criticized was the wording and my unwillingness to conform to the accepted views in that particular community; I almost lost my job over this, but once the paper was accepted, it won several awards. All of this leads me to believe that peer review is very subjective and prone to error, and I don't have a solution for that.
staunton 2y ago
I think part of the reason why nobody seems serious about replication is that it's actually mostly not worth it. Most research is useless and pointless, with only a few exceptions. We don't have a way to figure out which topics are the exceptions, so someone has to do the research. It's not worth it (or rather: extremely high financial risk) for companies or individuals to do it, so governments gave to fund it. At this point, the current amount of fraud does not justify replicating even 1% of studies. We would get less scientific advancement in total. The current situation likely does justify some small investments in shaping incentives.
- sgt101 2y ago
  >>It's not worth it (or rather: extremely high financial risk) for companies or individuals to do it The problem is that it's hard to reliably capture value from research. A good example is LLM's. If OpenAI, Google, Meta and AWS had been able to build a wall round GTP3.5 Turbo and above models then I expect that they could have captured all the value of the research effort.. as it is I don't think that is/will be the case - it's almost too easy to replicate as Mistral have shown. Note: I'm not saying it's trivial or something, but if you spend a few $million on it you can get close enough, and then spending a few $million more will get you all the way. Also, I am not talking about building a frontier model today (which requires $100millon or so and some difficult skills/organisation) but rather a model in say 3 years time with the frontier performance of todays models.

epgui 2y ago

I think if we devoted only 1% to this it would be a huge improvement.

ta988 2y ago
Agreed, once you know what to look for and how to reproduce it. What do you do if you can't reproduce? That may mean the original research paper doesn't disclose everything (malicious or not, but malicious is REALLY frequent) or missed an important factor (sometimes doing a reaction in a slightly scratched glass container will change the outcome entirely). To come back to the malicious part, for many researchers, not publishing the exact way they do things is part of how they protect themselves from people reproducing their work. Some do it for money (they want to start a business from that research), others to avoid competition, others because they believe they own the publicly funded research...
- jltsiren 2y ago
  And sometimes you fail to reproduce something because you failed to do it properly. I don't know how often that happens in the field on in the lab, but it's extremely common on the computational side. Very often, the thing you are trying to reproduce isn't exactly the same that was published. You have to adapt the instructions to your specific case, which can easily go wrong. Or maybe you did a mistake in following the instructions. Or maybe you mixed the instructions for two different cases, because you didn't fully understand the subtleties of the topic. Or maybe you made a mistake in translating the provided scripts to your institute's computational environment.
  - subroutine 2y ago
    Part of the problem is that methods sections in contemporary journals do not provide enough information for exact replication, and in the most egregious cases let authors write stuff like "cultured cells prepared according to prevailing standards".
    - fhsm 2y ago
      Or it’s exactly what was published but the replicator lacks the ability , eg very famously https://www.nature.com/articles/nm1002-1052 https://www.nature.com/articles/nm1002-1052
      - pas 2y ago
        From the site: "One of the problems was room temperature was too low elsewhere. Tumors don't grow when mice are chilly because the vessels vasocontrict, it cuts off the blood supply and drugs don't circulate.” That means there are important validation/verification steps left out of the whole process. Sure, it's impossible to give every detail, and naturally there's always a time constraint, but if there's a hypothesis of action it needs to be verified. (Again easier said than done.)
    - epgui 2y ago
      In biochem sometimes there are a lot of skills involved, to the point where it's almost magic intangible qualities making an experiment succeed. Especially for more manual work. For my masters' research I spent 6 years refining a super niche technique until I was able to reproduce my own work.
      - ta988 2y ago
        Did you end up publishing the details on how you did it?
    - ta988 2y ago
      No if you want you have virtually unlimited supplementary information you can attach to your publication. It is really about a mix of doing as little as possible and hiding details so competitors can't do it.

faeriechangling 2y ago

It could also stand to go out of its way to try and give more prestige to people who manage to get influential papers retracted.

spamizbad 2y ago
This is already happening in academia.
- passwordoops 2y ago
  You're going to have to provide evidence, as in actual bona-fide promotions handed out, or Tenure committees expressly spelling out and counting this as a factor in their decision
HPsquared 2y ago
Unfortunately there isn't a neat mechanism like short-selling.
- supriyo-biswas 2y ago
  Income tax departments in my country give a small reward (3-5% of the amount in question) if you report tax fraud. Maybe we could have something like that, where the vigilance department receives a small amount of money paid off from the penalty imposed on the researcher with bad/fraudulent research.
  - thechao 2y ago
    See, that’s weird to me? Wouldn’t the best reward be more like 99 or 100%? The gov’t’s already out the money; the incentive is to prevent tax fraud.
    - supriyo-biswas 2y ago
      The tax department is, in fact, not "out of money" in that sense, and tries to recover the money first with some penalties added. They're incredibly empowered to conduct raids and seize property and other belongings, if it comes down to that and the sum involved is large enough.

ThrowawayTestr 2y ago

If I was rich I'd start a journal solely dedicated to publishing papers trying to recreate other papers.

deleted 2y ago
[deleted]
317070 2y ago
Wait, you don't need to be rich. With pay-to-publish (also known as free access), it is an absolute goldmine. People generally don't want to do the work of editing and publishing, or lack the academic knowhow to do it. But if that is not an issue, I don't think money will be an issue either.
- caddemon 2y ago
  There isn't enough incentive currently to publish reproductions, starting a new journal using the same general publishing model isn't going to change that. But with money to burn you could add some incentive, or you could at least do things to improve publication quality like actually paying for good peer reviewers.
burkaman 2y ago
Here's one effort, unfortunately it looks like it never got off the ground: http://rescience.org/x http://rescience.org/x. Here's their motivation in a Nature letter to the editor: https://media.nature.com/original/magazine-assets/d41586-020-01328-2/d41586-020-01328-2.pdf https://media.nature.com/original/magazine-assets/d41586-020...
- nextos 2y ago
  As I said in another comment, I think it'd be much easier to convince Nature to create a new article type, Replication Study. Nature, as well as other top journals, do not publish results that report already published findings. Replication Studies would be the exception. These would provide independent validation of a recent prominent article. People would think twice before misrepresenting findings in top journals, and good ideas would spread out more quickly.

_DeadFred_ 2y ago

Better yet have bounties like how the tech world has bug bounties.

epistasis 2y ago

Do you really think that the current situation poses a >25% cost on scientific productivity? Do you think your system would be able to recapture that? That assessment does not match up with what any practicing scientist thinks is even within the realm of possibility for harm to science. Reading these conversations is like listening to C-suite execs at big companies talk about what employees are getting away with via work at home policies.

lastiteration 2y ago
Most stuff in papers can't be replicated so you can't really trust anything and are forced to see what actually works and is worth building upon. This is very expensive both in time and money.
j-wags 2y ago
> Do you really think that the current situation poses a >25% cost on scientific productivity? Do you think your system would be able to recapture that? Yes and yes. I'm 6 years past defending my PhD and I have low confidence in being able to reproduce results from papers in my field (computational biophysics). I was recently at an industry-heavy biophysics conference that ran a speed dating event, and my conversation starter was "what fraction of papers in our field do you trust?". I probably talked to ~20 people, with a median response of ~25%. Even a tiny amount invested in reproduction studies and accountability would go a long way. Most papers in _computational_ biophysics still don't publish usable code and data.
- BenFranklin100 2y ago
  It’s bad enough that too often I trust companies over academics nowadays. At the end of the day, a company has to answer to the customer. If what they offer doesn’t actually work, they go out of business. Academics often don’t have to answer to anyone. Just be smart and make the paper look good while being careful not to do something that could get you nailed for outright fraud.
  - SJC_Hacker 2y ago

moneywoes 2y ago

what’s the incentive for them?

AlexCoventry 2y ago
Perhaps it could be mandated by legislation or executive order.
ptero 2y ago
NIH (as many other grant giving sources) is a government agency, they will do this if told by the government.
- SJC_Hacker 2y ago
  The incentive to the PI (professor). "I replicated X's work" or even "I was unable to replicate X's work" isn't exactly a career maker Although I guess you could get the few staff scientists at the NIH to handle it. Some older professors who already have tenure might be willing to help out, if they don't already have much else on their plate
  - hoseja 2y ago
    There are oodles of people doing rote labwork/innovation-less but nonetheless skilled scientific-like work. How are they motivated? Academia's perverse reputational economy is the aberration, not the norm.
yencabulator 2y ago
Improving quality of health research seems pretty well aligned with their mission. https://www.nih.gov/about-nih/what-we-do/mission-goals https://www.nih.gov/about-nih/what-we-do/mission-goals

throwaway984393 2y ago

[dead]

ptero 2y ago

Yes, and also require publishing the full methodology for any article funded by public money. This should make verification easier and cheaper. Not necessarily published as part of the paper, a link to the separate document is fine.

idkdotcom 2y ago

[dead]

Helmut10001 2y ago

But who could do the spot-checking? In many fields, there is only one team or a few authors capable of understanding, reproducing, or verifying certain research. In a two-party situation, if you pay the opposing team, they have incentives to exaggerate doubts. Often research papers are written in such a confusing way that it is impossible or very expensive to reproduce or verify the results (e.g. repeat a year-long study). I think it would be better if there were incentives that rewarded quality over quantity. At the moment, my university always says that quality is of the utmost importance, but then threatens to terminate my job if I cannot publish x number of papers in a given year.

JPLeRouzic 2y ago
> Often research papers are written in such a confusing way that it is impossible or very expensive to reproduce or verify the results (e.g. repeat a year-long study) From an economic perspective, is this a very desirable situation?
logicchains 2y ago
>, there is only one team or a few authors capable of understanding, reproducing, or verifying certain research If the research isn't documented well enough to reproduce/verify, then the paper shouldn't pass review in the first place. The NIH could make it a condition of funding that papers are detailed enough to be reproducible.
- Helmut10001 2y ago
  My sense is that this would prevent 98% of papers in the social sciences from ever being published. How do you decide whether to renew a researcher's position when the average time to get a single paper through peer review is 5+ years (my average is 2-3 contract renewals per year)? This is not compatible with today's pace. I am not saying that all this is a desirable situation. It is very unfortunate, and I wish there was an easy solution. My first research paper took 5 major revisions and 6 years to get through peer review. All the reviewers criticized was the wording and my unwillingness to conform to the accepted views in that particular community; I almost lost my job over this, but once the paper was accepted, it won several awards. All of this leads me to believe that peer review is very subjective and prone to error, and I don't have a solution for that.

staunton 2y ago

I think part of the reason why nobody seems serious about replication is that it's actually mostly not worth it. Most research is useless and pointless, with only a few exceptions. We don't have a way to figure out which topics are the exceptions, so someone has to do the research. It's not worth it (or rather: extremely high financial risk) for companies or individuals to do it, so governments gave to fund it. At this point, the current amount of fraud does not justify replicating even 1% of studies. We would get less scientific advancement in total. The current situation likely does justify some small investments in shaping incentives.

sgt101 2y ago
>>It's not worth it (or rather: extremely high financial risk) for companies or individuals to do it The problem is that it's hard to reliably capture value from research. A good example is LLM's. If OpenAI, Google, Meta and AWS had been able to build a wall round GTP3.5 Turbo and above models then I expect that they could have captured all the value of the research effort.. as it is I don't think that is/will be the case - it's almost too easy to replicate as Mistral have shown. Note: I'm not saying it's trivial or something, but if you spend a few $million on it you can get close enough, and then spending a few $million more will get you all the way. Also, I am not talking about building a frontier model today (which requires $100millon or so and some difficult skills/organisation) but rather a model in say 3 years time with the frontier performance of todays models.