3 ms·

Do you really think that the current situation poses a >25% cost on scientific productivity? Do you think your system would be able to recapture that? That as

Do you really think that the current situation poses a >25% cost on scientific productivity? Do you think your system would be able to recapture that? That assessment does not match up with what any practicing scientist thinks is even within the realm of possibility for harm to science. Reading these conversations is like listening to C-suite execs at big companies talk about what employees are getting away with via work at home policies.

lastiteration 2y ago
Most stuff in papers can't be replicated so you can't really trust anything and are forced to see what actually works and is worth building upon. This is very expensive both in time and money.
j-wags 2y ago
> Do you really think that the current situation poses a >25% cost on scientific productivity? Do you think your system would be able to recapture that? Yes and yes. I'm 6 years past defending my PhD and I have low confidence in being able to reproduce results from papers in my field (computational biophysics). I was recently at an industry-heavy biophysics conference that ran a speed dating event, and my conversation starter was "what fraction of papers in our field do you trust?". I probably talked to ~20 people, with a median response of ~25%. Even a tiny amount invested in reproduction studies and accountability would go a long way. Most papers in _computational_ biophysics still don't publish usable code and data.
- BenFranklin100 2y ago
  It’s bad enough that too often I trust companies over academics nowadays. At the end of the day, a company has to answer to the customer. If what they offer doesn’t actually work, they go out of business. Academics often don’t have to answer to anyone. Just be smart and make the paper look good while being careful not to do something that could get you nailed for outright fraud.
  - SJC_Hacker 2y ago
    > If what they offer doesn’t actually work, they go out of business If this happens before founders/early investors aren't the ones left without a chair when the music stops, it doesn't matter See: Theranos.
  - mangamadaiyan 2y ago
    Actually, being answerable to the customer is not universally true. Typically, most companies are answerable to the investors and shareholders. Customers usually don't figure in the equation.
BenFranklin100 2y ago
We can quibble over the number, maybe it is low as 10%. The cost to reproduce a study will be significantly higher than to produce it in the first place, due to different expertises, equipment and so forth. I estimate at least a 10X factor. And I am intimately familiar with what researchers “get away with’ while ‘working at home’. As a researcher who tried to reproduce several research papers only to discover the original scientists were wildly exaggerating their claims or cleverly disguising fundamental shortcomings, I can assure the cost is quite high to the scientific community, well in excess of 25% of the annual $48B NIH budget. I hold a healthy disdain for my fellow scientists. The only way to get them to play by the rules in my view is to have a threat of a research audit hanging over them.
- nextos 2y ago
  This is the key part, even legitimate findings tend to be exaggerated. A great step forward would be to distribute grants to reproduce new key findings. For example, the NIH could identify the top findings from 2024 that need to be reproduced, and seek expressions of interest to reproduce these and/or other important findings identified by applicants. Perhaps, also reach an agreement with top journals to publish replications as a new article type, and link them to the original one, just like they do with comments/news & views. It would instantly make those publishing super edgy findings much more careful, just in case, and things would become more efficient.
  - BenFranklin100 2y ago
    I love the idea of putting each year’s top papers under the microscope. Present day hucksters publishing in Nature will think thrice about making unsupported claims. Further, the automatic reproduction of the research will serve as cement in the fundamental building block nature of science.
    - nextos 2y ago
      Yes, exactly. It'd act as deterrent for anyone considering to exaggerate or misrepresent findings. Also, it'd lead to a quicker dissemination of true findings. Currently, academic publications are in a bit of market for lemons situation [1], where the seller (authors) have much more information than the buyers (readers, funders). Time to change that. [1] https://en.wikipedia.org/wiki/The_Market_for_Lemons https://en.wikipedia.org/wiki/The_Market_for_Lemons
passwordoops 2y ago
In the 2000s a friend worked at a top 10 pharma company with the fun job of reproducing results from genetics papers... 25% were reproducible. The rest were not, even after communicating with the authors to get conditions as close as possible. So, yes, the current situation can safely be assumed to pose at least a 25% cost on science. And "productivity" is the wrong term here. The harm of fraudulent/bad science runs much deeper than productivity
- epistasis 2y ago
  I don't know about your friend, but there was an option piece, not a scientific study, that reported about that number of "landmark" cancer research papers as being "not-reproducible" in industry hands. However, it did not have methods, it didn't say how they were not reproducible, as in a figure or an effect etc. The closest thing to a definition of "reproducible" was a footnote on a table defining it as "sufficient to drive a drug development program," which is not at all the same thing as reproducible. Which is to say, I'm skeptical of these anecdotes.
  - passwordoops 2y ago
    Here is an overview of the three papers of the "Reproducibility Project: Cancer Biology" (1). The detailed results are in references 2-3 And if you've only ever encountered a single opinion piece on the reproducibility problem in biology/pre-clinical research, then I highly recommend you do a targeted keyword search. (1) https://elifesciences.org/articles/75830 https://elifesciences.org/articles/75830
    - epistasis 2y ago
      I have read that paper too, but you were referencing the ~25% from your friend in industry, which corresponds to the editorial I cited, not these far more rigorous studies. > then I highly recommend you do a targeted keyword search. There's no reason to be insulting, especially when linking to well known studies.

lastiteration 2y ago

Most stuff in papers can't be replicated so you can't really trust anything and are forced to see what actually works and is worth building upon. This is very expensive both in time and money.

j-wags 2y ago

> Do you really think that the current situation poses a >25% cost on scientific productivity? Do you think your system would be able to recapture that? Yes and yes. I'm 6 years past defending my PhD and I have low confidence in being able to reproduce results from papers in my field (computational biophysics). I was recently at an industry-heavy biophysics conference that ran a speed dating event, and my conversation starter was "what fraction of papers in our field do you trust?". I probably talked to ~20 people, with a median response of ~25%. Even a tiny amount invested in reproduction studies and accountability would go a long way. Most papers in _computational_ biophysics still don't publish usable code and data.

BenFranklin100 2y ago
It’s bad enough that too often I trust companies over academics nowadays. At the end of the day, a company has to answer to the customer. If what they offer doesn’t actually work, they go out of business. Academics often don’t have to answer to anyone. Just be smart and make the paper look good while being careful not to do something that could get you nailed for outright fraud.
- SJC_Hacker 2y ago
  > If what they offer doesn’t actually work, they go out of business If this happens before founders/early investors aren't the ones left without a chair when the music stops, it doesn't matter See: Theranos.
- mangamadaiyan 2y ago
  Actually, being answerable to the customer is not universally true. Typically, most companies are answerable to the investors and shareholders. Customers usually don't figure in the equation.

BenFranklin100 2y ago

We can quibble over the number, maybe it is low as 10%. The cost to reproduce a study will be significantly higher than to produce it in the first place, due to different expertises, equipment and so forth. I estimate at least a 10X factor. And I am intimately familiar with what researchers “get away with’ while ‘working at home’. As a researcher who tried to reproduce several research papers only to discover the original scientists were wildly exaggerating their claims or cleverly disguising fundamental shortcomings, I can assure the cost is quite high to the scientific community, well in excess of 25% of the annual $48B NIH budget. I hold a healthy disdain for my fellow scientists. The only way to get them to play by the rules in my view is to have a threat of a research audit hanging over them.

nextos 2y ago
This is the key part, even legitimate findings tend to be exaggerated. A great step forward would be to distribute grants to reproduce new key findings. For example, the NIH could identify the top findings from 2024 that need to be reproduced, and seek expressions of interest to reproduce these and/or other important findings identified by applicants. Perhaps, also reach an agreement with top journals to publish replications as a new article type, and link them to the original one, just like they do with comments/news & views. It would instantly make those publishing super edgy findings much more careful, just in case, and things would become more efficient.
- BenFranklin100 2y ago
  I love the idea of putting each year’s top papers under the microscope. Present day hucksters publishing in Nature will think thrice about making unsupported claims. Further, the automatic reproduction of the research will serve as cement in the fundamental building block nature of science.
  - nextos 2y ago
    Yes, exactly. It'd act as deterrent for anyone considering to exaggerate or misrepresent findings. Also, it'd lead to a quicker dissemination of true findings. Currently, academic publications are in a bit of market for lemons situation [1], where the seller (authors) have much more information than the buyers (readers, funders). Time to change that. [1] https://en.wikipedia.org/wiki/The_Market_for_Lemons https://en.wikipedia.org/wiki/The_Market_for_Lemons

passwordoops 2y ago

In the 2000s a friend worked at a top 10 pharma company with the fun job of reproducing results from genetics papers... 25% were reproducible. The rest were not, even after communicating with the authors to get conditions as close as possible. So, yes, the current situation can safely be assumed to pose at least a 25% cost on science. And "productivity" is the wrong term here. The harm of fraudulent/bad science runs much deeper than productivity

epistasis 2y ago
I don't know about your friend, but there was an option piece, not a scientific study, that reported about that number of "landmark" cancer research papers as being "not-reproducible" in industry hands. However, it did not have methods, it didn't say how they were not reproducible, as in a figure or an effect etc. The closest thing to a definition of "reproducible" was a footnote on a table defining it as "sufficient to drive a drug development program," which is not at all the same thing as reproducible. Which is to say, I'm skeptical of these anecdotes.
- passwordoops 2y ago
  Here is an overview of the three papers of the "Reproducibility Project: Cancer Biology" (1). The detailed results are in references 2-3 And if you've only ever encountered a single opinion piece on the reproducibility problem in biology/pre-clinical research, then I highly recommend you do a targeted keyword search. (1) https://elifesciences.org/articles/75830 https://elifesciences.org/articles/75830
  - epistasis 2y ago
    I have read that paper too, but you were referencing the ~25% from your friend in industry, which corresponds to the editorial I cited, not these far more rigorous studies. > then I highly recommend you do a targeted keyword search. There's no reason to be insulting, especially when linking to well known studies.