6 ms·

Even if academics could review all the papers on a preprint server (which the article argues -- rightly -- that they can't), it wouldn't solve the perceived pro

by timr 3y ago

Even if academics could review all the papers on a preprint server (which the article argues -- rightly -- that they can't), it wouldn't solve the perceived problems (or the actual problems) with scientific peer review. The vast majority of irreproducible papers aren't detectible as irreproducible at time of publication. They look fine, and many actually are fine. They just don't reproduce. That's an expected outcome in science. The system will self-correct over time. IMO, the main actual problem with peer review is that non-practitioners put too much faith in it. Nobody in science actually takes a paper on faith because it's been published, and you shouldn't either. Peer review is little more than a lightweight safeguard against complete nonsense being published. It barely works for that. Just because you found a paper doesn't mean you should believe it. You have to understand it. A secondary actual problem is that it's impossible to reproduce a lot of papers, or they're methodologically broken from the start (e.g. RCTs that are not pre-registered, or observational studies without control groups). These are problems we could actually solve. For example, just requiring that any paper publish the raw study data would help to self-control the system. There are high-profile researchers out there, right now, who do little more than statistically mine the same secret data set -- these people are likely publishing crap, but we have no way to prove it, because the data is secret.

s0rce 3y ago
There is no place to make a note that something doesn't reproduce so its extremely time consuming or you need some source of tribal knowledge. In my postdoc I was trying to make some porous films and a bunch of paper's methods didn't seem to work, maybe I did it wrong, maybe some detail wasn't described, who knows. I couldn't get it to work and there was no way to document that failure.
- bumby 3y ago
  I wonder if there's a way to document such replaceability failures as an erratum to the original manuscript. I feel like this would help in at least two major ways: 1) It provides a reproducibility filter. If a method isn't shown to be reproducible, publically documenting that adds to the body of knowledge, and this would help drive an incentive towards reproducing work rather than just searching for novelty. It would document work that would otherwise be lost because there's no incentive to showcase it. When the lack of reproducible results isn't public, it's now more likely that others may waste considerable effort in the same vein. 3) It may enlist the original authors to help understand why the work didn't reproduce well. Maybe the secondary effort lacked some crucial step or understanding. The people best positioned to remedy this are the original authors, and this secondary publication incentivizes them to dialogue with those who couldn't reproduce the outcome. It doesn't mean they have to engage, but it at least gives them some reason to involve themselves in the process.
  - warkdarrior 3y ago
    2) ???
    - bumby 3y ago
      If you are willing to provide a more substantive response, I might be able to clarify.
      - s0rce 3y ago
        I think they meant that you only list 1) and 3) and 2) was missing.
Retric 3y ago
Peer review is a spam filter, and it’s quite useful in that content. But it’s a spam filer for people who filter out 99.9+% of papers by having such a narrow scope they probably recognize several names on a given paper.
pacbard 3y ago
To your second point, I alway go back to this quote: "You can't fix by analysis what you bungled by design" (Light, Singer and Willett, 1990). If a paper is broken by design, there isn't much to do after the fact. It's just broken. The problem is that doing a good RCT takes both time and effort, with the huge risk of having null results, which usually results in a desk rejection from most top journals. So, you either are a top-fund raising researcher who can both fund multiple RCTs and people to support them, or you just try your best with what you have and hope to squeeze a paper out from you did. Releasing the data won't really help much if the data generating process is flawed. Sure, other people will be able to run different kind of analyses (e.g., jackknife your standard errors instead of just using a robust correction), but I'm not sure how helpful that will be. A third issue that I have also encountered is that journal editors have an agenda when putting together an issue, which sometimes overwrites the "quality' of the research with "fit" to the issue. This could lead to "lower quality" articles to be published because they fit the (often unspoken) direction of the journal. Most editors see their role as steering the field towards new directions (a sort of a meta service to the field) and sometimes that comes at the expense of the quality of the work.
- YeGoblynQueenne 3y ago
  >> (Light, Singer and Willett, 1990) A citation like the one above should normally point to a full reference in a bibliography section. Did you forget the \bibliography{} command at the end of your comment?
  - anticensor 3y ago
    Hacker News does not have a LaTeX compiler.
- timr 3y ago
  > Releasing the data won't really help much if the data generating process is flawed. Sure, other people will be able to run different kind of analyses (e.g., jackknife your standard errors instead of just using a robust correction), but I'm not sure how helpful that will be. It allows motivated people to catch more subtle forms of nonsense. Data colada, for example, has caught outright fraud, but only through herculean efforts. Imagine what groups like this might do if they had the raw data.
barfbagginus 3y ago
In AI work - which naturally lends itself to replicability improvements - we could get truly solid replicability by ratcheting up the standards for code quality, testing, and automation in AI projects. And I think llms can start doing a lot of that kind of QA and engineering / pre-operationalization work, because the best llms are already better at software and value engineering than the average postdoc AI researcher. Most AI codes are missing key replicability factors - either the training data/trainer are missing, the code has a poor testing / benchmark automation strategy, the code documentation is meager, or there's no real CI/CD practice for advancing the project or operationalizing it against problems caused by the anthropocentric collapse. Some researchers are even hardened against such things, seeing them as false worship of harmful business metrics, rather than a fundamental duty that could really improve the impact of their research, and it's applicability towards a universal crisis that faces us all. But we can put the lie to this view with just one glance at their code. Too much additional work is necessary to turn it into anything useful, either for further research iterations or productive operationalization. The gaps in code quality exist not because that form of code is optimal for research aims, but because researchers lack software engineering expertise, and cannot afford software engineering labor. But thankfully the level of software engineering labor is not even that great - llms can now help swing that effort. As a result I believe that we should work to create standards for AI assisted research repos that correct the major deficits of replicability, usability, and code quality that we see in most AI repos. Then we should campaign to adopt those standards into peer review. Let one of the reviewers be an AI that really grills your code on its quality. And actually incorporate the PRs that it proposes. I think that would change the situation, from the current standard where academic AI repos are mainly nonreplicating throw-away code, to an opposite situation where the majority of AI research repos are easy to replicate, improve, and mobilize against the social and environmental problems facing humanity, as it navigates through the anthropocene disaster.
ramblenode 3y ago
> The vast majority of irreproducible papers aren't detectible as irreproducible at time of publication. They look fine, and many actually are fine. They just don't reproduce. That's an expected outcome in science. This is not entirely true. A power analysis is how you determine reproducibility, and researchers should be doing it before they begin collecting data. Reviewers can do it post-hoc with assumptions about the expected effect size (which might come from similar studies). False positives produce inflated effect sizes, so if a result is marginally significant but shows a large effect, that is a good heuristic the result will not reproduce.
- timr 3y ago
  > A power analysis is how you determine reproducibility All a power analysis does is reduce the chance that the result is a false negative. It doesn't reduce the chance of a false positive. > False positives produce inflated effect sizes Not always. Lots of studies publish as "significant" as soon as they get a p-value just under .05. Inflated effect sizes are certainly a sign that something could be wrong, but it's just one indicator. Regardless, even if you have a power analysis at the conventional threshold of 80%, and a p-value of .05, you're still going to get spurious positive results 5% of the time, and spurious negative results 20% of the time, by definition.
  - ramblenode 3y ago
    > All a power analysis does is reduce the chance that the result is a false negative. It doesn't reduce the chance of a false positive. This is true when we are dealing with an uninformative prior, but published research is known to be biased toward positive results and uncorrected multiple comparisons. This situation leads to small sample studies with high random variance being paradoxically correlated with significant results. High random variance appears as a false large effect size in the published result, so if the power is low when calculated with a smaller (adjusted) effect, there is reason to believe that the p-value is inflated. See e.g. Andrew Gelman's work on small sample studies, garden of forking paths or [0]. > Not always. Lots of studies publish as "significant" as soon as they get a p-value just under .05. Inflated effect sizes are certainly a sign that something could be wrong, but it's just one indicator. Exactly! The implication being the above. [0] https://en.wikipedia.org/wiki/Why_Most_Published_Research_Findings_Are_False https://en.wikipedia.org/wiki/Why_Most_Published_Research_Fi...

by timr 3y ago

s0rce 3y ago

There is no place to make a note that something doesn't reproduce so its extremely time consuming or you need some source of tribal knowledge. In my postdoc I was trying to make some porous films and a bunch of paper's methods didn't seem to work, maybe I did it wrong, maybe some detail wasn't described, who knows. I couldn't get it to work and there was no way to document that failure.

bumby 3y ago
I wonder if there's a way to document such replaceability failures as an erratum to the original manuscript. I feel like this would help in at least two major ways: 1) It provides a reproducibility filter. If a method isn't shown to be reproducible, publically documenting that adds to the body of knowledge, and this would help drive an incentive towards reproducing work rather than just searching for novelty. It would document work that would otherwise be lost because there's no incentive to showcase it. When the lack of reproducible results isn't public, it's now more likely that others may waste considerable effort in the same vein. 3) It may enlist the original authors to help understand why the work didn't reproduce well. Maybe the secondary effort lacked some crucial step or understanding. The people best positioned to remedy this are the original authors, and this secondary publication incentivizes them to dialogue with those who couldn't reproduce the outcome. It doesn't mean they have to engage, but it at least gives them some reason to involve themselves in the process.
- warkdarrior 3y ago
  2) ???
  - bumby 3y ago
    If you are willing to provide a more substantive response, I might be able to clarify.
    - s0rce 3y ago
      I think they meant that you only list 1) and 3) and 2) was missing.

Retric 3y ago

Peer review is a spam filter, and it’s quite useful in that content. But it’s a spam filer for people who filter out 99.9+% of papers by having such a narrow scope they probably recognize several names on a given paper.

pacbard 3y ago

To your second point, I alway go back to this quote: "You can't fix by analysis what you bungled by design" (Light, Singer and Willett, 1990). If a paper is broken by design, there isn't much to do after the fact. It's just broken. The problem is that doing a good RCT takes both time and effort, with the huge risk of having null results, which usually results in a desk rejection from most top journals. So, you either are a top-fund raising researcher who can both fund multiple RCTs and people to support them, or you just try your best with what you have and hope to squeeze a paper out from you did. Releasing the data won't really help much if the data generating process is flawed. Sure, other people will be able to run different kind of analyses (e.g., jackknife your standard errors instead of just using a robust correction), but I'm not sure how helpful that will be. A third issue that I have also encountered is that journal editors have an agenda when putting together an issue, which sometimes overwrites the "quality' of the research with "fit" to the issue. This could lead to "lower quality" articles to be published because they fit the (often unspoken) direction of the journal. Most editors see their role as steering the field towards new directions (a sort of a meta service to the field) and sometimes that comes at the expense of the quality of the work.

YeGoblynQueenne 3y ago
>> (Light, Singer and Willett, 1990) A citation like the one above should normally point to a full reference in a bibliography section. Did you forget the \bibliography{} command at the end of your comment?
- anticensor 3y ago
  Hacker News does not have a LaTeX compiler.
timr 3y ago
> Releasing the data won't really help much if the data generating process is flawed. Sure, other people will be able to run different kind of analyses (e.g., jackknife your standard errors instead of just using a robust correction), but I'm not sure how helpful that will be. It allows motivated people to catch more subtle forms of nonsense. Data colada, for example, has caught outright fraud, but only through herculean efforts. Imagine what groups like this might do if they had the raw data.

barfbagginus 3y ago

In AI work - which naturally lends itself to replicability improvements - we could get truly solid replicability by ratcheting up the standards for code quality, testing, and automation in AI projects. And I think llms can start doing a lot of that kind of QA and engineering / pre-operationalization work, because the best llms are already better at software and value engineering than the average postdoc AI researcher. Most AI codes are missing key replicability factors - either the training data/trainer are missing, the code has a poor testing / benchmark automation strategy, the code documentation is meager, or there's no real CI/CD practice for advancing the project or operationalizing it against problems caused by the anthropocentric collapse. Some researchers are even hardened against such things, seeing them as false worship of harmful business metrics, rather than a fundamental duty that could really improve the impact of their research, and it's applicability towards a universal crisis that faces us all. But we can put the lie to this view with just one glance at their code. Too much additional work is necessary to turn it into anything useful, either for further research iterations or productive operationalization. The gaps in code quality exist not because that form of code is optimal for research aims, but because researchers lack software engineering expertise, and cannot afford software engineering labor. But thankfully the level of software engineering labor is not even that great - llms can now help swing that effort. As a result I believe that we should work to create standards for AI assisted research repos that correct the major deficits of replicability, usability, and code quality that we see in most AI repos. Then we should campaign to adopt those standards into peer review. Let one of the reviewers be an AI that really grills your code on its quality. And actually incorporate the PRs that it proposes. I think that would change the situation, from the current standard where academic AI repos are mainly nonreplicating throw-away code, to an opposite situation where the majority of AI research repos are easy to replicate, improve, and mobilize against the social and environmental problems facing humanity, as it navigates through the anthropocene disaster.

ramblenode 3y ago

> The vast majority of irreproducible papers aren't detectible as irreproducible at time of publication. They look fine, and many actually are fine. They just don't reproduce. That's an expected outcome in science. This is not entirely true. A power analysis is how you determine reproducibility, and researchers should be doing it before they begin collecting data. Reviewers can do it post-hoc with assumptions about the expected effect size (which might come from similar studies). False positives produce inflated effect sizes, so if a result is marginally significant but shows a large effect, that is a good heuristic the result will not reproduce.

timr 3y ago
> A power analysis is how you determine reproducibility All a power analysis does is reduce the chance that the result is a false negative. It doesn't reduce the chance of a false positive. > False positives produce inflated effect sizes Not always. Lots of studies publish as "significant" as soon as they get a p-value just under .05. Inflated effect sizes are certainly a sign that something could be wrong, but it's just one indicator. Regardless, even if you have a power analysis at the conventional threshold of 80%, and a p-value of .05, you're still going to get spurious positive results 5% of the time, and spurious negative results 20% of the time, by definition.
- ramblenode 3y ago
  > All a power analysis does is reduce the chance that the result is a false negative. It doesn't reduce the chance of a false positive. This is true when we are dealing with an uninformative prior, but published research is known to be biased toward positive results and uncorrected multiple comparisons. This situation leads to small sample studies with high random variance being paradoxically correlated with significant results. High random variance appears as a false large effect size in the published result, so if the power is low when calculated with a smaller (adjusted) effect, there is reason to believe that the p-value is inflated. See e.g. Andrew Gelman's work on small sample studies, garden of forking paths or [0]. > Not always. Lots of studies publish as "significant" as soon as they get a p-value just under .05. Inflated effect sizes are certainly a sign that something could be wrong, but it's just one indicator. Exactly! The implication being the above. [0] https://en.wikipedia.org/wiki/Why_Most_Published_Research_Findings_Are_False https://en.wikipedia.org/wiki/Why_Most_Published_Research_Fi...