14 ms·

Why isn't preprint review being adopted?

69 points by dbingham 3y ago

timr 3y ago
Even if academics could review all the papers on a preprint server (which the article argues -- rightly -- that they can't), it wouldn't solve the perceived problems (or the actual problems) with scientific peer review. The vast majority of irreproducible papers aren't detectible as irreproducible at time of publication. They look fine, and many actually are fine. They just don't reproduce. That's an expected outcome in science. The system will self-correct over time. IMO, the main actual problem with peer review is that non-practitioners put too much faith in it. Nobody in science actually takes a paper on faith because it's been published, and you shouldn't either. Peer review is little more than a lightweight safeguard against complete nonsense being published. It barely works for that. Just because you found a paper doesn't mean you should believe it. You have to understand it. A secondary actual problem is that it's impossible to reproduce a lot of papers, or they're methodologically broken from the start (e.g. RCTs that are not pre-registered, or observational studies without control groups). These are problems we could actually solve. For example, just requiring that any paper publish the raw study data would help to self-control the system. There are high-profile researchers out there, right now, who do little more than statistically mine the same secret data set -- these people are likely publishing crap, but we have no way to prove it, because the data is secret.
- s0rce 3y ago
  There is no place to make a note that something doesn't reproduce so its extremely time consuming or you need some source of tribal knowledge. In my postdoc I was trying to make some porous films and a bunch of paper's methods didn't seem to work, maybe I did it wrong, maybe some detail wasn't described, who knows. I couldn't get it to work and there was no way to document that failure.
  - bumby 3y ago
    I wonder if there's a way to document such replaceability failures as an erratum to the original manuscript. I feel like this would help in at least two major ways: 1) It provides a reproducibility filter. If a method isn't shown to be reproducible, publically documenting that adds to the body of knowledge, and this would help drive an incentive towards reproducing work rather than just searching for novelty. It would document work that would otherwise be lost because there's no incentive to showcase it. When the lack of reproducible results isn't public, it's now more likely that others may waste considerable effort in the same vein. 3) It may enlist the original authors to help understand why the work didn't reproduce well. Maybe the secondary effort lacked some crucial step or understanding. The people best positioned to remedy this are the original authors, and this secondary publication incentivizes them to dialogue with those who couldn't reproduce the outcome. It doesn't mean they have to engage, but it at least gives them some reason to involve themselves in the process.
    - warkdarrior 3y ago
      2) ???
      - bumby 3y ago
        If you are willing to provide a more substantive response, I might be able to clarify.
        s0rce 3y ago
        I think they meant that you only list 1) and 3) and 2) was missing.
- Retric 3y ago
  Peer review is a spam filter, and it’s quite useful in that content. But it’s a spam filer for people who filter out 99.9+% of papers by having such a narrow scope they probably recognize several names on a given paper.
- pacbard 3y ago
  To your second point, I alway go back to this quote: "You can't fix by analysis what you bungled by design" (Light, Singer and Willett, 1990). If a paper is broken by design, there isn't much to do after the fact. It's just broken. The problem is that doing a good RCT takes both time and effort, with the huge risk of having null results, which usually results in a desk rejection from most top journals. So, you either are a top-fund raising researcher who can both fund multiple RCTs and people to support them, or you just try your best with what you have and hope to squeeze a paper out from you did. Releasing the data won't really help much if the data generating process is flawed. Sure, other people will be able to run different kind of analyses (e.g., jackknife your standard errors instead of just using a robust correction), but I'm not sure how helpful that will be. A third issue that I have also encountered is that journal editors have an agenda when putting together an issue, which sometimes overwrites the "quality' of the research with "fit" to the issue. This could lead to "lower quality" articles to be published because they fit the (often unspoken) direction of the journal. Most editors see their role as steering the field towards new directions (a sort of a meta service to the field) and sometimes that comes at the expense of the quality of the work.
  - YeGoblynQueenne 3y ago
    >> (Light, Singer and Willett, 1990) A citation like the one above should normally point to a full reference in a bibliography section. Did you forget the \bibliography{} command at the end of your comment?
    - anticensor 3y ago
      Hacker News does not have a LaTeX compiler.
  - timr 3y ago
    > Releasing the data won't really help much if the data generating process is flawed. Sure, other people will be able to run different kind of analyses (e.g., jackknife your standard errors instead of just using a robust correction), but I'm not sure how helpful that will be. It allows motivated people to catch more subtle forms of nonsense. Data colada, for example, has caught outright fraud, but only through herculean efforts. Imagine what groups like this might do if they had the raw data.
- barfbagginus 3y ago
  In AI work - which naturally lends itself to replicability improvements - we could get truly solid replicability by ratcheting up the standards for code quality, testing, and automation in AI projects. And I think llms can start doing a lot of that kind of QA and engineering / pre-operationalization work, because the best llms are already better at software and value engineering than the average postdoc AI researcher. Most AI codes are missing key replicability factors - either the training data/trainer are missing, the code has a poor testing / benchmark automation strategy, the code documentation is meager, or there's no real CI/CD practice for advancing the project or operationalizing it against problems caused by the anthropocentric collapse. Some researchers are even hardened against such things, seeing them as false worship of harmful business metrics, rather than a fundamental duty that could really improve the impact of their research, and it's applicability towards a universal crisis that faces us all. But we can put the lie to this view with just one glance at their code. Too much additional work is necessary to turn it into anything useful, either for further research iterations or productive operationalization. The gaps in code quality exist not because that form of code is optimal for research aims, but because researchers lack software engineering expertise, and cannot afford software engineering labor. But thankfully the level of software engineering labor is not even that great - llms can now help swing that effort. As a result I believe that we should work to create standards for AI assisted research repos that correct the major deficits of replicability, usability, and code quality that we see in most AI repos. Then we should campaign to adopt those standards into peer review. Let one of the reviewers be an AI that really grills your code on its quality. And actually incorporate the PRs that it proposes. I think that would change the situation, from the current standard where academic AI repos are mainly nonreplicating throw-away code, to an opposite situation where the majority of AI research repos are easy to replicate, improve, and mobilize against the social and environmental problems facing humanity, as it navigates through the anthropocene disaster.
- ramblenode 3y ago
  > The vast majority of irreproducible papers aren't detectible as irreproducible at time of publication. They look fine, and many actually are fine. They just don't reproduce. That's an expected outcome in science. This is not entirely true. A power analysis is how you determine reproducibility, and researchers should be doing it before they begin collecting data. Reviewers can do it post-hoc with assumptions about the expected effect size (which might come from similar studies). False positives produce inflated effect sizes, so if a result is marginally significant but shows a large effect, that is a good heuristic the result will not reproduce.
  - timr 3y ago
    > A power analysis is how you determine reproducibility All a power analysis does is reduce the chance that the result is a false negative. It doesn't reduce the chance of a false positive. > False positives produce inflated effect sizes Not always. Lots of studies publish as "significant" as soon as they get a p-value just under .05. Inflated effect sizes are certainly a sign that something could be wrong, but it's just one indicator. Regardless, even if you have a power analysis at the conventional threshold of 80%, and a p-value of .05, you're still going to get spurious positive results 5% of the time, and spurious negative results 20% of the time, by definition.
    - ramblenode 3y ago
      > All a power analysis does is reduce the chance that the result is a false negative. It doesn't reduce the chance of a false positive. This is true when we are dealing with an uninformative prior, but published research is known to be biased toward positive results and uncorrected multiple comparisons. This situation leads to small sample studies with high random variance being paradoxically correlated with significant results. High random variance appears as a false large effect size in the published result, so if the power is low when calculated with a smaller (adjusted) effect, there is reason to believe that the p-value is inflated. See e.g. Andrew Gelman's work on small sample studies, garden of forking paths or [0]. > Not always. Lots of studies publish as "significant" as soon as they get a p-value just under .05. Inflated effect sizes are certainly a sign that something could be wrong, but it's just one indicator. Exactly! The implication being the above. [0] https://en.wikipedia.org/wiki/Why_Most_Published_Research_Findings_Are_False https://en.wikipedia.org/wiki/Why_Most_Published_Research_Fi...
llm_trw 3y ago
Because I don't want to spend two years fighting the second referee when she's fundamentally misunderstood the point of my paper. I'm no longer in academia. Either take what I put up on arxiv or leave it. I _really_ don't care.
- s0rce 3y ago
  While not always true my metric for clear/understandable is for other people to understand it. This usually supports my argument when people show me a document and I have no idea what its saying, they argue its perfectly clear... my definition was for people other than the author to grasp the intended meaning.
  - Muller20 3y ago
    Peer review goes beyond simple issues about clarity or misunderstanding. In particular, peer review is sometimes seen as an adversarial process. Often, the reviewer will not understand because he is not the intended audience. Other times, he will understand but he just doesn't like your method, because he is working in an opposite direction. Or maybe your method is a direct competitor of his and yours work better, which incentivizes some people to block your work.
    - wrycoder 3y ago
      Or your paper goes against the current paradigm or is otherwise politically unpalatable.
      - deleted 3y ago
        [deleted]
      - deleted 3y ago
        [deleted]
    - anticensor 3y ago
      > Other times, he will understand but he just doesn't like your method, because he is working in an opposite direction. Or maybe your method is a direct competitor of his and yours work better, which incentivizes some people to block your work. Oh you mean those phantom "off topic"/"out of scope" reviews.
daft_pink 3y ago
I would worry that preprint review would turn into another front of the culture wars for certain fields and science by consensus.
zer00eyz 3y ago
The review process is broken. Reviewing pre print papers isnt any more effective than reviewing printed papers. Review, and publication is a meaningless bar. Publish -> people find insight and try to pick it apart -> You either have flaws or you get reproduced... Only then should your paper be of any worth to be quoted or sighted from. The current system is glad-handing, intellectual protectionism and mastrubation. Academia has only itself to blame for this, and they are apparently unwilling to fix it.
- bumby 3y ago
  I think we need to find ways of giving status to reproducing studies. Maybe not as much as novelty, but definitely something greater than what it is currently.
  - glial 3y ago
    IMO reproducing findings could/should be a mandatory part of PhD training.
- ajmurmann 3y ago
  "Publish -> people find insight and try to pick it apart -> You either have flaws or you get reproduced... Only then should your paper be of any worth to be quoted or sighted from." This is already how it's supposed to work. The review before publication is a fairly superficial check that just confirms that what you describe follows basic scientific practices. There is no validation of the actual research. A proper reproduction is what's supposed to come after publication. IMO the real problems are that a) there isn't much glamour and funding for reproducing other's studies and b) "science journalists", university PR departments and now in part people on social media are picking up research before people on the field looked at it or misrepresent it. Suddenly the audience is a lot of folks who never were the intended audience of the process.
  - zer00eyz 3y ago
    There is a pretty simple way to change all of that. Academic standards: You are not longer allowed to site a non reproduced paper in yours. Citations matter as much as the print, put the hurdle there and all of a sudden things will change real quick.
rhelz 3y ago
One of my profs once remarked, "All of science is done on a volunteer basis." He was talking about peer review, which--as crucial as it is--is not something you get paid for. Writing a review--a good review--is 1) hard work, 2) can only be done by somebody who has spent years in postgraduate study, and 3) takes up a lot of time, which has many other demands on it. The solution? Its obvious. In a free market, how do you signal if you want more of something to be produced? Bueller? Bueller? Yeah, that's right, you gotta pay for it. This cost should just be estimated and factored into the original grant proposals--if its not worth $5k or $10k to fund a round of peer review, and perhaps also funds to run confirming experiments--well, then its probably not research worth doing in the first place. So yeah, write up the grants to include the actual full cost of doing and publishing the research. It would be a great way for starving grad students to earn some coin, and the experience gained in running confirming experiments would be invaluable to help them get that R.A. position or postdoc.
- bumby 3y ago
  I don't disagree with the proposed idea of paying for review, but I would prefer also to have guardrails to ensure a good review. I would be willing to pay for a good review because it makes the paper/investigation better. But let's face it: under the current paradigm, there are also a lot of really bad reviews. It's one thing when it's apparent that a reviewer doesn't understand something because of a lack of clarity in the writing. But it's also extremely frustrating when it's obvious the reviewer hasn't even bothered to carefully read the manuscript. Under a payment paradigm, we need mechanisms to limit the incentive to maximize throughput as a means of getting the most pay and instead maximize the review quality. I assume there'd be good ways to do that, but I don't know what those would be.
  - heikkilevanto 3y ago
    So, we just need a meta-review to review the reviews. At a cost, of course. And in order to keep that honest, we need a meta-meta-review...
    - notatoad 3y ago
frozenport 3y ago
They should just open a comment field on arXiv. Then I can anonymously critique the paper without fear of the authors rejecting my career making Nature paper.
- patel011393 3y ago
  I know someone working on a plugin for that currently.
_delirium 3y ago
Does openreview.net not count as preprint review in the sense the author means? It has substantial uptake in computer science.
- dbingham 3y ago
  Nice catch! I was going from the data shared in that paper[1] and didn't notice that it excluded OpenReview.net (which I'm aware of). The paper got their data[2, 3] from Sciety and it looks like OpenReview isn't included in Sciety's data. It may have been excluded because OpenReview (as I understand it) seems to be primarily used to provide open review of conference proceedings, which I suspect the article puts in a different category than generally shared preprints. But it would be worth analyzing OpenReview's uptake separately and thinking about what it's doing differently! [1] https://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.3002502 https://journals.plos.org/plosbiology/article?id=10.1371/jou... [2]https://zenodo.org/records/10070536 https://zenodo.org/records/10070536 [3] https://lookerstudio.google.com/u/0/reporting/b09cf3e8-88c7-4928-b639-2c48dc786191/page/p_kr9n81etoc?s=g4fLuNRACRM https://lookerstudio.google.com/u/0/reporting/b09cf3e8-88c7-...
  - _delirium 3y ago
    I do agree it's a bit different. How close maybe depends on what motivates you to be interested in the preprint review model in the first place? Could imagine this varies by person. In a certain sense, the entire field of comp sci has become reorganized around preprint review. The 100% normal workflow now is that you first upload your paper to arXiv, circulate it informally, then whenever you want a formal review, submit to whatever conference or journal you want. The conferences and journals have basically become stamp-of-approval providers rather than really "publishers". If they accept it, you edit the arXiv entry to upload a v2 camera-ready PDF and put the venue's acceptance stamp-of-approval in the comments field. A few reasons this might not fit the vision of preprint review, all with different solutions: 1. The reviews might not be public. 2. If accepted, it sometimes costs $$ (e.g. NeurIPS has a $800 registration fee, and some OA journals charge APCs). 3. Many of the prestigious review providers mix together two different types of review: review for technical quality and errors, versus review for perceived importance and impact. Some also have quite low acceptance rates (due to either prestige reasons or literal capacity constraints). TMLR [1] might be the closest to addressing all three points, and has some similarity to eLife, except that unlike eLife it doesn't charge authors. It's essentially an overlay journal on openreview.net preprints (covers #1), is platinum OA (covers #2), and explicitly excludes "subjective significance" as a review criterion (covers #3). [1] https://jmlr.org/tmlr/ https://jmlr.org/tmlr/
adastra22 3y ago
PREPUBLICATION REVIEW IS BAD! STOP TRYING TO REINTRODUCE IT. Sorry for the all caps. Publishing papers without “peer review” isn’t some radical new concept—it’s how all scientific fields operated prior to ca. 1970. That’s about when the pace of article writing outstripped available pages in journals and this system of pre-publication review was adopted and formalized. For the first 300 years of science you published papers by sending it off as a letter to the editor (sometimes via a sponsor if you were new to the journal), and they either accepted or rejected it as-is. The idea of having your intellectual competitors review your work and potentially sabotage your publication prospects as a standard process is a relatively recent addition. And one that has not been shown to actually be effective. The rise of Arxiv is a recognition by researchers that we don’t need or want that system, and we should do away with it entirely in this era of digital print where page counts don’t matter. So please stop trying to force it back on us!
- freedomben 3y ago
  > The idea of having your intellectual competitors review your work potentially sabotage your publication prospects as a standard process is a relatively recent addition. And one that has not been shown to actually be effective. If this is true (and I'm not doubting you, just acknowledging that I'm taking your word for it) then why abandon the entire system? Why not just roll it back to the state before we added the intellectual competitor review?
  - frozenport 3y ago
    >> Why not just roll it back to the state before we added the intellectual competitor review? Journals don't add much value outside of their peer review. Most researcher don't care about the paper copies, or pagination, or document formatting services provided by publishers. Their old paper based distribution channels are simply not used.
  - adastra22 3y ago
    The prior state was a situation of no pre-publication review other than the editorial staff of the journal. We should go back to that, yes. By disbanding entirely the “peer review” system that currently exists.
ccppurcell 3y ago
Peer review is pretty unpopular round these parts. In mathematics/TCS I've had mostly good experiences. Actually most of the time the review process improved my papers. Clearly something is rotten about the way peer review is implemented in the empirical sciences. But think of all those high profile retractions you read about these days. Usually that comes about by a sort of post hoc peer review, not by anything resembling market forces.
- YeGoblynQueenne 3y ago
  Not to be mean to the HN community but at least a substantial minority of people who complain about peer review on here have no experience of peer review, even in applied CS and AI and machine learning, which are the hot topics today. But they've read that peer review is broken and, by god, they'll let the world know! For science! I publish in machine learning and my experience is the same as yours: reviews have mainly helped me to improve my papers. Though to be fair this is mainly the case in journals; in conferences it's true that reviewers will often look for reasons to reject and don't try to be constructive (I always do; I still find it very hard to reject). This is the result of the field of AI research having experienced a huge explosion of interest, and therefore submissions, in the last few years, so that all the conferences are creaking under the strain. Most of the new entrants are also junior researchers without too much experience- and that is true for both authors and reviewers (who are only invited to review after they publish in a venue). So the conferences are a bit of a mess at the moment, and the quality of the papers that get submitted and published, overall low. But that's not because "peer review is broken", it's because too many people start a PhD in machine learning thinking they'll get immediately hired by Google, or OpenAI I guess. That too shall pass, and then things will calm down.
  - ccppurcell 3y ago
    Agreed wholeheartedly but didn't want to come out swinging! Not only that but my experience of reviewing has also been positive, and has given me ideas for research and how to present research (notation, paper structure etc) Except my first review which was a 100 page survey paper on a very specific kind of inequality that exists for practically any graph invariant, so every page was pretty much identical just with alpha then beta then omega then chi... And the deadline was my birthday!
milancurcic 3y ago
"True peer review begins after publication." --Eric Weinstein
- patel011393 3y ago
  Do you have a URL/citation for that? I'd like to use this quote in a paper.
cachemoi 3y ago
The solution to the dated model exists, it's git/github. I'm trying to build a "git journal" (essentially a github org where projects/research gets published paired with a substack newsletter), details here [0] Let me know if you have a project you'd like to get on there! Here's what it looks like, a paper on directed evolution [1] [0] https://versioned.science/ https://versioned.science/ [1] https://github.com/versioned-science/DNA_polymerase_directed_evolution https://github.com/versioned-science/DNA_polymerase_directed...
dianekenneth 3y ago
[dead]
1letterunixname 3y ago
Up until the 1940's, "publish or perish" wasn't the obsession, publishing what was thoroughly vetted but not before it was ready. The sheer volume of substandard, barely novel papers allowed and the artificial expectations to produce a blizzard of publications foisted on researchers are the central problems.
somethingsome 3y ago
A fun system would be: you 'have to' peer review publicly a (part of a) paper (at least) when you cite it in your paper. So that often cited paper get a lot of different (small) public reviews that can be curated from time to time, obscure papers get at least one review justifying why it's relevant to cite them in the new work. Some could argue that this is too much work added to the writing process.. But.. At the same time.. Shouldn't we read the papers we cite? Why not automatically write a small review of it? It has not to be huge, only the justification on why we (can) use it in our work.
- roflmaostc 3y ago
  Yeah, such a system would be great. A bit like Google Scholar. Papers are indexed and you can access the references easily. And you would be able to comment and review certain lines. Everyone could add notes that certain equations are wrong, etc. In best case authors would engage in the discussion too. But obviously this won't work because some papers are behind paywalls :/
erictleung 3y ago
Here's a recent effort at peer reviewing pre-prints that started in 2017 https://prereview.org/ https://prereview.org/
alecky 3y ago
[dead]
ElijahMadelyn 3y ago
[dead]
jacksonbeatrice 3y ago
[dead]
lizzystrick685 3y ago
[dead]
cyari6908 3y ago
[dead]

69 points by dbingham 3y ago

timr 3y ago

Even if academics could review all the papers on a preprint server (which the article argues -- rightly -- that they can't), it wouldn't solve the perceived problems (or the actual problems) with scientific peer review. The vast majority of irreproducible papers aren't detectible as irreproducible at time of publication. They look fine, and many actually are fine. They just don't reproduce. That's an expected outcome in science. The system will self-correct over time. IMO, the main actual problem with peer review is that non-practitioners put too much faith in it. Nobody in science actually takes a paper on faith because it's been published, and you shouldn't either. Peer review is little more than a lightweight safeguard against complete nonsense being published. It barely works for that. Just because you found a paper doesn't mean you should believe it. You have to understand it. A secondary actual problem is that it's impossible to reproduce a lot of papers, or they're methodologically broken from the start (e.g. RCTs that are not pre-registered, or observational studies without control groups). These are problems we could actually solve. For example, just requiring that any paper publish the raw study data would help to self-control the system. There are high-profile researchers out there, right now, who do little more than statistically mine the same secret data set -- these people are likely publishing crap, but we have no way to prove it, because the data is secret.

s0rce 3y ago
There is no place to make a note that something doesn't reproduce so its extremely time consuming or you need some source of tribal knowledge. In my postdoc I was trying to make some porous films and a bunch of paper's methods didn't seem to work, maybe I did it wrong, maybe some detail wasn't described, who knows. I couldn't get it to work and there was no way to document that failure.
- bumby 3y ago
  I wonder if there's a way to document such replaceability failures as an erratum to the original manuscript. I feel like this would help in at least two major ways: 1) It provides a reproducibility filter. If a method isn't shown to be reproducible, publically documenting that adds to the body of knowledge, and this would help drive an incentive towards reproducing work rather than just searching for novelty. It would document work that would otherwise be lost because there's no incentive to showcase it. When the lack of reproducible results isn't public, it's now more likely that others may waste considerable effort in the same vein. 3) It may enlist the original authors to help understand why the work didn't reproduce well. Maybe the secondary effort lacked some crucial step or understanding. The people best positioned to remedy this are the original authors, and this secondary publication incentivizes them to dialogue with those who couldn't reproduce the outcome. It doesn't mean they have to engage, but it at least gives them some reason to involve themselves in the process.
  - warkdarrior 3y ago
    2) ???
    - bumby 3y ago
      If you are willing to provide a more substantive response, I might be able to clarify.
      - s0rce 3y ago
        I think they meant that you only list 1) and 3) and 2) was missing.
Retric 3y ago
Peer review is a spam filter, and it’s quite useful in that content. But it’s a spam filer for people who filter out 99.9+% of papers by having such a narrow scope they probably recognize several names on a given paper.
pacbard 3y ago
To your second point, I alway go back to this quote: "You can't fix by analysis what you bungled by design" (Light, Singer and Willett, 1990). If a paper is broken by design, there isn't much to do after the fact. It's just broken. The problem is that doing a good RCT takes both time and effort, with the huge risk of having null results, which usually results in a desk rejection from most top journals. So, you either are a top-fund raising researcher who can both fund multiple RCTs and people to support them, or you just try your best with what you have and hope to squeeze a paper out from you did. Releasing the data won't really help much if the data generating process is flawed. Sure, other people will be able to run different kind of analyses (e.g., jackknife your standard errors instead of just using a robust correction), but I'm not sure how helpful that will be. A third issue that I have also encountered is that journal editors have an agenda when putting together an issue, which sometimes overwrites the "quality' of the research with "fit" to the issue. This could lead to "lower quality" articles to be published because they fit the (often unspoken) direction of the journal. Most editors see their role as steering the field towards new directions (a sort of a meta service to the field) and sometimes that comes at the expense of the quality of the work.
- YeGoblynQueenne 3y ago
  >> (Light, Singer and Willett, 1990) A citation like the one above should normally point to a full reference in a bibliography section. Did you forget the \bibliography{} command at the end of your comment?
  - anticensor 3y ago
    Hacker News does not have a LaTeX compiler.
- timr 3y ago
  > Releasing the data won't really help much if the data generating process is flawed. Sure, other people will be able to run different kind of analyses (e.g., jackknife your standard errors instead of just using a robust correction), but I'm not sure how helpful that will be. It allows motivated people to catch more subtle forms of nonsense. Data colada, for example, has caught outright fraud, but only through herculean efforts. Imagine what groups like this might do if they had the raw data.
barfbagginus 3y ago
In AI work - which naturally lends itself to replicability improvements - we could get truly solid replicability by ratcheting up the standards for code quality, testing, and automation in AI projects. And I think llms can start doing a lot of that kind of QA and engineering / pre-operationalization work, because the best llms are already better at software and value engineering than the average postdoc AI researcher. Most AI codes are missing key replicability factors - either the training data/trainer are missing, the code has a poor testing / benchmark automation strategy, the code documentation is meager, or there's no real CI/CD practice for advancing the project or operationalizing it against problems caused by the anthropocentric collapse. Some researchers are even hardened against such things, seeing them as false worship of harmful business metrics, rather than a fundamental duty that could really improve the impact of their research, and it's applicability towards a universal crisis that faces us all. But we can put the lie to this view with just one glance at their code. Too much additional work is necessary to turn it into anything useful, either for further research iterations or productive operationalization. The gaps in code quality exist not because that form of code is optimal for research aims, but because researchers lack software engineering expertise, and cannot afford software engineering labor. But thankfully the level of software engineering labor is not even that great - llms can now help swing that effort. As a result I believe that we should work to create standards for AI assisted research repos that correct the major deficits of replicability, usability, and code quality that we see in most AI repos. Then we should campaign to adopt those standards into peer review. Let one of the reviewers be an AI that really grills your code on its quality. And actually incorporate the PRs that it proposes. I think that would change the situation, from the current standard where academic AI repos are mainly nonreplicating throw-away code, to an opposite situation where the majority of AI research repos are easy to replicate, improve, and mobilize against the social and environmental problems facing humanity, as it navigates through the anthropocene disaster.
ramblenode 3y ago
> The vast majority of irreproducible papers aren't detectible as irreproducible at time of publication. They look fine, and many actually are fine. They just don't reproduce. That's an expected outcome in science. This is not entirely true. A power analysis is how you determine reproducibility, and researchers should be doing it before they begin collecting data. Reviewers can do it post-hoc with assumptions about the expected effect size (which might come from similar studies). False positives produce inflated effect sizes, so if a result is marginally significant but shows a large effect, that is a good heuristic the result will not reproduce.
- timr 3y ago
  > A power analysis is how you determine reproducibility All a power analysis does is reduce the chance that the result is a false negative. It doesn't reduce the chance of a false positive. > False positives produce inflated effect sizes Not always. Lots of studies publish as "significant" as soon as they get a p-value just under .05. Inflated effect sizes are certainly a sign that something could be wrong, but it's just one indicator. Regardless, even if you have a power analysis at the conventional threshold of 80%, and a p-value of .05, you're still going to get spurious positive results 5% of the time, and spurious negative results 20% of the time, by definition.
  - ramblenode 3y ago
    > All a power analysis does is reduce the chance that the result is a false negative. It doesn't reduce the chance of a false positive. This is true when we are dealing with an uninformative prior, but published research is known to be biased toward positive results and uncorrected multiple comparisons. This situation leads to small sample studies with high random variance being paradoxically correlated with significant results. High random variance appears as a false large effect size in the published result, so if the power is low when calculated with a smaller (adjusted) effect, there is reason to believe that the p-value is inflated. See e.g. Andrew Gelman's work on small sample studies, garden of forking paths or [0]. > Not always. Lots of studies publish as "significant" as soon as they get a p-value just under .05. Inflated effect sizes are certainly a sign that something could be wrong, but it's just one indicator. Exactly! The implication being the above. [0] https://en.wikipedia.org/wiki/Why_Most_Published_Research_Findings_Are_False https://en.wikipedia.org/wiki/Why_Most_Published_Research_Fi...

llm_trw 3y ago

Because I don't want to spend two years fighting the second referee when she's fundamentally misunderstood the point of my paper. I'm no longer in academia. Either take what I put up on arxiv or leave it. I _really_ don't care.

s0rce 3y ago
While not always true my metric for clear/understandable is for other people to understand it. This usually supports my argument when people show me a document and I have no idea what its saying, they argue its perfectly clear... my definition was for people other than the author to grasp the intended meaning.
- Muller20 3y ago
  Peer review goes beyond simple issues about clarity or misunderstanding. In particular, peer review is sometimes seen as an adversarial process. Often, the reviewer will not understand because he is not the intended audience. Other times, he will understand but he just doesn't like your method, because he is working in an opposite direction. Or maybe your method is a direct competitor of his and yours work better, which incentivizes some people to block your work.
  - wrycoder 3y ago
    Or your paper goes against the current paradigm or is otherwise politically unpalatable.
    - deleted 3y ago
      [deleted]
    - deleted 3y ago
      [deleted]
  - anticensor 3y ago
    > Other times, he will understand but he just doesn't like your method, because he is working in an opposite direction. Or maybe your method is a direct competitor of his and yours work better, which incentivizes some people to block your work. Oh you mean those phantom "off topic"/"out of scope" reviews.

daft_pink 3y ago

I would worry that preprint review would turn into another front of the culture wars for certain fields and science by consensus.

zer00eyz 3y ago

The review process is broken. Reviewing pre print papers isnt any more effective than reviewing printed papers. Review, and publication is a meaningless bar. Publish -> people find insight and try to pick it apart -> You either have flaws or you get reproduced... Only then should your paper be of any worth to be quoted or sighted from. The current system is glad-handing, intellectual protectionism and mastrubation. Academia has only itself to blame for this, and they are apparently unwilling to fix it.

bumby 3y ago
I think we need to find ways of giving status to reproducing studies. Maybe not as much as novelty, but definitely something greater than what it is currently.
- glial 3y ago
  IMO reproducing findings could/should be a mandatory part of PhD training.
ajmurmann 3y ago
"Publish -> people find insight and try to pick it apart -> You either have flaws or you get reproduced... Only then should your paper be of any worth to be quoted or sighted from." This is already how it's supposed to work. The review before publication is a fairly superficial check that just confirms that what you describe follows basic scientific practices. There is no validation of the actual research. A proper reproduction is what's supposed to come after publication. IMO the real problems are that a) there isn't much glamour and funding for reproducing other's studies and b) "science journalists", university PR departments and now in part people on social media are picking up research before people on the field looked at it or misrepresent it. Suddenly the audience is a lot of folks who never were the intended audience of the process.
- zer00eyz 3y ago
  There is a pretty simple way to change all of that. Academic standards: You are not longer allowed to site a non reproduced paper in yours. Citations matter as much as the print, put the hurdle there and all of a sudden things will change real quick.

rhelz 3y ago

One of my profs once remarked, "All of science is done on a volunteer basis." He was talking about peer review, which--as crucial as it is--is not something you get paid for. Writing a review--a good review--is 1) hard work, 2) can only be done by somebody who has spent years in postgraduate study, and 3) takes up a lot of time, which has many other demands on it. The solution? Its obvious. In a free market, how do you signal if you want more of something to be produced? Bueller? Bueller? Yeah, that's right, you gotta pay for it. This cost should just be estimated and factored into the original grant proposals--if its not worth $5k or $10k to fund a round of peer review, and perhaps also funds to run confirming experiments--well, then its probably not research worth doing in the first place. So yeah, write up the grants to include the actual full cost of doing and publishing the research. It would be a great way for starving grad students to earn some coin, and the experience gained in running confirming experiments would be invaluable to help them get that R.A. position or postdoc.

bumby 3y ago
I don't disagree with the proposed idea of paying for review, but I would prefer also to have guardrails to ensure a good review. I would be willing to pay for a good review because it makes the paper/investigation better. But let's face it: under the current paradigm, there are also a lot of really bad reviews. It's one thing when it's apparent that a reviewer doesn't understand something because of a lack of clarity in the writing. But it's also extremely frustrating when it's obvious the reviewer hasn't even bothered to carefully read the manuscript. Under a payment paradigm, we need mechanisms to limit the incentive to maximize throughput as a means of getting the most pay and instead maximize the review quality. I assume there'd be good ways to do that, but I don't know what those would be.
- heikkilevanto 3y ago
  So, we just need a meta-review to review the reviews. At a cost, of course. And in order to keep that honest, we need a meta-meta-review...
  - notatoad 3y ago

frozenport 3y ago

They should just open a comment field on arXiv. Then I can anonymously critique the paper without fear of the authors rejecting my career making Nature paper.

patel011393 3y ago
I know someone working on a plugin for that currently.

_delirium 3y ago

Does openreview.net not count as preprint review in the sense the author means? It has substantial uptake in computer science.

dbingham 3y ago
Nice catch! I was going from the data shared in that paper[1] and didn't notice that it excluded OpenReview.net (which I'm aware of). The paper got their data[2, 3] from Sciety and it looks like OpenReview isn't included in Sciety's data. It may have been excluded because OpenReview (as I understand it) seems to be primarily used to provide open review of conference proceedings, which I suspect the article puts in a different category than generally shared preprints. But it would be worth analyzing OpenReview's uptake separately and thinking about what it's doing differently! [1] https://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.3002502 https://journals.plos.org/plosbiology/article?id=10.1371/jou... [2]https://zenodo.org/records/10070536 https://zenodo.org/records/10070536 [3] https://lookerstudio.google.com/u/0/reporting/b09cf3e8-88c7-4928-b639-2c48dc786191/page/p_kr9n81etoc?s=g4fLuNRACRM https://lookerstudio.google.com/u/0/reporting/b09cf3e8-88c7-...
- _delirium 3y ago
  I do agree it's a bit different. How close maybe depends on what motivates you to be interested in the preprint review model in the first place? Could imagine this varies by person. In a certain sense, the entire field of comp sci has become reorganized around preprint review. The 100% normal workflow now is that you first upload your paper to arXiv, circulate it informally, then whenever you want a formal review, submit to whatever conference or journal you want. The conferences and journals have basically become stamp-of-approval providers rather than really "publishers". If they accept it, you edit the arXiv entry to upload a v2 camera-ready PDF and put the venue's acceptance stamp-of-approval in the comments field. A few reasons this might not fit the vision of preprint review, all with different solutions: 1. The reviews might not be public. 2. If accepted, it sometimes costs $$ (e.g. NeurIPS has a $800 registration fee, and some OA journals charge APCs). 3. Many of the prestigious review providers mix together two different types of review: review for technical quality and errors, versus review for perceived importance and impact. Some also have quite low acceptance rates (due to either prestige reasons or literal capacity constraints). TMLR [1] might be the closest to addressing all three points, and has some similarity to eLife, except that unlike eLife it doesn't charge authors. It's essentially an overlay journal on openreview.net preprints (covers #1), is platinum OA (covers #2), and explicitly excludes "subjective significance" as a review criterion (covers #3). [1] https://jmlr.org/tmlr/ https://jmlr.org/tmlr/

adastra22 3y ago

PREPUBLICATION REVIEW IS BAD! STOP TRYING TO REINTRODUCE IT. Sorry for the all caps. Publishing papers without “peer review” isn’t some radical new concept—it’s how all scientific fields operated prior to ca. 1970. That’s about when the pace of article writing outstripped available pages in journals and this system of pre-publication review was adopted and formalized. For the first 300 years of science you published papers by sending it off as a letter to the editor (sometimes via a sponsor if you were new to the journal), and they either accepted or rejected it as-is. The idea of having your intellectual competitors review your work and potentially sabotage your publication prospects as a standard process is a relatively recent addition. And one that has not been shown to actually be effective. The rise of Arxiv is a recognition by researchers that we don’t need or want that system, and we should do away with it entirely in this era of digital print where page counts don’t matter. So please stop trying to force it back on us!

freedomben 3y ago
> The idea of having your intellectual competitors review your work potentially sabotage your publication prospects as a standard process is a relatively recent addition. And one that has not been shown to actually be effective. If this is true (and I'm not doubting you, just acknowledging that I'm taking your word for it) then why abandon the entire system? Why not just roll it back to the state before we added the intellectual competitor review?
- frozenport 3y ago
  >> Why not just roll it back to the state before we added the intellectual competitor review? Journals don't add much value outside of their peer review. Most researcher don't care about the paper copies, or pagination, or document formatting services provided by publishers. Their old paper based distribution channels are simply not used.
- adastra22 3y ago
  The prior state was a situation of no pre-publication review other than the editorial staff of the journal. We should go back to that, yes. By disbanding entirely the “peer review” system that currently exists.

ccppurcell 3y ago

Peer review is pretty unpopular round these parts. In mathematics/TCS I've had mostly good experiences. Actually most of the time the review process improved my papers. Clearly something is rotten about the way peer review is implemented in the empirical sciences. But think of all those high profile retractions you read about these days. Usually that comes about by a sort of post hoc peer review, not by anything resembling market forces.

YeGoblynQueenne 3y ago
Not to be mean to the HN community but at least a substantial minority of people who complain about peer review on here have no experience of peer review, even in applied CS and AI and machine learning, which are the hot topics today. But they've read that peer review is broken and, by god, they'll let the world know! For science! I publish in machine learning and my experience is the same as yours: reviews have mainly helped me to improve my papers. Though to be fair this is mainly the case in journals; in conferences it's true that reviewers will often look for reasons to reject and don't try to be constructive (I always do; I still find it very hard to reject). This is the result of the field of AI research having experienced a huge explosion of interest, and therefore submissions, in the last few years, so that all the conferences are creaking under the strain. Most of the new entrants are also junior researchers without too much experience- and that is true for both authors and reviewers (who are only invited to review after they publish in a venue). So the conferences are a bit of a mess at the moment, and the quality of the papers that get submitted and published, overall low. But that's not because "peer review is broken", it's because too many people start a PhD in machine learning thinking they'll get immediately hired by Google, or OpenAI I guess. That too shall pass, and then things will calm down.
- ccppurcell 3y ago
  Agreed wholeheartedly but didn't want to come out swinging! Not only that but my experience of reviewing has also been positive, and has given me ideas for research and how to present research (notation, paper structure etc) Except my first review which was a 100 page survey paper on a very specific kind of inequality that exists for practically any graph invariant, so every page was pretty much identical just with alpha then beta then omega then chi... And the deadline was my birthday!

milancurcic 3y ago

"True peer review begins after publication." --Eric Weinstein

patel011393 3y ago
Do you have a URL/citation for that? I'd like to use this quote in a paper.

cachemoi 3y ago

The solution to the dated model exists, it's git/github. I'm trying to build a "git journal" (essentially a github org where projects/research gets published paired with a substack newsletter), details here [0] Let me know if you have a project you'd like to get on there! Here's what it looks like, a paper on directed evolution [1] [0] https://versioned.science/ https://versioned.science/ [1] https://github.com/versioned-science/DNA_polymerase_directed_evolution https://github.com/versioned-science/DNA_polymerase_directed...

dianekenneth 3y ago

[dead]

1letterunixname 3y ago

Up until the 1940's, "publish or perish" wasn't the obsession, publishing what was thoroughly vetted but not before it was ready. The sheer volume of substandard, barely novel papers allowed and the artificial expectations to produce a blizzard of publications foisted on researchers are the central problems.

somethingsome 3y ago

A fun system would be: you 'have to' peer review publicly a (part of a) paper (at least) when you cite it in your paper. So that often cited paper get a lot of different (small) public reviews that can be curated from time to time, obscure papers get at least one review justifying why it's relevant to cite them in the new work. Some could argue that this is too much work added to the writing process.. But.. At the same time.. Shouldn't we read the papers we cite? Why not automatically write a small review of it? It has not to be huge, only the justification on why we (can) use it in our work.

roflmaostc 3y ago
Yeah, such a system would be great. A bit like Google Scholar. Papers are indexed and you can access the references easily. And you would be able to comment and review certain lines. Everyone could add notes that certain equations are wrong, etc. In best case authors would engage in the discussion too. But obviously this won't work because some papers are behind paywalls :/

erictleung 3y ago

Here's a recent effort at peer reviewing pre-prints that started in 2017 https://prereview.org/ https://prereview.org/

alecky 3y ago

[dead]

ElijahMadelyn 3y ago

[dead]

jacksonbeatrice 3y ago

[dead]

lizzystrick685 3y ago

[dead]

cyari6908 3y ago

[dead]