6 ms·

> I don't see how this could ever work, and non-scientists seem to often dramatically underestimate the amount of work it would be to replicate every published

> I don't see how this could ever work, and non-scientists seem to often dramatically underestimate the amount of work it would be to replicate every published paper. They also tend to over-estimate the effect of peer review (often equating peer review with validity). > If someone cares enough about the work to build on it, they will replicate it anyway. And in that case they have a good incentive to spend the effort. If that works this will indirectly support the original paper even if the following papers don't specifically replicate the original results. Though this part is much more problematic if the following experiments fail, then this will likely remain entirely unpublished. It can also remain unpublished if other things did not work out, even if the results could be replicated. A half-fictional example: a team is working on a revolutionary new material to solve complicated engineering problems. They found a material that was synthesised by someone in the 1980s, published once and never reproduced, which they think could have the specific property they are after. So they synthesise it, and it turns out that the material exists, with the expected structure but not with the property they hoped. They aren’t going to write it up and publish it; they’re just going to scrap it and move on to the next candidate. Different teams might be doing the same thing at the same time, and nobody coming after them will have a clue.

techdragon 3y ago
This waste of effort by way of duplicating unpublished negative results is a big factor in why replicated results deserve to be rated more highly than results that have not been replicated regardless of the prestige of the researchers or the institutions involved… if no one can prove your work work was correct… how much can anyone trust your work… I have gone down the rabbit hole of engineering research before and 90% of the time I’ve managed to find an anecdote or subsequent research footnotes or actual subsequent research publications, that substantially invalidated the lofty claims of the engineers in the 70s or 80s (which is amazing still despite this, a genuine treasure trove of research unused and sometimes useful aerospace engineering research and development) and unfortunately outside the few proper publications, a lot of the invalidations are not properly reverse cited research material and I could have spent a week cross referencing before I spot the link and realise the unnamed work they are saying they are proving wrong is actually some footnotes containing the only published data (before their new paper) on some old work that has a bad scan copy on the NASA NTRS server under some obscure title and no related keywords to the topic the research is notionally about… Academic research can genuinely suck sometimes… particularly when you want to actually apply it.
- aeternum 3y ago
  Publishing positive/noteworth results only does seem like an embarrassingly obvious major flaw in academia and the greater scientific community. A research assistant would quickly be thrown out if he/she refused to record negative experimental results, yet we somehow decide that is fine when operating as a collective.
- godelski 3y ago
  Alternatively, maybe researchers should be encouraged to publish all kinds of results and not just "novel" work. Successes, failures, insights, etc. This idea of how we evaluate researchers is often silly. We're asking people to push the bounds of knowledge and we don't know what will come from it or how long it will take. But we are hyper focused on this short term evaluation.
vibrio 3y ago
“They also tend to over-estimate the effect of peer review (often equating peer review with validity).“ In my experience, scientists ate comfortably cynical about peer review- even those that serve as reviewers and editors- except maybe junior scientists that haven’t gotten burned yet.
- kergonath 3y ago
  Yes, because we know how the metaphorical sausage is made: with unpaid reviewers who have many other, more interesting things to do and often an axe to grind. That is, if they don’t delegate the review to one of their post-docs.
  - aftoprokrustes 3y ago
    Post doc? In what kind of utopian field did you work? In my former institute virtually all papers were written by PhD candidates, and reviewed by PhD candidates. With the expected effect on quality (due to lack of experience and impostor-syndrome-induced "how can I propose to reject? They are likely better than me"). But the Prof-to-postdoc-to-PhD-ratio was particularly bad (1-2-15).
    - kelipso 3y ago
      I was reviewing papers starting second semester of grad school with my advisor just signing off on it, so not even PhD candidates, and it was the same for my lab mates too. Initially we spent probably a few hours on a paper for peer review because we were relatively unfamiliar with the field but eventually I spent maybe a couple of hours doing the review. Wouldn't say peer review is a joke but it's definitely overrated by the public.
    - godelski 3y ago
      > "how can I propose to reject? They are likely better than me" Funny enough, I see exactly the opposite. I've seen this in both reviews I've done and reviews I've received. Just this week I reviewed and saw one of my fellow reviewers write in their justifications: I am not familiar with X, but I am skeptical that the method can scale to a more complex application. Their weaknesses section was extremely generic and it was very clear they didn't understand the work. They gave a weak reject. In fact, when I first started reviewing, I was explicitly told to _only_ accept if I was confident that the work was good. So in my experience, the bias goes the other way that you are proposing. Btw, I've even seen undergrads acting as reviewers. I was asked to review in in my first year of grad school. I don't think I was qualified then, but I was always a junior reviewer rather than a full so idk.

techdragon 3y ago

This waste of effort by way of duplicating unpublished negative results is a big factor in why replicated results deserve to be rated more highly than results that have not been replicated regardless of the prestige of the researchers or the institutions involved… if no one can prove your work work was correct… how much can anyone trust your work… I have gone down the rabbit hole of engineering research before and 90% of the time I’ve managed to find an anecdote or subsequent research footnotes or actual subsequent research publications, that substantially invalidated the lofty claims of the engineers in the 70s or 80s (which is amazing still despite this, a genuine treasure trove of research unused and sometimes useful aerospace engineering research and development) and unfortunately outside the few proper publications, a lot of the invalidations are not properly reverse cited research material and I could have spent a week cross referencing before I spot the link and realise the unnamed work they are saying they are proving wrong is actually some footnotes containing the only published data (before their new paper) on some old work that has a bad scan copy on the NASA NTRS server under some obscure title and no related keywords to the topic the research is notionally about… Academic research can genuinely suck sometimes… particularly when you want to actually apply it.

aeternum 3y ago
Publishing positive/noteworth results only does seem like an embarrassingly obvious major flaw in academia and the greater scientific community. A research assistant would quickly be thrown out if he/she refused to record negative experimental results, yet we somehow decide that is fine when operating as a collective.
godelski 3y ago
Alternatively, maybe researchers should be encouraged to publish all kinds of results and not just "novel" work. Successes, failures, insights, etc. This idea of how we evaluate researchers is often silly. We're asking people to push the bounds of knowledge and we don't know what will come from it or how long it will take. But we are hyper focused on this short term evaluation.

vibrio 3y ago

“They also tend to over-estimate the effect of peer review (often equating peer review with validity).“ In my experience, scientists ate comfortably cynical about peer review- even those that serve as reviewers and editors- except maybe junior scientists that haven’t gotten burned yet.

kergonath 3y ago
Yes, because we know how the metaphorical sausage is made: with unpaid reviewers who have many other, more interesting things to do and often an axe to grind. That is, if they don’t delegate the review to one of their post-docs.
- aftoprokrustes 3y ago
  Post doc? In what kind of utopian field did you work? In my former institute virtually all papers were written by PhD candidates, and reviewed by PhD candidates. With the expected effect on quality (due to lack of experience and impostor-syndrome-induced "how can I propose to reject? They are likely better than me"). But the Prof-to-postdoc-to-PhD-ratio was particularly bad (1-2-15).
  - kelipso 3y ago
    I was reviewing papers starting second semester of grad school with my advisor just signing off on it, so not even PhD candidates, and it was the same for my lab mates too. Initially we spent probably a few hours on a paper for peer review because we were relatively unfamiliar with the field but eventually I spent maybe a couple of hours doing the review. Wouldn't say peer review is a joke but it's definitely overrated by the public.
  - godelski 3y ago
    > "how can I propose to reject? They are likely better than me" Funny enough, I see exactly the opposite. I've seen this in both reviews I've done and reviews I've received. Just this week I reviewed and saw one of my fellow reviewers write in their justifications: I am not familiar with X, but I am skeptical that the method can scale to a more complex application. Their weaknesses section was extremely generic and it was very clear they didn't understand the work. They gave a weak reject. In fact, when I first started reviewing, I was explicitly told to _only_ accept if I was confident that the work was good. So in my experience, the bias goes the other way that you are proposing. Btw, I've even seen undergrads acting as reviewers. I was asked to review in in my first year of grad school. I don't think I was qualified then, but I was always a junior reviewer rather than a full so idk.