4 ms·

If you were to ask experts in a given subfield which papers are reliable, I'm sure they would be able to tell you. The problem is that there's no process in sci

by parton 4y ago

If you were to ask experts in a given subfield which papers are reliable, I'm sure they would be able to tell you. The problem is that there's no process in science for expert consensus to make it to out to doctors/laypeople. People assume that peer review means a paper is good, which couldn't be farther from the truth. Science journalists aren't any better, they care more about hype than consensus. Honestly, it's dangerous to give a random peer reviewed article to someone who doesn't have broad knowledge of the field. Maybe we need middle-ground journals that publish review articles at the level of a Scientific American reader?

epgui 4y ago
That’s pretty bang on (biochemist).
AlotOfReading 4y ago
That's one of the intended purposes of review journals like Annual Reviews (https://www.annualreviews.org/ https://www.annualreviews.org/). There are some pretty big practical issues with them though: 1. Sometimes the latest article on a subject was written 2/5/10 years ago. Depending on quickly the field is moving, that review could be perfectly acceptable, completely obsolete, or anywhere in between. 2. Sometimes the author themselves is just out of date with the field. It's difficult to identify this without deep prior familiarity. 3. Paywalls. I don't know what the solutions are beyond "making experts accessible for questions" and other public outreach things.
pella 4y ago
> People assume that peer review means a paper is good ... "Conclusions: Parachute use did not reduce death or major traumatic injury when jumping from aircraft in the first randomized evaluation of this intervention." https://www.bmj.com/content/363/bmj.k5094 https://www.bmj.com/content/363/bmj.k5094 see the "Peer review" https://www.bmj.com/content/363/bmj.k5094/peer-review https://www.bmj.com/content/363/bmj.k5094/peer-review
- jnsie 4y ago
  Please tell me this is satire
  - nickff 4y ago
    From the first review: >"While this trial illustrates many important points about participant screening and recruitment in RCTs along with the need for equipoise, we feel it is important to emphasise that this is not intended to undermine the use of RCTs, but to illustrate some of the issues that may arise. We felt another important message from this paper is that if you make a decision based on abstract alone, this may lead you to an incorrect assumption, so this trial illustrates very well the importance of reading the whole paper."
    - PeterisP 4y ago
      The abstract is quite explicit about the limitations of the study, namely, that all the people without a parachute jumped from a very low altitude (mean of 0.6m) and that the results should not be extrapolated to jumping out of planes which are much higher than that..
  - CrazyStat 4y ago
    The BMJ Christmas issue is full of funny (but maybe also serious) articles each year. My favorite is an analysis of reported virgin births [1]. [1] https://www.bmj.com/content/347/bmj.f7102 https://www.bmj.com/content/347/bmj.f7102
  - avs733 4y ago
    The BMJ is a very real and serious. This article is from the “Christmas Issue” which is a little different…obviously. It is meant to be lighthearted and tongue in cheek - but no less serious of science. It often exists as a form of satirical critique of bad habits of science and science communication. There are flaws in that model though that have been noted. Prime among them is that nothing clearly identifies an article as from the Christmas issue online, and even if it did that may not carry any meaning for a reader unfamiliar with the journal. https://www.bmj.com/about-bmj/resources-authors/article-types/christmas-issue https://www.bmj.com/about-bmj/resources-authors/article-type...
ramraj07 4y ago
It’s not a given experts know what papers are reliable. Here’s a paper from Genentech https://pubmed.ncbi.nlm.nih.gov/15385631/ https://pubmed.ncbi.nlm.nih.gov/15385631/ with 500 citations which my advisors swore is reliable because “they trust the authors.” Don’t mind the fact that the central thesis of the article is barely supported by the last figure in which they conveniently average data from completely different sets of experiments for each time point. And did I mention my advisors released a drug into the market just recently? Lol.
brnaftr361 4y ago
Journals could open replication wings? I'd rather read about experiments that have been validated by an independent third party at this point.
- bumby 4y ago
  Part of the problem is the lack of prestige in replication. What gets researchers promoted is finding novel methods, so most researchers don’t want to spend their precious time on replication.
  - naasking 4y ago
    There is prestige in what gets you recognition, so if journals and departments started recognizing contributions to replication research, like giving them dedicated space in journals, then the prestige will follow.
    - bumby 4y ago
      Fair enough, but the incentives don’t align. Novel research attracts funding, so every if replication gets prestige in the future the business side of research will still run counter to it. A researcher would still gain more prestige work novel work that attracts substantial funding.
      - naasking 4y ago
        Most novel findings don't replicate, so a researcher at Yale can gain plenty of prestige by debunking the sensationalized findings from Harvard, for instance.
        bumby 4y ago
        >Most novel findings don't replicate This is very context dependent so such a broad generalization probably goes too far. It seems like the “soft” sciences have a much bigger replication crises. But that point aside, your Yale vs. Harvard example doesn’t fix the funding issue I alluded to.

by parton 4y ago

epgui 4y ago

That’s pretty bang on (biochemist).

AlotOfReading 4y ago

That's one of the intended purposes of review journals like Annual Reviews (https://www.annualreviews.org/ https://www.annualreviews.org/). There are some pretty big practical issues with them though: 1. Sometimes the latest article on a subject was written 2/5/10 years ago. Depending on quickly the field is moving, that review could be perfectly acceptable, completely obsolete, or anywhere in between. 2. Sometimes the author themselves is just out of date with the field. It's difficult to identify this without deep prior familiarity. 3. Paywalls. I don't know what the solutions are beyond "making experts accessible for questions" and other public outreach things.

pella 4y ago

> People assume that peer review means a paper is good ... "Conclusions: Parachute use did not reduce death or major traumatic injury when jumping from aircraft in the first randomized evaluation of this intervention." https://www.bmj.com/content/363/bmj.k5094 https://www.bmj.com/content/363/bmj.k5094 see the "Peer review" https://www.bmj.com/content/363/bmj.k5094/peer-review https://www.bmj.com/content/363/bmj.k5094/peer-review

jnsie 4y ago
Please tell me this is satire
- nickff 4y ago
  From the first review: >"While this trial illustrates many important points about participant screening and recruitment in RCTs along with the need for equipoise, we feel it is important to emphasise that this is not intended to undermine the use of RCTs, but to illustrate some of the issues that may arise. We felt another important message from this paper is that if you make a decision based on abstract alone, this may lead you to an incorrect assumption, so this trial illustrates very well the importance of reading the whole paper."
  - PeterisP 4y ago
    The abstract is quite explicit about the limitations of the study, namely, that all the people without a parachute jumped from a very low altitude (mean of 0.6m) and that the results should not be extrapolated to jumping out of planes which are much higher than that..
- CrazyStat 4y ago
  The BMJ Christmas issue is full of funny (but maybe also serious) articles each year. My favorite is an analysis of reported virgin births [1]. [1] https://www.bmj.com/content/347/bmj.f7102 https://www.bmj.com/content/347/bmj.f7102
- avs733 4y ago
  The BMJ is a very real and serious. This article is from the “Christmas Issue” which is a little different…obviously. It is meant to be lighthearted and tongue in cheek - but no less serious of science. It often exists as a form of satirical critique of bad habits of science and science communication. There are flaws in that model though that have been noted. Prime among them is that nothing clearly identifies an article as from the Christmas issue online, and even if it did that may not carry any meaning for a reader unfamiliar with the journal. https://www.bmj.com/about-bmj/resources-authors/article-types/christmas-issue https://www.bmj.com/about-bmj/resources-authors/article-type...

ramraj07 4y ago

It’s not a given experts know what papers are reliable. Here’s a paper from Genentech https://pubmed.ncbi.nlm.nih.gov/15385631/ https://pubmed.ncbi.nlm.nih.gov/15385631/ with 500 citations which my advisors swore is reliable because “they trust the authors.” Don’t mind the fact that the central thesis of the article is barely supported by the last figure in which they conveniently average data from completely different sets of experiments for each time point. And did I mention my advisors released a drug into the market just recently? Lol.

brnaftr361 4y ago

Journals could open replication wings? I'd rather read about experiments that have been validated by an independent third party at this point.

bumby 4y ago
Part of the problem is the lack of prestige in replication. What gets researchers promoted is finding novel methods, so most researchers don’t want to spend their precious time on replication.
- naasking 4y ago
  There is prestige in what gets you recognition, so if journals and departments started recognizing contributions to replication research, like giving them dedicated space in journals, then the prestige will follow.
  - bumby 4y ago
    Fair enough, but the incentives don’t align. Novel research attracts funding, so every if replication gets prestige in the future the business side of research will still run counter to it. A researcher would still gain more prestige work novel work that attracts substantial funding.
    - naasking 4y ago
      Most novel findings don't replicate, so a researcher at Yale can gain plenty of prestige by debunking the sensationalized findings from Harvard, for instance.
      - bumby 4y ago
        >Most novel findings don't replicate This is very context dependent so such a broad generalization probably goes too far. It seems like the “soft” sciences have a much bigger replication crises. But that point aside, your Yale vs. Harvard example doesn’t fix the funding issue I alluded to.