4 ms·

It feels like a large part of the value of a journal like the Lancet is that they DON'T do stuff like this. That one of the authors is ceo of the company that o

It feels like a large part of the value of a journal like the Lancet is that they DON'T do stuff like this. That one of the authors is ceo of the company that owns the data behind the study should be a glaring red flag. What value does the Lancet still have? Note - I edited this post because it was much more crass and reactionary, and realized its not at all my domain, so I rephrased.

knzhou 6y ago
The value is that even given this, they're much more reliable than preprints, and much much more reliable than sensational Twitter threads or public forums. They manage to filter out a tremendous amount of crap, but a sufficiently crafty and dishonest author can still sometimes slip past -- as is possible in any system.
- sbierwagen 6y ago
  Sure, but lots of high profile papers in top end journals like Nature fail to replicate: https://www.vox.com/science-and-health/2018/8/27/17761466/psychology-replication-crisis-nature-social-science https://www.vox.com/science-and-health/2018/8/27/17761466/ps... If we are to learn anything from the last 10 years of the replication crisis, it has to be that a single paper is a data point, not a conclusion.
  - Fishysoup 6y ago
    Scientists, especially in biology (possibly psychology too but i don't know), do look at papers as data points and not conclusions. The "conclusion-minded" treatment of papers is a sad artefact of the publish-or-perish system (have to add a ton of spin to your results to make them publishable) and of pop science journalism.
  - DoingIsLearning 6y ago
    This is neither new nor surprising. There is a whole field of science dedicated to the statistical analysis of multiple studies in prior literature, e.g. just search for _Meta-analysis_ in the context of medical research. A state-of-the-art Meta-Analysis is indeed the strongest level of scientific evidence you can have to advise a healthcare decisions or guidelines. The only purpose of these journals with what they call a high 'impact factor', is that they typically have their peer reviews performed by people who are regarded as the _best_ in the their particular field. Nevertheless, to assume that an expert cannot be conned is also naive. edit: typo
  - meowface 6y ago
    >If we are to learn anything from the last 10 years of the replication crisis, it has to be that a single paper is a data point, not a conclusion. I highly recommend this elaboration upon that idea: https://slatestarcodex.com/2014/12/12/beware-the-man-of-one-study/ https://slatestarcodex.com/2014/12/12/beware-the-man-of-one-...
saalweachter 6y ago
A large part of the value of a journal like the Lancet is that when stuff like this does happen, people notice and corrections get made.
- occamrazor 6y ago
  And in all the cases in which this doesn’t happen, either because the editors/referees notice it or because the aspiring fraudsters don’t even try, because they don’t think that they could slip through.
  - yostrovs 6y ago
    They did publish a paper the numbers within which were not reviewed by anyone. I, along with others, have been under the impression that such things are indeed double checked by other scientists reviewing papers. Turns out they are not.
    - saalweachter 6y ago
      I'm not saying this isn't a black mark on a prestigious journal, or that someone isn't getting chewed out by their boss for failing to catch this problem in pre-publication reviews, but ... mistakes are always going to happen. As much as we all eagerly drool over and discuss the latest publications -- on this very website! -- in everything from astrophysics to medicine, something being in published in a prestigious journal is not the gold stamp that it is true. It's just the middle of the process; the reviewers thought it was interesting enough -- and not fraudulent! -- to share with the scientific community at large, and now everyone will examine it, poke holes in it, or try to build on it. A lot of things are published in journals that end up being wrong. Most of the time it's not fraud, it's just some combination of math errors, experimental errors, and statistical flukes. And back to the original point: you can find dozens, hundreds of articles published five, ten, twenty years ago that say all sorts of amazing and outlandish things, published in sufficiently minor journals that they didn't receive sufficient attention or follow-up work. Don't take them as gospel just because they got published: you need to study the entire body of scientific research on a subject to understand what the consensus is on what's probably true, what's probably not, and what's in the nebulous zone of unknown.

knzhou 6y ago

The value is that even given this, they're much more reliable than preprints, and much much more reliable than sensational Twitter threads or public forums. They manage to filter out a tremendous amount of crap, but a sufficiently crafty and dishonest author can still sometimes slip past -- as is possible in any system.

sbierwagen 6y ago
Sure, but lots of high profile papers in top end journals like Nature fail to replicate: https://www.vox.com/science-and-health/2018/8/27/17761466/psychology-replication-crisis-nature-social-science https://www.vox.com/science-and-health/2018/8/27/17761466/ps... If we are to learn anything from the last 10 years of the replication crisis, it has to be that a single paper is a data point, not a conclusion.
- Fishysoup 6y ago
  Scientists, especially in biology (possibly psychology too but i don't know), do look at papers as data points and not conclusions. The "conclusion-minded" treatment of papers is a sad artefact of the publish-or-perish system (have to add a ton of spin to your results to make them publishable) and of pop science journalism.
- DoingIsLearning 6y ago
  This is neither new nor surprising. There is a whole field of science dedicated to the statistical analysis of multiple studies in prior literature, e.g. just search for _Meta-analysis_ in the context of medical research. A state-of-the-art Meta-Analysis is indeed the strongest level of scientific evidence you can have to advise a healthcare decisions or guidelines. The only purpose of these journals with what they call a high 'impact factor', is that they typically have their peer reviews performed by people who are regarded as the _best_ in the their particular field. Nevertheless, to assume that an expert cannot be conned is also naive. edit: typo
- meowface 6y ago
  >If we are to learn anything from the last 10 years of the replication crisis, it has to be that a single paper is a data point, not a conclusion. I highly recommend this elaboration upon that idea: https://slatestarcodex.com/2014/12/12/beware-the-man-of-one-study/ https://slatestarcodex.com/2014/12/12/beware-the-man-of-one-...

saalweachter 6y ago

A large part of the value of a journal like the Lancet is that when stuff like this does happen, people notice and corrections get made.

occamrazor 6y ago
And in all the cases in which this doesn’t happen, either because the editors/referees notice it or because the aspiring fraudsters don’t even try, because they don’t think that they could slip through.
- yostrovs 6y ago
  They did publish a paper the numbers within which were not reviewed by anyone. I, along with others, have been under the impression that such things are indeed double checked by other scientists reviewing papers. Turns out they are not.
  - saalweachter 6y ago
    I'm not saying this isn't a black mark on a prestigious journal, or that someone isn't getting chewed out by their boss for failing to catch this problem in pre-publication reviews, but ... mistakes are always going to happen. As much as we all eagerly drool over and discuss the latest publications -- on this very website! -- in everything from astrophysics to medicine, something being in published in a prestigious journal is not the gold stamp that it is true. It's just the middle of the process; the reviewers thought it was interesting enough -- and not fraudulent! -- to share with the scientific community at large, and now everyone will examine it, poke holes in it, or try to build on it. A lot of things are published in journals that end up being wrong. Most of the time it's not fraud, it's just some combination of math errors, experimental errors, and statistical flukes. And back to the original point: you can find dozens, hundreds of articles published five, ten, twenty years ago that say all sorts of amazing and outlandish things, published in sufficiently minor journals that they didn't receive sufficient attention or follow-up work. Don't take them as gospel just because they got published: you need to study the entire body of scientific research on a subject to understand what the consensus is on what's probably true, what's probably not, and what's in the nebulous zone of unknown.