4 ms·

A lot of the controversy surrounding this Lancet paper was about the data being used, which was provided by a firm named Surgisphere [1]. One of the co-authors

by pen2l 6y ago

A lot of the controversy surrounding this Lancet paper was about the data being used, which was provided by a firm named Surgisphere [1]. One of the co-authors of the Lancet paper is Sapan Desai, he is incidentally also the chief executive of this firm. The lone one who chose not to retract the paper was him. I recommend this twitter thread that explains the thrust of the issue: https://twitter.com/reverendofdoubt/status/1266902969045463041 https://twitter.com/reverendofdoubt/status/12669029690454630... Essentially, doubt began swirling around when this company claimed to acquire in a very short time massive amounts of EHR (Electronic Health Record, basically large datasets showing how patients are doing, how they're responding to medication, etc.). When Sapan Desai was pressed on by the other three authors of this very paper, he refused to cooperate, invoking a somewhat shoddy defense of not being at liberty to reveal sensitive information. Surgisphere refuses to say which hospitals chose to give it the EHR. The thing is, this EHR data is a big deal. Pharma companies give _big_ money to hospitals for this data. One of the key questions has been how this newcomer company managed to acquire these EHR deals with such a large number of hospitals. Since the scrutiny, NEJM has also chosen to retract a Covid-19-related paper which sourced data from this company: https://www.nejm.org/doi/full/10.1056/NEJMoa2007621 https://www.nejm.org/doi/full/10.1056/NEJMoa2007621 [1]: The company is choosing to stand behind the data: https://surgisphere.com/2020/05/29/response-to-widespread-reaction-to-recent-lancet-article-on-hydroxychloroquine/ https://surgisphere.com/2020/05/29/response-to-widespread-re...

taurath 6y ago
It feels like a large part of the value of a journal like the Lancet is that they DON'T do stuff like this. That one of the authors is ceo of the company that owns the data behind the study should be a glaring red flag. What value does the Lancet still have? Note - I edited this post because it was much more crass and reactionary, and realized its not at all my domain, so I rephrased.
- knzhou 6y ago
  The value is that even given this, they're much more reliable than preprints, and much much more reliable than sensational Twitter threads or public forums. They manage to filter out a tremendous amount of crap, but a sufficiently crafty and dishonest author can still sometimes slip past -- as is possible in any system.
  - sbierwagen 6y ago
    Sure, but lots of high profile papers in top end journals like Nature fail to replicate: https://www.vox.com/science-and-health/2018/8/27/17761466/psychology-replication-crisis-nature-social-science https://www.vox.com/science-and-health/2018/8/27/17761466/ps... If we are to learn anything from the last 10 years of the replication crisis, it has to be that a single paper is a data point, not a conclusion.
    - Fishysoup 6y ago
      Scientists, especially in biology (possibly psychology too but i don't know), do look at papers as data points and not conclusions. The "conclusion-minded" treatment of papers is a sad artefact of the publish-or-perish system (have to add a ton of spin to your results to make them publishable) and of pop science journalism.
    - DoingIsLearning 6y ago
      This is neither new nor surprising. There is a whole field of science dedicated to the statistical analysis of multiple studies in prior literature, e.g. just search for _Meta-analysis_ in the context of medical research. A state-of-the-art Meta-Analysis is indeed the strongest level of scientific evidence you can have to advise a healthcare decisions or guidelines. The only purpose of these journals with what they call a high 'impact factor', is that they typically have their peer reviews performed by people who are regarded as the _best_ in the their particular field. Nevertheless, to assume that an expert cannot be conned is also naive. edit: typo
    - meowface 6y ago
      >If we are to learn anything from the last 10 years of the replication crisis, it has to be that a single paper is a data point, not a conclusion. I highly recommend this elaboration upon that idea: https://slatestarcodex.com/2014/12/12/beware-the-man-of-one-study/ https://slatestarcodex.com/2014/12/12/beware-the-man-of-one-...
- saalweachter 6y ago
  A large part of the value of a journal like the Lancet is that when stuff like this does happen, people notice and corrections get made.
  - occamrazor 6y ago
    And in all the cases in which this doesn’t happen, either because the editors/referees notice it or because the aspiring fraudsters don’t even try, because they don’t think that they could slip through.
    - yostrovs 6y ago
      They did publish a paper the numbers within which were not reviewed by anyone. I, along with others, have been under the impression that such things are indeed double checked by other scientists reviewing papers. Turns out they are not.
      - saalweachter 6y ago
        I'm not saying this isn't a black mark on a prestigious journal, or that someone isn't getting chewed out by their boss for failing to catch this problem in pre-publication reviews, but ... mistakes are always going to happen. As much as we all eagerly drool over and discuss the latest publications -- on this very website! -- in everything from astrophysics to medicine, something being in published in a prestigious journal is not the gold stamp that it is true. It's just the middle of the process; the reviewers thought it was interesting enough -- and not fraudulent! -- to share with the scientific community at large, and now everyone will examine it, poke holes in it, or try to build on it. A lot of things are published in journals that end up being wrong. Most of the time it's not fraud, it's just some combination of math errors, experimental errors, and statistical flukes. And back to the original point: you can find dozens, hundreds of articles published five, ten, twenty years ago that say all sorts of amazing and outlandish things, published in sufficiently minor journals that they didn't receive sufficient attention or follow-up work. Don't take them as gospel just because they got published: you need to study the entire body of scientific research on a subject to understand what the consensus is on what's probably true, what's probably not, and what's in the nebulous zone of unknown.
kgwgk 6y ago
> NEJM has also chosen to retract a Covid-19-related paper For the NEJM paper, as for the Lancet paper, the retraction is coming from the authors (including Desai in the NEJM case): https://www.nejm.org/doi/full/10.1056/NEJMc2021225 https://www.nejm.org/doi/full/10.1056/NEJMc2021225
- CodeWriter23 6y ago
  Because they got caught with their hands in the Data Laundering Cookie Jar.
  - kgwgk 6y ago
    Of course. But it was “voluntary”, not done by the journals (and it would probably have taken longer in that case).
mattmanser 6y ago
As far as I've read, Surgisphere claimed to have a study based on deaths in Australia using paid for data from Australian hospitals. But the data was citing a death figure that was corrected weeks after the apparent 'study', which looked really suspect. When contacted not a single hospital in Australia knew of Surgisphere or the study, and none of them had any commercial relationship with Surgisphere.
- giarc 6y ago
  >When contacted not a single hospital in Australia knew of Surgisphere or the study, and none of them had any commercial relationship with Surgisphere. To be fair, I wouldn't expect any individual hospital (or individual at the hospital) to be aware of this. Australia has public healthcare and therefore data is stored centrally. There's likely a government department dedicated to data and they would likely be the ones to provide the data, not someone at the hospital.
  - pen2l 6y ago
    Someone at the hospital in US would certainly be aware of it for two reasons: 1) It takes a significant amount of work to format and standardize this data 2) Sharing this data is a business model for hospitals, it's a means of income for hospitals in this day and age. It's not unheard of that a pharma company will pay up double-digit million dollars for this data. It is thus unlikely that top brass for a hospital network is ignorant of the nature of this data and with whom it is being shared.
  - baylisscg 6y ago
    No, it's not centrally collected. There is a national eHealth system, MyHealthRecord, but it's opt in for patients, supplementary to your provider's records, and various pieces of privacy legislation at the state and national level make handing out this data, at best, legally fraught. Speaking from experience. Getting data out of a hospital is phenomenally hard even when the hospital is wants to provide it. Patient confidentiality is a thing they all take very seriously.

by pen2l 6y ago

taurath 6y ago

It feels like a large part of the value of a journal like the Lancet is that they DON'T do stuff like this. That one of the authors is ceo of the company that owns the data behind the study should be a glaring red flag. What value does the Lancet still have? Note - I edited this post because it was much more crass and reactionary, and realized its not at all my domain, so I rephrased.

knzhou 6y ago
The value is that even given this, they're much more reliable than preprints, and much much more reliable than sensational Twitter threads or public forums. They manage to filter out a tremendous amount of crap, but a sufficiently crafty and dishonest author can still sometimes slip past -- as is possible in any system.
- sbierwagen 6y ago
  Sure, but lots of high profile papers in top end journals like Nature fail to replicate: https://www.vox.com/science-and-health/2018/8/27/17761466/psychology-replication-crisis-nature-social-science https://www.vox.com/science-and-health/2018/8/27/17761466/ps... If we are to learn anything from the last 10 years of the replication crisis, it has to be that a single paper is a data point, not a conclusion.
  - Fishysoup 6y ago
    Scientists, especially in biology (possibly psychology too but i don't know), do look at papers as data points and not conclusions. The "conclusion-minded" treatment of papers is a sad artefact of the publish-or-perish system (have to add a ton of spin to your results to make them publishable) and of pop science journalism.
  - DoingIsLearning 6y ago
    This is neither new nor surprising. There is a whole field of science dedicated to the statistical analysis of multiple studies in prior literature, e.g. just search for _Meta-analysis_ in the context of medical research. A state-of-the-art Meta-Analysis is indeed the strongest level of scientific evidence you can have to advise a healthcare decisions or guidelines. The only purpose of these journals with what they call a high 'impact factor', is that they typically have their peer reviews performed by people who are regarded as the _best_ in the their particular field. Nevertheless, to assume that an expert cannot be conned is also naive. edit: typo
  - meowface 6y ago
    >If we are to learn anything from the last 10 years of the replication crisis, it has to be that a single paper is a data point, not a conclusion. I highly recommend this elaboration upon that idea: https://slatestarcodex.com/2014/12/12/beware-the-man-of-one-study/ https://slatestarcodex.com/2014/12/12/beware-the-man-of-one-...
saalweachter 6y ago
A large part of the value of a journal like the Lancet is that when stuff like this does happen, people notice and corrections get made.
- occamrazor 6y ago
  And in all the cases in which this doesn’t happen, either because the editors/referees notice it or because the aspiring fraudsters don’t even try, because they don’t think that they could slip through.
  - yostrovs 6y ago
    They did publish a paper the numbers within which were not reviewed by anyone. I, along with others, have been under the impression that such things are indeed double checked by other scientists reviewing papers. Turns out they are not.
    - saalweachter 6y ago
      I'm not saying this isn't a black mark on a prestigious journal, or that someone isn't getting chewed out by their boss for failing to catch this problem in pre-publication reviews, but ... mistakes are always going to happen. As much as we all eagerly drool over and discuss the latest publications -- on this very website! -- in everything from astrophysics to medicine, something being in published in a prestigious journal is not the gold stamp that it is true. It's just the middle of the process; the reviewers thought it was interesting enough -- and not fraudulent! -- to share with the scientific community at large, and now everyone will examine it, poke holes in it, or try to build on it. A lot of things are published in journals that end up being wrong. Most of the time it's not fraud, it's just some combination of math errors, experimental errors, and statistical flukes. And back to the original point: you can find dozens, hundreds of articles published five, ten, twenty years ago that say all sorts of amazing and outlandish things, published in sufficiently minor journals that they didn't receive sufficient attention or follow-up work. Don't take them as gospel just because they got published: you need to study the entire body of scientific research on a subject to understand what the consensus is on what's probably true, what's probably not, and what's in the nebulous zone of unknown.

kgwgk 6y ago

> NEJM has also chosen to retract a Covid-19-related paper For the NEJM paper, as for the Lancet paper, the retraction is coming from the authors (including Desai in the NEJM case): https://www.nejm.org/doi/full/10.1056/NEJMc2021225 https://www.nejm.org/doi/full/10.1056/NEJMc2021225

CodeWriter23 6y ago
Because they got caught with their hands in the Data Laundering Cookie Jar.
- kgwgk 6y ago
  Of course. But it was “voluntary”, not done by the journals (and it would probably have taken longer in that case).

mattmanser 6y ago

As far as I've read, Surgisphere claimed to have a study based on deaths in Australia using paid for data from Australian hospitals. But the data was citing a death figure that was corrected weeks after the apparent 'study', which looked really suspect. When contacted not a single hospital in Australia knew of Surgisphere or the study, and none of them had any commercial relationship with Surgisphere.

giarc 6y ago
>When contacted not a single hospital in Australia knew of Surgisphere or the study, and none of them had any commercial relationship with Surgisphere. To be fair, I wouldn't expect any individual hospital (or individual at the hospital) to be aware of this. Australia has public healthcare and therefore data is stored centrally. There's likely a government department dedicated to data and they would likely be the ones to provide the data, not someone at the hospital.
- pen2l 6y ago
  Someone at the hospital in US would certainly be aware of it for two reasons: 1) It takes a significant amount of work to format and standardize this data 2) Sharing this data is a business model for hospitals, it's a means of income for hospitals in this day and age. It's not unheard of that a pharma company will pay up double-digit million dollars for this data. It is thus unlikely that top brass for a hospital network is ignorant of the nature of this data and with whom it is being shared.
- baylisscg 6y ago
  No, it's not centrally collected. There is a national eHealth system, MyHealthRecord, but it's opt in for patients, supplementary to your provider's records, and various pieces of privacy legislation at the state and national level make handing out this data, at best, legally fraught. Speaking from experience. Getting data out of a hospital is phenomenally hard even when the hospital is wants to provide it. Patient confidentiality is a thing they all take very seriously.