4 ms·

How can you do science if around a quarter of the data is just straight up noise? Even when analysing large amounts of studies the results becone contaminated v

How can you do science if around a quarter of the data is just straight up noise? Even when analysing large amounts of studies the results becone contaminated very easily. And why is it not standard practice to provide anonymous data or even publish the data? What reason exists for that? So that only the researchers them selves can analyze it?

thechao 3y ago
The working stuff becomes part of a body of lore of "real science" that you soak up in the lab. Not a great method, for sure.
- constantcrying 3y ago
  But the point of RCTs is that you can get unbiased, high quality results without having to rely on "lore" spreading among medical professionals about which treatments are effective and under which circumstances.
mike_hearn 3y ago
Yes the problem is that the research system rewards publishing papers, but most of the work is collecting the data. So if you release your data then other groups can write papers based on your effort for far less cost. It's sort of analogous to the problem of open source business models in the software world: if company A writes the code and releases it for free and earns money from running a cloud service, and company B just offers a cloud service, then the second company can get much higher margins because they don't have to develop it. Unfortunately it's not easy to see what the alternatives are, beyond simply not funding research through government/foundation grants. When science is paid for by companies you don't have this incentive issue because the research is judged based on (ultimately) whether it leads to successful products, not whether it leads to lots of papers getting published. You have other incentive issues of course.
twic 3y ago
That's actually mentioned in the article, which is quite good and worth reading: > In 2016, the International Committee of Medical Journal Editors (ICMJE), an influential body that sets policy for many major medical titles, had proposed requiring mandatory data-sharing from RCTs. But it got pushback — including over perceived risks to the privacy of trial participants who might not have consented to their data being shared, and the availability of resources for archiving the data. As a result, in the latest update to its guidance, in 2017, it settled for merely encouraging data sharing and requiring statements about whether and where data would be shared.
jonlucc 3y ago
I work in preclinical pharma research, so I have spent a good amount of time trying to recapitulate published data in animal models, so not quite clinical trial data. People who do this work learn how to evaluate trustworthiness. It can be as granular as "this lab is the only one publishing this kind of information, so we'll be skeptical" to a bit more broad "this class of drug isn't expected to have that biology" to "I trust this company over that institution". We route around the fact that some information isn't reliable, and that's not always because of dishonesty or fraud. I'm not a clinician and don't deal with them regularly, but the impression I get is that new studies are published by researchers who have a lot of connections (dubbed thought leaders). They present at conferences. Other clinicians pick up the use case that matches their need (this patient has failed other therapies for this indication, let's try this new thing I'm now aware of). Then as experience grows, clinicians have more nuanced understanding of the use cases for that new information and its reliability. Frustratingly, this can take years, but that's bug that's also a feature.
evandijk70 3y ago
Privacy concerns are the most important reason. A more cynical reason might that the odds that a papers is deemed 'ok' is far larger if the raw/anonymous data is not provided. Remember, the authors of the suspect studies provided their data voluntarily, and in the end it only hurt their reputation/impact.
lost_tourist 3y ago
Not a quarter of the data, quarter of the papers. There's a big difference there.

thechao 3y ago

The working stuff becomes part of a body of lore of "real science" that you soak up in the lab. Not a great method, for sure.

constantcrying 3y ago
But the point of RCTs is that you can get unbiased, high quality results without having to rely on "lore" spreading among medical professionals about which treatments are effective and under which circumstances.

mike_hearn 3y ago

Yes the problem is that the research system rewards publishing papers, but most of the work is collecting the data. So if you release your data then other groups can write papers based on your effort for far less cost. It's sort of analogous to the problem of open source business models in the software world: if company A writes the code and releases it for free and earns money from running a cloud service, and company B just offers a cloud service, then the second company can get much higher margins because they don't have to develop it. Unfortunately it's not easy to see what the alternatives are, beyond simply not funding research through government/foundation grants. When science is paid for by companies you don't have this incentive issue because the research is judged based on (ultimately) whether it leads to successful products, not whether it leads to lots of papers getting published. You have other incentive issues of course.

twic 3y ago

That's actually mentioned in the article, which is quite good and worth reading: > In 2016, the International Committee of Medical Journal Editors (ICMJE), an influential body that sets policy for many major medical titles, had proposed requiring mandatory data-sharing from RCTs. But it got pushback — including over perceived risks to the privacy of trial participants who might not have consented to their data being shared, and the availability of resources for archiving the data. As a result, in the latest update to its guidance, in 2017, it settled for merely encouraging data sharing and requiring statements about whether and where data would be shared.

jonlucc 3y ago

I work in preclinical pharma research, so I have spent a good amount of time trying to recapitulate published data in animal models, so not quite clinical trial data. People who do this work learn how to evaluate trustworthiness. It can be as granular as "this lab is the only one publishing this kind of information, so we'll be skeptical" to a bit more broad "this class of drug isn't expected to have that biology" to "I trust this company over that institution". We route around the fact that some information isn't reliable, and that's not always because of dishonesty or fraud. I'm not a clinician and don't deal with them regularly, but the impression I get is that new studies are published by researchers who have a lot of connections (dubbed thought leaders). They present at conferences. Other clinicians pick up the use case that matches their need (this patient has failed other therapies for this indication, let's try this new thing I'm now aware of). Then as experience grows, clinicians have more nuanced understanding of the use cases for that new information and its reliability. Frustratingly, this can take years, but that's bug that's also a feature.

evandijk70 3y ago

Privacy concerns are the most important reason. A more cynical reason might that the odds that a papers is deemed 'ok' is far larger if the raw/anonymous data is not provided. Remember, the authors of the suspect studies provided their data voluntarily, and in the end it only hurt their reputation/impact.

lost_tourist 3y ago

Not a quarter of the data, quarter of the papers. There's a big difference there.