3 ms·

1. You simply can’t do this in many fields, including wide swathes of health research. It’s a non-starter. 2a. Even if you could, see the OP’s point about

1. You simply can’t do this in many fields, including wide swathes of health research. It’s a non-starter. 2a. Even if you could, see the OP’s point about research design. You need to understand the methods. 2b. Even if you could part (b), the raw doesn’t help you with publication bias. Your selection of raw data is going to be heavily biased in favor of the data for the papers which managed to be published. It’s not going to contain the data for the results which never saw the light of day due to failure to reach statistical significance.

NoMoreNicksLeft 3y ago
What prevents it from becoming the norm for the "statistically insignificant" data to be published alongside the rest? Even point 2a seems weak. That must be true of any research... someone who did not know or did not understand the methodology should fail to make sense of any data provided, and that's on them not on the research itself. Point 1 is really the one I don't understand. Is it medical/health privacy that prevents this? Surely no one expects unanonymized data to be provided. This is about teasing out bad research, not indicting anyone for the falsified sort.
- huitzitziltzin 3y ago
  >>What prevents it from becoming the norm for the "statistically insignificant" data to be published alongside the rest? The incentives of the researcher. You don’t want to be the guy whose lab produces ten failures and one success, even if that’s the reality of every lab. The guy who doesn’t share the null results will look better than the one who does every time. Deans and chairs only know how to count one thing in tenure decisions - lines on your CV. To put this another way, nothing stops this kind of sharing right now, and yet we have a well known file drawer problem. If the proposed solution involves “oh let’s just completely change the social norms, despite the incentives of the individuals involved,” then it’s a non starter. The kind of sharing being proposed is that kind of idea. >>Point 1 is really the one I don't understand. Is it medical/health privacy that prevents this? Surely no one expects unanonymized data to be provided. This is about teasing out bad research, not indicting anyone for the falsified sort. What you consider “anonymized” is generally not what regulatory agencies will consider anonymized and the best sources for health data at scale are always governments. (Frequently they are the only sources.) The agencies are too risk averse about the potential harms from data releases, but that’s the reality. You can’t share it. You can’t even have it on your own laptop, encrypted or not. Interesting data that lets you answer complicated questions cannot be made public. The more interesting and the more complicated, the harder the underlying data is to access and if you say you want to share it publicly they’ll laugh you out of the room. You’ll never make it through the already excessively burdensome IRB process.
- huitzitziltzin 3y ago
  The point I am trying to make in 2(a) (maybe inexpertly) is that all the data sharing in the world isn’t going to solve any problems when the people looking at to do their own “replications” are uninformed about methods. Data sharing on its own will not solve this.
  - agentgumshoe 3y ago
    That's irrelevant though. What can happen is that others who understand the many methodologies can apply them and either retest the same study or come to different finding with other methods and compare. At least the base data is the same.

NoMoreNicksLeft 3y ago

What prevents it from becoming the norm for the "statistically insignificant" data to be published alongside the rest? Even point 2a seems weak. That must be true of any research... someone who did not know or did not understand the methodology should fail to make sense of any data provided, and that's on them not on the research itself. Point 1 is really the one I don't understand. Is it medical/health privacy that prevents this? Surely no one expects unanonymized data to be provided. This is about teasing out bad research, not indicting anyone for the falsified sort.

huitzitziltzin 3y ago
>>What prevents it from becoming the norm for the "statistically insignificant" data to be published alongside the rest? The incentives of the researcher. You don’t want to be the guy whose lab produces ten failures and one success, even if that’s the reality of every lab. The guy who doesn’t share the null results will look better than the one who does every time. Deans and chairs only know how to count one thing in tenure decisions - lines on your CV. To put this another way, nothing stops this kind of sharing right now, and yet we have a well known file drawer problem. If the proposed solution involves “oh let’s just completely change the social norms, despite the incentives of the individuals involved,” then it’s a non starter. The kind of sharing being proposed is that kind of idea. >>Point 1 is really the one I don't understand. Is it medical/health privacy that prevents this? Surely no one expects unanonymized data to be provided. This is about teasing out bad research, not indicting anyone for the falsified sort. What you consider “anonymized” is generally not what regulatory agencies will consider anonymized and the best sources for health data at scale are always governments. (Frequently they are the only sources.) The agencies are too risk averse about the potential harms from data releases, but that’s the reality. You can’t share it. You can’t even have it on your own laptop, encrypted or not. Interesting data that lets you answer complicated questions cannot be made public. The more interesting and the more complicated, the harder the underlying data is to access and if you say you want to share it publicly they’ll laugh you out of the room. You’ll never make it through the already excessively burdensome IRB process.
huitzitziltzin 3y ago
The point I am trying to make in 2(a) (maybe inexpertly) is that all the data sharing in the world isn’t going to solve any problems when the people looking at to do their own “replications” are uninformed about methods. Data sharing on its own will not solve this.
- agentgumshoe 3y ago
  That's irrelevant though. What can happen is that others who understand the many methodologies can apply them and either retest the same study or come to different finding with other methods and compare. At least the base data is the same.