3 ms·
Two separate issues here - 1. Make the underlying model public (e.g., you ran such and such regression, here’s what you included and here’s a table of coeffici
by huitzitziltzin 6y ago
Two separate issues here -
1. Make the underlying model public (e.g., you ran such and such regression, here’s what you included and here’s a table of coefficients). Inarguably a good idea. Totally unobjectionable. You should correctly be suspicious if the model (and the R, Stata, SAS or whatever code to produce it) is not available.
2. Make the underlying data public. Completely different! I have done research with (e.g.) detailed birth certificate level data. Such data are confidential by state and federal laws generally. Other researchers work with detailed income histories from the IRS. As another example, I have worked with electronic health records, which include names, addresses, health conditions, etc. Almost no one thinks this stuff should be public.
It is fair IMO to require researchers to describe what data they use and how someone else might get it (even if that includes the line “find a collaborator at the IRS”). But it is unreasonable to require researchers to make data itself public, when they may not have the right to do so.
What’s worse is that the reason this regulation is being proposed is to prevent research informed by private, confidential health data (eg birth certs) from being used to make public policy, especially in areas around pollution and environmental regulation.
- esrauch 6y agoI think this separation falls down badly in some cases: after someone claims a method worked but it's a dubious model, you can't possibly reproduce their results then there's no way to know that their model actually even worked once when they did it or if they just made an entirely stupid mistake (like using the complete data set for training). Concretely there's applied ML papers on confidential military data, where I'm fairly sure the results are just bogus or cherry-picked or done with incorrect procedure, but with the data being unobtainable (even to those with clearance the data isn't materialized in some DoD database for reproducing) it makes the published results entirely unreproducible and unchallengeable.
- Imnimo 6y agoPoint 1 might be less clear cut in some settings. Obviously, if you're just doing a simple regression, it's no big deal, but more powerful models can memorize and expose parts of their training data. See for example: https://arxiv.org/abs/1802.08232 https://arxiv.org/abs/1802.08232