5 ms·

I only have a bachelor’s degree, but I’ve worked in research labs and read a lot of papers. I’ve always thought it was silly that all data, design documents, an

I only have a bachelor’s degree, but I’ve worked in research labs and read a lot of papers. I’ve always thought it was silly that all data, design documents, and code is not published as a standard. Is it just protocol from a time when it was more difficult to share data and designs? In the modern era, why shouldn’t research colleges and journals maintain a digital database of data, design documents, and code?

bsder 4y ago
> In the modern era, why shouldn’t research colleges and journals maintain a digital database of data, design documents, and code? Because the incentive in academia is to take a single idea that kinda works and parcel it out into as many, generally low quality, published papers as you can. The moment you release the data somebody else can start parceling out those papers instead of you.
DantesKite 4y ago
Makes me curious whether researchers would use software that integrates all these features for them automatically, making it easy to publish and share data sets in a consistent format online.
- anitil 4y ago
  This is a niche that datasette (by simonw on here) could fill very nicely. If you can wrangle it in to a csv, then datasette can publish it as an api with an sql interface. I'm not sure how this could work with things like photos etc (though there are plugins for some of this)
- gpm 4y ago
  I think you've got to be asking - What benefit does this give to the researchers who are publishing the data? - Who is paying for storing the data - frequently in the TB? When the answers are "approximately none", and "the researchers" you're not going to get many takers. And that's assuming it's easy, it reality it often won't be. If there's a lot of data, it's literally just a pain to upload it (network bandwidth). If there's sensitive data (PII), it's a pain to redact it and make sure you aren't leaking any. Data is frequently in strange formats, and it's a pain to translate it to a standard one. Etc. --- I've worked with 3 university labs as a contract programmer. In all of them I worked with data with one of the issues mentioned above. Health information in one, TBs of photonics data in another (which was being parsed by extremely janky code too), and 4 16-bit channel images in the last (hundreds of GB of them too). Admittedly for the last it would have been easy to upload it as long as you didn't want people to be able to actually view it (on the other hand I wrote some software to let people false color them live in a browser for the lab, so that the researchers could view them).
  - tgb 4y ago
    This already exists in some fields though. Gene expression sequencing data is almost universally made public through the Gene Expression Omnibus website, and that’s quite storage intensive. It’s used since regulators and journals require it to be used.
    - gpm 4y ago
      > It’s used since regulators and journals require it to be used. Which answers the "what benefit does it provide to the researchers publishing the data" question. A quick search answers the funding question as well, it's funded by the NIH, not the individual labs using it. I think this example supports my point. The NIH came up with a way to give different answers to the two questions I asked, and it gets used. I'm glad the NIH has been making this a thing, it's a great use of public funds. I'd still caution anyone from trying a "make a data platform and researchers will use it" approach to the problem unless they can answer those questions.
striking 4y ago
Because, from my understanding: it's hard to preserve things and the code is usually pretty crappy and the data is often unintelligible without a lot of help from whoever was processing it. Obviously preservation of research outputs is a noble cause, but there's very little incentive for anyone to actually document and preserve these things unless it's a requirement of the field (e.g. high-profile AI stuff). The incentive to not do that is that you save time and can jump to the next project, and also no one is pestering you about your crappy code and/or data. Think about all of the fancy dependencies a piece of code might need to run. Think of all of the processing steps some data needed to go through. Think of the work required to set up some computational cluster without which the work just doesn't happen. Doing it once is hard enough, documenting it so anyone else can do it is maybe an order of magnitude harder.
- throwaway14356 4y ago
  That would be a nice problem to have. In order to fix a problem it first needs to be an actual problem rather than a theoretical one. Maybe the goal should merely be for the author to be able replicate the results from the torrent archive (for a fee of course). That way anyone who needs the conclusion to be correct can buy the validation.
- __blockcipher__ 4y ago
  > it's hard to preserve things and the code is usually pretty crappy and the data is often unintelligible without a lot of help from whoever was processing it Which is a huge part of why the replication crisis is such a thing (besides the outright fraud and the publication bias). The very fact that the datasets and codebases are so disgusting is precisely why the results coming from that data can't be trusted.
  - seoaeu 4y ago
    > Which is a huge part of why the replication crisis is such a thing Is it though? From what I’ve seen it is mostly caused by p-hacking/small sample sizes/poor experimental design.
JBorrow 4y ago
From my perspective (postdoc in astrophysics) things are quite complicated. I agree generally with data and code sharing policies (all of my recent papers include zenodo deposits of data, and I work on a large open source simulation code). However, there are some serious issues here: 1. Code licensing is not always clear. Sometimes you are working with a code that some PhD student wrote, who left the field, whose advisor shared it with you. Can you share this code? Not necessarily... but this problem will go away with time. 2. Some codes include proprietary components, and chopping them out can make the code inoperable (or useless). One notable instance of this is a very famous hydro code that was used to produce predictions for nukes, that was then repurposed for running simulations of planetary collisions. 3. Sharing data is prohibitive, and if you don't have the data having the code might not really help. Our projects can produce Pbs of data, costing 10s-100s Ms of CPU hours, and as such making the data open source can be really tricky. This then means you can only share a selection of runs. On the other hand, even if you had the code, you can't reproduce a lot of the results, because of the cost of running the simulations... This actually will get _worse_ over time as we run bigger and bigger simulations.
- max51 4y ago
  >1. Code licensing is not always clear. Sometimes you are working with a code that some PhD student wrote, who left the field, whose advisor shared it with you. Can you share this code? Not necessarily... but this problem will go away with time. Isn't it the university who owns everything? I don't know about astrophysics, but it's a big problem in engineering. Universities have been known to patent and resell research made by msc/phd student and profit from it without paying any compensation.
  - JBorrow 4y ago
    This depends on the region/university/timescale/etc. Consider this: student at institution A (UK) writes code, postdoc at institution B (USA) modifies it, shared with researcher at institution C (China) who runs it for a paper first-authored by student at institution D (Germany). Who has the rights to share the code and data? Which jurisdiction would this fall under?
Fomite 4y ago
To illustrate a couple problems with it being standard (importantly, note this paper was in a clinical epidemiology journal, so we are invariably talking about human health data): 1) Are you sure the data is deidentified? Can your analysis be done on the deidentified data? 2) Do you actually own the data? This is more complicated than "Well you published a paper on it...". The data is likely governed by data use agreements. Those are often extremely complex - especially at the international level. Or if you're working with particular populations - they're not wrong to be wary of who their data is given to, given the history of how science generally has treated them. 3) Is the data in a form that's genuinely suited to being shared? Is it in a flat file, with clear and easy to understand variable names, a good data dictionary, etc.? Is missingness, and the reasons why, clear and evident (the difference between a variable with a lot of missingness and a variable that's a bit garbage is often subtle)? 4) Where do you put it - and importantly, who pays for that, and how long?

bsder 4y ago

> In the modern era, why shouldn’t research colleges and journals maintain a digital database of data, design documents, and code? Because the incentive in academia is to take a single idea that kinda works and parcel it out into as many, generally low quality, published papers as you can. The moment you release the data somebody else can start parceling out those papers instead of you.

DantesKite 4y ago

Makes me curious whether researchers would use software that integrates all these features for them automatically, making it easy to publish and share data sets in a consistent format online.

anitil 4y ago
This is a niche that datasette (by simonw on here) could fill very nicely. If you can wrangle it in to a csv, then datasette can publish it as an api with an sql interface. I'm not sure how this could work with things like photos etc (though there are plugins for some of this)
gpm 4y ago
I think you've got to be asking - What benefit does this give to the researchers who are publishing the data? - Who is paying for storing the data - frequently in the TB? When the answers are "approximately none", and "the researchers" you're not going to get many takers. And that's assuming it's easy, it reality it often won't be. If there's a lot of data, it's literally just a pain to upload it (network bandwidth). If there's sensitive data (PII), it's a pain to redact it and make sure you aren't leaking any. Data is frequently in strange formats, and it's a pain to translate it to a standard one. Etc. --- I've worked with 3 university labs as a contract programmer. In all of them I worked with data with one of the issues mentioned above. Health information in one, TBs of photonics data in another (which was being parsed by extremely janky code too), and 4 16-bit channel images in the last (hundreds of GB of them too). Admittedly for the last it would have been easy to upload it as long as you didn't want people to be able to actually view it (on the other hand I wrote some software to let people false color them live in a browser for the lab, so that the researchers could view them).
- tgb 4y ago
  This already exists in some fields though. Gene expression sequencing data is almost universally made public through the Gene Expression Omnibus website, and that’s quite storage intensive. It’s used since regulators and journals require it to be used.
  - gpm 4y ago
    > It’s used since regulators and journals require it to be used. Which answers the "what benefit does it provide to the researchers publishing the data" question. A quick search answers the funding question as well, it's funded by the NIH, not the individual labs using it. I think this example supports my point. The NIH came up with a way to give different answers to the two questions I asked, and it gets used. I'm glad the NIH has been making this a thing, it's a great use of public funds. I'd still caution anyone from trying a "make a data platform and researchers will use it" approach to the problem unless they can answer those questions.

striking 4y ago

Because, from my understanding: it's hard to preserve things and the code is usually pretty crappy and the data is often unintelligible without a lot of help from whoever was processing it. Obviously preservation of research outputs is a noble cause, but there's very little incentive for anyone to actually document and preserve these things unless it's a requirement of the field (e.g. high-profile AI stuff). The incentive to not do that is that you save time and can jump to the next project, and also no one is pestering you about your crappy code and/or data. Think about all of the fancy dependencies a piece of code might need to run. Think of all of the processing steps some data needed to go through. Think of the work required to set up some computational cluster without which the work just doesn't happen. Doing it once is hard enough, documenting it so anyone else can do it is maybe an order of magnitude harder.

throwaway14356 4y ago
That would be a nice problem to have. In order to fix a problem it first needs to be an actual problem rather than a theoretical one. Maybe the goal should merely be for the author to be able replicate the results from the torrent archive (for a fee of course). That way anyone who needs the conclusion to be correct can buy the validation.
__blockcipher__ 4y ago
> it's hard to preserve things and the code is usually pretty crappy and the data is often unintelligible without a lot of help from whoever was processing it Which is a huge part of why the replication crisis is such a thing (besides the outright fraud and the publication bias). The very fact that the datasets and codebases are so disgusting is precisely why the results coming from that data can't be trusted.
- seoaeu 4y ago
  > Which is a huge part of why the replication crisis is such a thing Is it though? From what I’ve seen it is mostly caused by p-hacking/small sample sizes/poor experimental design.

JBorrow 4y ago

From my perspective (postdoc in astrophysics) things are quite complicated. I agree generally with data and code sharing policies (all of my recent papers include zenodo deposits of data, and I work on a large open source simulation code). However, there are some serious issues here: 1. Code licensing is not always clear. Sometimes you are working with a code that some PhD student wrote, who left the field, whose advisor shared it with you. Can you share this code? Not necessarily... but this problem will go away with time. 2. Some codes include proprietary components, and chopping them out can make the code inoperable (or useless). One notable instance of this is a very famous hydro code that was used to produce predictions for nukes, that was then repurposed for running simulations of planetary collisions. 3. Sharing data is prohibitive, and if you don't have the data having the code might not really help. Our projects can produce Pbs of data, costing 10s-100s Ms of CPU hours, and as such making the data open source can be really tricky. This then means you can only share a selection of runs. On the other hand, even if you had the code, you can't reproduce a lot of the results, because of the cost of running the simulations... This actually will get _worse_ over time as we run bigger and bigger simulations.

max51 4y ago
>1. Code licensing is not always clear. Sometimes you are working with a code that some PhD student wrote, who left the field, whose advisor shared it with you. Can you share this code? Not necessarily... but this problem will go away with time. Isn't it the university who owns everything? I don't know about astrophysics, but it's a big problem in engineering. Universities have been known to patent and resell research made by msc/phd student and profit from it without paying any compensation.
- JBorrow 4y ago
  This depends on the region/university/timescale/etc. Consider this: student at institution A (UK) writes code, postdoc at institution B (USA) modifies it, shared with researcher at institution C (China) who runs it for a paper first-authored by student at institution D (Germany). Who has the rights to share the code and data? Which jurisdiction would this fall under?

Fomite 4y ago

To illustrate a couple problems with it being standard (importantly, note this paper was in a clinical epidemiology journal, so we are invariably talking about human health data): 1) Are you sure the data is deidentified? Can your analysis be done on the deidentified data? 2) Do you actually own the data? This is more complicated than "Well you published a paper on it...". The data is likely governed by data use agreements. Those are often extremely complex - especially at the international level. Or if you're working with particular populations - they're not wrong to be wary of who their data is given to, given the history of how science generally has treated them. 3) Is the data in a form that's genuinely suited to being shared? Is it in a flat file, with clear and easy to understand variable names, a good data dictionary, etc.? Is missingness, and the reasons why, clear and evident (the difference between a variable with a lot of missingness and a variable that's a bit garbage is often subtle)? 4) Where do you put it - and importantly, who pays for that, and how long?