3 ms·

No mention of git. If it's not on the global information tracker, it ain't science. There's no excuse anymore. Let me repeat because this is important. If you

by breck 6y ago

No mention of git. If it's not on the global information tracker, it ain't science. There's no excuse anymore. Let me repeat because this is important. If you aren't putting your research in Git, and publishing the repo, and are instead doing things like PDFs and Docs and private datasets, please don't call yourself scientist (exception: "bench" folks who are grinding out the data and not too involved in the analysis/publishing process). That was acceptable 50 years ago, but the environment has changed. That is not acceptable anymore. I don't care if that's not how academia does it. It's so obviously the right thing to do.

vngzs 6y ago
You have the right idea, but an unhelpful perspective. Certainly not everyone who isn't a software developer is a "phony scientist!" If you'd like to convince scientists to come around to your line of thinking, explain the problems with current research papers. Tell them about the problems with publishing papers only, how others struggle to reproduce their research, and how it leaves much bad research that would be invalidated far sooner except that the information to recreate their trial was absent. Then explain how sharing the code and data with Git (or similar) can solve this problem. You'll find that you are much more persuasive.
- breck 6y ago
  I agree with you that this isn't a persuasive approach. But I'm cutting to the chase. If anyone is working on this—helping onboard academics and non-programmers to git—this is an idea I'm looking to support (as an investor, donor, or whatever).
  - a_conservative 6y ago
    Isn't git an implementation detail here? Git repos don't have to be publicly available. They also don't have to contain history or multiple versions at all. I'm guessing what you really want is: 1) open formats for data 2) plain text instead of proprietary blobs 3) the ability to replicate calculations easily 4) the ability to see some sort of evolution or mistake-fixing in the project's history.
    - breck 6y ago
      Git is an implementation detail in the way x86 in the 2000's was an implementation detail. Of course in theory what I'm talking about doesn't have to be git, it could be another extremely well engineered, capable, reliable, trustworthy, auditable, fast, ubiquitous version control system. But in practice that means Git and there is nothing on my forecasts that will come close to dethroning it in the next 10 years. I have seen 50 startups that have pitched building a shittier version of git for academics because they can't be bothered to spend time reading the internals of git and have no idea how well done it is.
      - a_conservative 6y ago
        > I have seen 50 startups that have pitched building a shittier version of git for academics because they can't be bothered to spend time reading the internals of git and have no idea how well done it is. I suspect git would make a fine back end for a more polished product for scientists. A github for scientists sounds very interesting.
        breck 6y ago
        > I suspect git would make a fine back end for a more polished product for scientists. Agreed! That could be a winning idea.
    - webmaven 6y ago
      > 4) the ability to see some sort of evolution or mistake-fixing in the project's history. Rather than 'mistake fixing', what I want out of a project's history is evidence that the author hasn't done some form of p-hacking. But that purpose can be served just as well by preregistration. https://en.m.wikipedia.org/wiki/Preregistration_(science) https://en.m.wikipedia.org/wiki/Preregistration_(science)
the_snooze 6y ago
The reality is much more nuanced than that, especially when you're dealing with sensitive human-subjects data. Confidentiality and consent are important factors to consider. Let's say you're doing work on how abusive romantic partners use technology to keep tabs on their victims [1]. Would it be reasonable to expect the raw recordings, transcripts, and recruitment process be made publicly available? [1] https://dl.acm.org/doi/10.1145/3173574.3174241 https://dl.acm.org/doi/10.1145/3173574.3174241
- breck 6y ago
  Synthetic data. Very easy. This example is a little clunky, bc was ~2 years ago, but for a GWAS study we did on Early-Onset Preeclampsia where we had clinical and genomic data for 109 pregnant woman in Hawai'i, we put everything on GitHub. [1] We created a grammar for the schema, which not only allowed us to double check the type safety of our data but then also synthesize a CSV [2] with one click and stick that in the repo so that reviewers/researchers could `git clone` the repo and reproduce everything without any permissions bottlenecks (obviously the results are gibberish with the synth data, and you have to contact us to get the real data, but this way you can get the code working and then it's just a simple file move to reproduce the full results). This is the way. [1] https://github.com/breckuh/eopegwas/blob/master/eopClinicalSsv.grammar https://github.com/breckuh/eopegwas/blob/master/eopClinicalS... [2] https://github.com/breckuh/eopegwas/blob/master/mockData/clinical.csv https://github.com/breckuh/eopegwas/blob/master/mockData/cli...

by breck 6y ago

vngzs 6y ago

You have the right idea, but an unhelpful perspective. Certainly not everyone who isn't a software developer is a "phony scientist!" If you'd like to convince scientists to come around to your line of thinking, explain the problems with current research papers. Tell them about the problems with publishing papers only, how others struggle to reproduce their research, and how it leaves much bad research that would be invalidated far sooner except that the information to recreate their trial was absent. Then explain how sharing the code and data with Git (or similar) can solve this problem. You'll find that you are much more persuasive.

breck 6y ago
I agree with you that this isn't a persuasive approach. But I'm cutting to the chase. If anyone is working on this—helping onboard academics and non-programmers to git—this is an idea I'm looking to support (as an investor, donor, or whatever).
- a_conservative 6y ago
  Isn't git an implementation detail here? Git repos don't have to be publicly available. They also don't have to contain history or multiple versions at all. I'm guessing what you really want is: 1) open formats for data 2) plain text instead of proprietary blobs 3) the ability to replicate calculations easily 4) the ability to see some sort of evolution or mistake-fixing in the project's history.
  - breck 6y ago
    Git is an implementation detail in the way x86 in the 2000's was an implementation detail. Of course in theory what I'm talking about doesn't have to be git, it could be another extremely well engineered, capable, reliable, trustworthy, auditable, fast, ubiquitous version control system. But in practice that means Git and there is nothing on my forecasts that will come close to dethroning it in the next 10 years. I have seen 50 startups that have pitched building a shittier version of git for academics because they can't be bothered to spend time reading the internals of git and have no idea how well done it is.
    - a_conservative 6y ago
      > I have seen 50 startups that have pitched building a shittier version of git for academics because they can't be bothered to spend time reading the internals of git and have no idea how well done it is. I suspect git would make a fine back end for a more polished product for scientists. A github for scientists sounds very interesting.
      - breck 6y ago
        > I suspect git would make a fine back end for a more polished product for scientists. Agreed! That could be a winning idea.
  - webmaven 6y ago
    > 4) the ability to see some sort of evolution or mistake-fixing in the project's history. Rather than 'mistake fixing', what I want out of a project's history is evidence that the author hasn't done some form of p-hacking. But that purpose can be served just as well by preregistration. https://en.m.wikipedia.org/wiki/Preregistration_(science) https://en.m.wikipedia.org/wiki/Preregistration_(science)

the_snooze 6y ago

The reality is much more nuanced than that, especially when you're dealing with sensitive human-subjects data. Confidentiality and consent are important factors to consider. Let's say you're doing work on how abusive romantic partners use technology to keep tabs on their victims [1]. Would it be reasonable to expect the raw recordings, transcripts, and recruitment process be made publicly available? [1] https://dl.acm.org/doi/10.1145/3173574.3174241 https://dl.acm.org/doi/10.1145/3173574.3174241

breck 6y ago
Synthetic data. Very easy. This example is a little clunky, bc was ~2 years ago, but for a GWAS study we did on Early-Onset Preeclampsia where we had clinical and genomic data for 109 pregnant woman in Hawai'i, we put everything on GitHub. [1] We created a grammar for the schema, which not only allowed us to double check the type safety of our data but then also synthesize a CSV [2] with one click and stick that in the repo so that reviewers/researchers could `git clone` the repo and reproduce everything without any permissions bottlenecks (obviously the results are gibberish with the synth data, and you have to contact us to get the real data, but this way you can get the code working and then it's just a simple file move to reproduce the full results). This is the way. [1] https://github.com/breckuh/eopegwas/blob/master/eopClinicalSsv.grammar https://github.com/breckuh/eopegwas/blob/master/eopClinicalS... [2] https://github.com/breckuh/eopegwas/blob/master/mockData/clinical.csv https://github.com/breckuh/eopegwas/blob/master/mockData/cli...