5 ms·

> We need something like a Github for scientific data where papers can reference data at a particular commit and then offer improvements through pull requests.

> We need something like a Github for scientific data where papers can reference data at a particular commit and then offer improvements through pull requests. exactly my thought, do you know any working group that is working toward that goal?

DocSavage 8y ago
If by working group you mean a cross-company collection of people, I don't know of any or I would've joined them :) I've been working toward that goal for the last 5 years, but primarily with an eye to our kinds of data problems in the Connectomics field. I've been meaning to look at RDA again but reluctant to start a working group myself.
- b_fiive 8y ago
  Hey DocSavage! I'm one of these Qri folks, I'd love to see that working group exist. I have a friend or two at the RDA. maybe we should get an email going on the subject? Projects like these are bigger than any one company or tool :)
  - DocSavage 8y ago
    Agreed. Will follow up on email through your Qri contact page.
    - b_fiive 8y ago
      delightful. thanks!
  - amirouche 8y ago
    I started a awesome list dubbed "awesome data distribution" feedback welcome at https://github.com/amirouche/awesome-data-distribution https://github.com/amirouche/awesome-data-distribution
- benhamner 8y ago
  Any way we can help at Kaggle? Is https://www.kaggle.com/datasets https://www.kaggle.com/datasets helpful for your work in connectomics?
brynb 8y ago
We’re building something along these lines at Axon (http://axon.science http://axon.science). Sign up for our beta if you’re interested in checking it out, and we should be able to get you set up in the next few days (we’re just starting to roll things out to the public this week). The basic idea is distributed version control, like git, but over p2p swarms rather than clusters around “central” repositories. We have special handling for large datasets (but still using git) to improve transfer efficiency and diffing. There’s a UI layer for collaboration (discussion, PRs, review) that supports deep linking to and embedding of files at specific commits, which sounds a bit like what you’re looking for. Feedback is very much appreciated!
- DocSavage 8y ago
  That looks very interesting, particularly the UI layer for collaboration. Your website says it supports “massive data sets” but I would spell out what you mean since data for different fields vary by several orders of magnitude. (Massive for me starts at TBs and goes to petabytes.) One of the issues for me is file-based versioning, which then requires the means to parse the format. A number of ventures and organizations (e.g., NeuroData without Borders) address versioning of the entire ecosystem necessary to correctly use the underlying data files, so not sure if that’s an explicit part of your ecosystem. Most importantly, is your stack going to be open source?
j88439h84 8y ago
http://Dvc.org http://Dvc.org does this
- DocSavage 8y ago
  What are the differences between Dvc and Pachyderm.io, which I should have mentioned earlier?
  - dmpetrov 8y ago
    "From a very high level perspective - Pachyderm is a data engineering tool designed with ML in mind, DVC.org is a tool to organize and version control an ML project. Probably, one way to think would be Spark/Hadoop or Airflow vs Git/Github." from https://news.ycombinator.com/item?id=19130499 https://news.ycombinator.com/item?id=19130499
smarx007 8y ago
Zenodo?
benhamner 8y ago
We're working on that through Kaggle Datasets https://www.kaggle.com/datasets https://www.kaggle.com/datasets We support data versioning, interactive web previews, seamless loading into hosted Jupyter notebooks (Kaggle Kernels), seeing/sharing analytic results built on the data version, and adding direct collaborators right now. We don't support a data-oriented version of an "issue" or a "pull request" quite yet, but these needs are definitely on our radar.
mbreese 8y ago
It's probably too late for this year, but ISMB is one of the traditional locations for such a working group in the biological sciences. It might be interesting for the meeting next year though. If anyone is interested in putting together a proposal, let me know. I'd be happy to help.

DocSavage 8y ago

If by working group you mean a cross-company collection of people, I don't know of any or I would've joined them :) I've been working toward that goal for the last 5 years, but primarily with an eye to our kinds of data problems in the Connectomics field. I've been meaning to look at RDA again but reluctant to start a working group myself.

b_fiive 8y ago
Hey DocSavage! I'm one of these Qri folks, I'd love to see that working group exist. I have a friend or two at the RDA. maybe we should get an email going on the subject? Projects like these are bigger than any one company or tool :)
- DocSavage 8y ago
  Agreed. Will follow up on email through your Qri contact page.
  - b_fiive 8y ago
    delightful. thanks!
- amirouche 8y ago
  I started a awesome list dubbed "awesome data distribution" feedback welcome at https://github.com/amirouche/awesome-data-distribution https://github.com/amirouche/awesome-data-distribution
benhamner 8y ago
Any way we can help at Kaggle? Is https://www.kaggle.com/datasets https://www.kaggle.com/datasets helpful for your work in connectomics?

brynb 8y ago

We’re building something along these lines at Axon (http://axon.science http://axon.science). Sign up for our beta if you’re interested in checking it out, and we should be able to get you set up in the next few days (we’re just starting to roll things out to the public this week). The basic idea is distributed version control, like git, but over p2p swarms rather than clusters around “central” repositories. We have special handling for large datasets (but still using git) to improve transfer efficiency and diffing. There’s a UI layer for collaboration (discussion, PRs, review) that supports deep linking to and embedding of files at specific commits, which sounds a bit like what you’re looking for. Feedback is very much appreciated!

DocSavage 8y ago
That looks very interesting, particularly the UI layer for collaboration. Your website says it supports “massive data sets” but I would spell out what you mean since data for different fields vary by several orders of magnitude. (Massive for me starts at TBs and goes to petabytes.) One of the issues for me is file-based versioning, which then requires the means to parse the format. A number of ventures and organizations (e.g., NeuroData without Borders) address versioning of the entire ecosystem necessary to correctly use the underlying data files, so not sure if that’s an explicit part of your ecosystem. Most importantly, is your stack going to be open source?

j88439h84 8y ago

http://Dvc.org http://Dvc.org does this

DocSavage 8y ago
What are the differences between Dvc and Pachyderm.io, which I should have mentioned earlier?
- dmpetrov 8y ago
  "From a very high level perspective - Pachyderm is a data engineering tool designed with ML in mind, DVC.org is a tool to organize and version control an ML project. Probably, one way to think would be Spark/Hadoop or Airflow vs Git/Github." from https://news.ycombinator.com/item?id=19130499 https://news.ycombinator.com/item?id=19130499

smarx007 8y ago

Zenodo?

benhamner 8y ago

We're working on that through Kaggle Datasets https://www.kaggle.com/datasets https://www.kaggle.com/datasets We support data versioning, interactive web previews, seamless loading into hosted Jupyter notebooks (Kaggle Kernels), seeing/sharing analytic results built on the data version, and adding direct collaborators right now. We don't support a data-oriented version of an "issue" or a "pull request" quite yet, but these needs are definitely on our radar.

mbreese 8y ago

It's probably too late for this year, but ISMB is one of the traditional locations for such a working group in the biological sciences. It might be interesting for the meeting next year though. If anyone is interested in putting together a proposal, let me know. I'd be happy to help.