5 ms·
What would you use a "git for data" for?
by arxpoetica 7y ago
What would you use a "git for data" for?
- mhh__ 7y agoData? Although having said then when you say data to me, I do imagine some level of immutability. Logging scientific results using git would be pretty good if required, in the sense that there is a habit to just use folders and text files which is fine at the time but is really really hard to take over sometimes (Like code written by scientists with their single-letter variable names and hatred of functions, in my experience of Fortran - yuck)
- arxpoetica 7y agoI think I meant more in terms of use cases. So it sounds like one of your use cases is logging scientific results...?
- DocSavage 7y agoWe’ve released our branched versioning system for data at http://dvid.io http://dvid.io. It’s entirely open source and uses a science HTTP API with pluggable data types and a versioned key-value backend. I’m currently developing a new backend for it: DAGStore, a lower-level ordered key-value store that will have an explicit immutable store and is tailored for distributed versioning of big data.
- gumby 7y agoRecording outputs of analytical runs or deep learning models; the ability to say ‘this is the output of program X v <git hash> run on the output of program Y <git hash> whose source data was Z <data-git hash>” for all the reasons of repeatability and audit ability needed for actual science.