Lately I have been looking into data science workflows and tools. One of the most promising ones I have seen is https://pachyderm.io https://pachyderm.io Data

Lately I have been looking into data science workflows and tools. One of the most promising ones I have seen is https://pachyderm.io https://pachyderm.io Data is stored and versioned on a custom docker-like filesystem. Processes are defined by yaml and run based on incoming new data to the filesystem. The processes can be versioned with the data and results, written into the fs are also versioned. It seems like a very good approach to many of the difficulties. I'm not affiliated with them, but vcs helped my programming so much that I am looking for the best applications of versioning to data management. Edit: this is mainly concerned with the center box, analysis and records. For provenance you get a consistent snapshot of the incoming data, but that data is fixed. For my work consistency and repeatability is more important than accuracy with respect to an external source.