3 ms·
He also suggests leveraging a programming language for benefits that include reproducibility, data provenance, and the ability to see how data analysis has evol
by nl 9y ago
He also suggests leveraging a programming language for benefits that include reproducibility, data provenance, and the ability to see how data analysis has evolved over time.
It's true that tracking data and data provenance is very important and hard in GUI tools.
But it's not really that much easier in code either.
To be specific about the kinds of problems here, I'm thinking of things like when errors are found in a dataset, new labels are introduced, or you want multiple splits on the data, but you still want the old version to check your metrics against.
Yes, you can do things like versioned directories for different data versions (although this tend to break when you are talking about TBs of data).
Or you can try using traditional version control tools, but that involves switching between code and your version control tool.
Or you can try transformation orientated programming, where you keep the original version of the data and then always transform it to get to the new version. This is slow on large data and fails when new information is introduced.
Also, normal version control doesn't work well with modelling code, because you want to use both the old and new versions of the code simultaneously.
Greg Brockman talked about this exact problem in the OpenAI/Ycombinator podcast.
This is a hard problem to solve - not sure who is working on it.