3 ms·
> The data versioning system enables us to run pipelines based on changes to your data; the pipelines declare what files they read, and that allows us to schedu
by ilyt 4y ago
> The data versioning system enables us to run pipelines based on changes to your data; the pipelines declare what files they read, and that allows us to schedule processing jobs that only reprocess new or changed data, while still giving you a full view of what "would" have happened if all the data had been reprocessed. This, to me, is the key advantage of data versioning; you can save hundreds of thousands of dollars on compute. Being able to undo an oopsie is just icing on the cake.
...isn't that just parsing git diff --name-only A..B tho ? "Process only files that changed since last commit" is extremely simple problem to solve.
- jrockway 4y agoYeah, that's the basics of it. Just make sure all the output is atomic, scale up the workers, handle inputs that are joins, retry when the workers get rescheduled, etc.