4 ms·
Definitely agree that that is one of the great points with Luigi. Airflow's UI of course blows everything out of the water IMO. As for organizing your data, my
by jaz46 10y ago
Definitely agree that that is one of the great points with Luigi. Airflow's UI of course blows everything out of the water IMO.
As for organizing your data, my personal and very biased opinion is that version control semantics similar to Git [0] are a pretty good way to help tame the complexity of ever-changing data sets. We already version code, but with versioned data too, now everything becomes completely reproducible.
[0] Our Data Science Bill of Rights: http://www.pachyderm.io/dsbor.html http://www.pachyderm.io/dsbor.html
- mtrn 10y agoThe git angle would be a huge step forward. What I found is that reproducability is not always on people's mind when they designing such systems, whereas I believe it's one of the most important properties. Pachyderm is on my TODO list for a while, so thanks for reminding me, I'll try to implement something real with it soon.