3 ms·
Agree with the main thrust, but think that it misses a couple additional key costs: lack of reproducibility and lack of collaboration. Both of these are actuall
by fphhotchips 9y ago
Agree with the main thrust, but think that it misses a couple additional key costs: lack of reproducibility and lack of collaboration. Both of these are actually covered by the product that Domino sells, too!
Essentially, work done on laptops often doesn't make it past the laptop. Perhaps the modelling code makes it into Git, but the test data gets lost, the process that lead to it gets lost, etc. etc. Then reproducing and potentially modifying the work that has already been done becomes a labor and the work gets left behind or completely redone as a result. Let alone all the differences in environment and setup that can manifest between different laptops.
- dwhitena 9y agoReproducibility and provenance are essential for the data science workflows. And it's important to maintain these at scale. Take a look at Pachyderm some time as well if you get a chance (I work on the project for full disclosure). We version data in addition to code for full reproducibility. Even for distributed multi-stage and multi-language pipelines.
- deleted 9y ago[deleted]