This is really interesting. Firstly, I wonder how fundamental notebooks have becoming to the preparation phase in the last couple of years (to avoid https://cac

This is really interesting. Firstly, I wonder how fundamental notebooks have becoming to the preparation phase in the last couple of years (to avoid https://cacm.acm.org/system/assets/0001/3683/long-filenames.jpg https://cacm.acm.org/system/assets/0001/3683/long-filenames....). It also strikes me that a lot of these problems in dissemination have been solved to a large degree in software development. I guess 'writing code everyday' and 'knowing how to architect systems and use software development principles' are getting increasingly less mutually exclusive, which lends itself well to further abstractions. Distribution is one of the things we deal with most frequently at https://nstack.com https://nstack.com. We let data scientists deploy their models into bit-for-bit reproducible modules which run in the cloud and can be composed with various data sources. Because these are semantically versioned and packaged on the server as bit-for-bit reproducible builds, this solves the fact that "one's own operating system and software inevitably get upgraded in some incompatible manner such that the original code no longer runs". Our background was in writing programming languages, so we took a lot of the ideas of type-safety and composition and applied them to this problem. Actually the most common way we see dissemination solved in enterprises is folks writing back to the warehouse and using Looker / Tableau to visualise, with a workflow like: import Redshift:0.1.0 as Redshift import CustomerModel:0.0.2 as Classifier def workflow = Schedule { interval = "Daily"} | Redshift.query("cookies.sql") | Classifier.predict | Redshift Looking forward to jumping into his http://pgbovine.net/cde.html http://pgbovine.net/cde.html project. We explored a ton of solutions here (including Nix), but ended up settling on btrfs.