3 ms·
I talked to Fernando about this at Neuroinformatics 2018. In principle generating dependencies from a notebook should be as easy as parsing the ast, finding all
by tgbugs 6y ago
I talked to Fernando about this at Neuroinformatics 2018. In principle generating dependencies from a notebook should be as easy as parsing the ast, finding all the import statements, and then using sys to lookup what package provided that module. In practice I stopped using notebooks entirely due to many of the issues described by pavlov above, though in my case it was mostly because it was at a point in time where the Gentoo python ecosystem was having a rough time dealing with the 2/3 transition and there was no minimal amount of work that could be done to get the jupyter server up and running.
As for the reason why this still hasn't happened? I'm guessing because it is because the people who care about reproducibility have found other ways to achieve it, and as a result have no need nor incentive to implement it as part of the notebook format. Also because, if someone is running notebooks at all they know enough to bootstrap the jupyter server, which is non-trivial, or they are using a managed environment (be it remote, conda, etc.) where they are blissfully unaware that all their code has massive implicit and undocumented context.
In the mean time I have implemented the start of an equivalent system for org files because the implicit environment is much worse given normal Emacs workflows. I have encountered some significant issues and considerations which the notebook ecosystem would have to deal with, and which the default python packaging ecosystem (or any ghettoized language specific packaging solution for that matter) is not equipped to handle.
The primary issues is that only language agnostic package managers can actually bootstrap the environment that is required. What do you do if someone doesn't have a toolchain to build numpy and they are on some strange arch where there are no wheels? Do you just fail? The 80% solution here is the one we have with conda or managed solutions. The 90% solution requires that all the language communities stop pretending that they are the only language in the world and stop wasting time on their bespoke package manager.
- Galanwe 6y ago> In principle generating dependencies from a notebook should be as easy as parsing the ast, finding all the import statements, and then using sys to lookup what package provided that module. Well not really. First, there is no link between a distribution package name (what you install with pip) and python package name (what you import). e.g. The distribution package "foobar" can install the python package "baz". Well, it's even worst than that. There could be name space packages so multiple distribution package actually install parts of a python package. And there could be monkey patching. Second, Python is not a static language. The import system is part of the interpretation of python code, and as such, is dynamic. You have absolutely 0 guarantee that "import foo" on two different environments, even if they have the same version of package foo, will produce the same results. There could be an infinite amount of magic done in foo's __init__ that depend on other libraries being installed, etc. Third, don't get me started on the actual version of these libraries.
- zelphirkalt 6y agoI agree on some of your points. I just want to add: Then comes the point, where one of your dependencies in conda simply wont install on the machine of your coworker or a windows machine of another coworker, or in your docker image and you are screwed, scrambling for the next package manager. Been there, experienced that. I can only recommend to people not to go that route and to not get into trying to get the same environment on windows. A lot of headaches. It will be simpler to tell people to use another OS, if they want to use the same environment. Ultimately I ended up providing a virtual machine. A managed solution -- Yes, this takes a lot of load off of the shoulders of people, who would like to actually get productive in other ways.