4 ms·

Rapid Prototyping of Interactive Data Science Workflows in Jupyter

10 points by wcrichton 8y ago

mlthoughts2018 8y ago
I’ve invested a lot of time into the IPython & Jupyter tools. In one job, I even wrote a 0MQ kernel to leverage ipython as a repl for a programming language created inside my company (which had horrible developer tooling). After working on several large machine learning and data science projects over the years, I’ve sadly come to the conclusion that the notebook environment is a very wrong approach to take for shareable, extensible or reproduceable systems. On my team now, aside from the tiniest, most ephemeral prototypes that can be thrown away entirely, nobody ever uses Jupyter notebooks for anything. The projects I’m referring to are very similar to those of this post too — an in-house image annotation tool we wrote with PyQT instead of notebook widgets, lots of interactive Flask web apps with simple React pages to demo and explore face detection results, diagnostic graphs, search engine output, and prototyping systems for layouts of image search results pages under different ML based ranking algorithms. The main reason the notebook caused problems is that it leads to poor software craftsmanship, which should be a consideration even from the earliest stages of models and prototypes (because it will make you produce higher quality output faster, not for any spiritual commitment to coding standards). When notebooks contain code you wouldn’t want merged into a given library or project’s main branch, it needs code review. And notebooks are awful units of work to review because they break models of automated testing, lack the same context for enforced style guidelines, and are generally written with a mix of priorities usually focused on the way the code author wants to think about the prototype, instead of separating these concerns into decoupled units. Through review, you quickly realize that anything that needs code reuse has to be using good design principles, separating concerns, organizing things into decoupled functions or possibly classes. When you factor these all out of the notebook and into a helper module (and write tests), then you see a bunch of the magic constants or initialized parameters that need to be factored out into a parameter file so that you can easily vary parameters, have it version controlled, and have a way to connect results artifacts (usually embedded plots or tables and saved data files) with parameter settings used to generate them. Then you realize you need to make the overall software environment also reproducible, because a notebook itself is never “reproduceable” apart from the overall software environment it was run in, and other people picking up your notebook would need the same data and possibly a Docker container or other container or VM to match the same software, libraries, network settings, port number, etc. And this process goes on until you realize you had to factor things into a tested module, automate the creation of the surrounding Docker container or other environment, extract out & document all your parameters, and so on. Until finally there is nothing left in the notebook but custom plotting code or data displays. And the display units could be e.g. a version controlled latex file that picks up image files or renders table templates with data, or a separate web app that uses good design to implement reusable video display or something, all called from a boring non-notebook launch script, all of which could be driven by a simple custom Makefile — making the whole thing just as interactive, generally more so, as the notebook which suffered all the problems. I fully agree this requires a team with good software craftsmanship skills. But anybody competent enough to write the original notebook can also learn these skills, and reorganize the work with just a few craftsmanship principles that almost immediately reveal the notebook as something that just gets in the way and slows you down. And for teams needing to make reproducible data studies or reproducible model training environments for real business situations, this effectively makes a notebook inappropriate most of the time. The main use cases where the notebook remains value-additive are situations where you can throw away the prototype at any time, it does not need any maintenance and it has no code or analysis within it that needs to be reused. This does happen for some kinds of tutorials, some pedagogical uses, and some totally ad hoc work, like slinging some code to whip together a quick answer for someone on the business team. The notebook can still be good for these cases. But these really represent a small number of use cases, certainly much smaller than the set of use cases the notebook is marketed towards. My experience, after being a zealot for IPython notebooks in the early days, is that notebooks are just way oversold, and they encourage thinking in a low-craftsmanship manner, and are best avoided as a general rule of thumb.
- wcrichton 8y ago
  Thanks for the time to write up this thoughtful reply. I agree with all your major points about maintaining/sharing notebooks. I don't work in the same collaborative setting (only 1 or 2 other people on the project), so I don't encounter the same code review issues, but I can certainly imagine. However, I do wonder--do you think notebooks are still valuable intermediate steps towards developing the code you eventually commit, even if the notebooks themselves never see the light of day, or do you think using them at all just leads towards lower-quality code and isn't worth the productivity (in practice)?
  - mlthoughts2018 8y ago
    Trying to use dispassionate code review is a good practice even when it’s a 1-person project. Using e.g. branches in GitHub as logical units of implementation is very important, since it allows you to test incremental changes, group changes into small, conceptually-related batches, and perform code review by inspecting the diff between the branch’s changes and the existing main implementation. The benefits of this are increased in multi-person collaborations, but it’s an important practice all the time, even when working alone on a small project. Notebooks, especially if you have to worry about artificial aspects of the diff (like whether you suppressed cell output) are a pain to work with in this type of code review workflow, even for just one person. The same is true for using automated testing. Many people don’t develop tests for code in a notebook because all along the process they imagine it’s all ad hoc, single-use code, and they arrive at components they wish to reuse in a fragmented and unplanned way. And since it can be hard to create unit tests for functions or cells from a notebook (which is not a module, the language’s standard for how to share an implementation), this means unless you manually convert to a regular source file, then testing is either very shallow (e.g. does some gross-level output of the notebook vaguely appear correct & bug-free?) or else very manually intensive (like manually executing a bunch of function calls or separate cells with different inputs to verify it doesn’t have a bug yourself). In terms of using a notebook as an integrated development environment, no, I don’t find that it’s useful for that and it also does not seem like one of the notebook’s goals (at least it’s design is antithetical to development environments). If I need to rapidly see interactive output, then the Jupyter interactive shell program (among many other interactive interpreters) would be fine, but the notebook (which forces cell-based interaction in a separate browser-driven environment) would not be. Just imagine comparing the developmemt power of using a non-windowed emacs or vim session to write code, then alt-tab to a shell with an interactive prompt to run that code. The productivity loop is much tighter and has less visual clutter than a browser notebook, while letting you simply do plain, boring text editing when writing the code (no cells, no embedded outputs, no need to remember what variable initialization or package imports have to be re-evaluated after you made a code change). And even better, testing can be automated, so you can just invoke a test command after some code changes to get feedback on whether it unexpectedly broke anything— letting you separate your design choices about comprehensiveness of testing from design choices of the code itself. For me, the notebook is like trying to use MS Word after you’ve already gotten past the learning curve of latex. Word feels restricting, visually cluttered, nasty to do proper version control or peer review, by comparison to even intermediate knowledge of latex. The same comparison is analogously useful for the notebook compared to using proper software approaches and shell-centric tools. Of course, Word has its use cases, especially for people who only occasionally write documents, or who write documents that rarely need collaborative edits or historical versioned rollbacks. But for people who are professionally writing documents, there are many much better tools. The same for the notebook.

10 points by wcrichton 8y ago

mlthoughts2018 8y ago

I’ve invested a lot of time into the IPython & Jupyter tools. In one job, I even wrote a 0MQ kernel to leverage ipython as a repl for a programming language created inside my company (which had horrible developer tooling). After working on several large machine learning and data science projects over the years, I’ve sadly come to the conclusion that the notebook environment is a very wrong approach to take for shareable, extensible or reproduceable systems. On my team now, aside from the tiniest, most ephemeral prototypes that can be thrown away entirely, nobody ever uses Jupyter notebooks for anything. The projects I’m referring to are very similar to those of this post too — an in-house image annotation tool we wrote with PyQT instead of notebook widgets, lots of interactive Flask web apps with simple React pages to demo and explore face detection results, diagnostic graphs, search engine output, and prototyping systems for layouts of image search results pages under different ML based ranking algorithms. The main reason the notebook caused problems is that it leads to poor software craftsmanship, which should be a consideration even from the earliest stages of models and prototypes (because it will make you produce higher quality output faster, not for any spiritual commitment to coding standards). When notebooks contain code you wouldn’t want merged into a given library or project’s main branch, it needs code review. And notebooks are awful units of work to review because they break models of automated testing, lack the same context for enforced style guidelines, and are generally written with a mix of priorities usually focused on the way the code author wants to think about the prototype, instead of separating these concerns into decoupled units. Through review, you quickly realize that anything that needs code reuse has to be using good design principles, separating concerns, organizing things into decoupled functions or possibly classes. When you factor these all out of the notebook and into a helper module (and write tests), then you see a bunch of the magic constants or initialized parameters that need to be factored out into a parameter file so that you can easily vary parameters, have it version controlled, and have a way to connect results artifacts (usually embedded plots or tables and saved data files) with parameter settings used to generate them. Then you realize you need to make the overall software environment also reproducible, because a notebook itself is never “reproduceable” apart from the overall software environment it was run in, and other people picking up your notebook would need the same data and possibly a Docker container or other container or VM to match the same software, libraries, network settings, port number, etc. And this process goes on until you realize you had to factor things into a tested module, automate the creation of the surrounding Docker container or other environment, extract out & document all your parameters, and so on. Until finally there is nothing left in the notebook but custom plotting code or data displays. And the display units could be e.g. a version controlled latex file that picks up image files or renders table templates with data, or a separate web app that uses good design to implement reusable video display or something, all called from a boring non-notebook launch script, all of which could be driven by a simple custom Makefile — making the whole thing just as interactive, generally more so, as the notebook which suffered all the problems. I fully agree this requires a team with good software craftsmanship skills. But anybody competent enough to write the original notebook can also learn these skills, and reorganize the work with just a few craftsmanship principles that almost immediately reveal the notebook as something that just gets in the way and slows you down. And for teams needing to make reproducible data studies or reproducible model training environments for real business situations, this effectively makes a notebook inappropriate most of the time. The main use cases where the notebook remains value-additive are situations where you can throw away the prototype at any time, it does not need any maintenance and it has no code or analysis within it that needs to be reused. This does happen for some kinds of tutorials, some pedagogical uses, and some totally ad hoc work, like slinging some code to whip together a quick answer for someone on the business team. The notebook can still be good for these cases. But these really represent a small number of use cases, certainly much smaller than the set of use cases the notebook is marketed towards. My experience, after being a zealot for IPython notebooks in the early days, is that notebooks are just way oversold, and they encourage thinking in a low-craftsmanship manner, and are best avoided as a general rule of thumb.

wcrichton 8y ago
Thanks for the time to write up this thoughtful reply. I agree with all your major points about maintaining/sharing notebooks. I don't work in the same collaborative setting (only 1 or 2 other people on the project), so I don't encounter the same code review issues, but I can certainly imagine. However, I do wonder--do you think notebooks are still valuable intermediate steps towards developing the code you eventually commit, even if the notebooks themselves never see the light of day, or do you think using them at all just leads towards lower-quality code and isn't worth the productivity (in practice)?
- mlthoughts2018 8y ago
  Trying to use dispassionate code review is a good practice even when it’s a 1-person project. Using e.g. branches in GitHub as logical units of implementation is very important, since it allows you to test incremental changes, group changes into small, conceptually-related batches, and perform code review by inspecting the diff between the branch’s changes and the existing main implementation. The benefits of this are increased in multi-person collaborations, but it’s an important practice all the time, even when working alone on a small project. Notebooks, especially if you have to worry about artificial aspects of the diff (like whether you suppressed cell output) are a pain to work with in this type of code review workflow, even for just one person. The same is true for using automated testing. Many people don’t develop tests for code in a notebook because all along the process they imagine it’s all ad hoc, single-use code, and they arrive at components they wish to reuse in a fragmented and unplanned way. And since it can be hard to create unit tests for functions or cells from a notebook (which is not a module, the language’s standard for how to share an implementation), this means unless you manually convert to a regular source file, then testing is either very shallow (e.g. does some gross-level output of the notebook vaguely appear correct & bug-free?) or else very manually intensive (like manually executing a bunch of function calls or separate cells with different inputs to verify it doesn’t have a bug yourself). In terms of using a notebook as an integrated development environment, no, I don’t find that it’s useful for that and it also does not seem like one of the notebook’s goals (at least it’s design is antithetical to development environments). If I need to rapidly see interactive output, then the Jupyter interactive shell program (among many other interactive interpreters) would be fine, but the notebook (which forces cell-based interaction in a separate browser-driven environment) would not be. Just imagine comparing the developmemt power of using a non-windowed emacs or vim session to write code, then alt-tab to a shell with an interactive prompt to run that code. The productivity loop is much tighter and has less visual clutter than a browser notebook, while letting you simply do plain, boring text editing when writing the code (no cells, no embedded outputs, no need to remember what variable initialization or package imports have to be re-evaluated after you made a code change). And even better, testing can be automated, so you can just invoke a test command after some code changes to get feedback on whether it unexpectedly broke anything— letting you separate your design choices about comprehensiveness of testing from design choices of the code itself. For me, the notebook is like trying to use MS Word after you’ve already gotten past the learning curve of latex. Word feels restricting, visually cluttered, nasty to do proper version control or peer review, by comparison to even intermediate knowledge of latex. The same comparison is analogously useful for the notebook compared to using proper software approaches and shell-centric tools. Of course, Word has its use cases, especially for people who only occasionally write documents, or who write documents that rarely need collaborative edits or historical versioned rollbacks. But for people who are professionally writing documents, there are many much better tools. The same for the notebook.