3 ms·

I feel like most of the problems you list come from trying to version control notebooks, which may just be a futile endeavor. I primarily use notebooks as a wo

I feel like most of the problems you list come from trying to version control notebooks, which may just be a futile endeavor. I primarily use notebooks as a worklog for explorative analysis. I can use them to track down the history of experiments and thought processes that led me to those experiments, to help me avoid treading over similar waters and to keep a single cognitive thread throughout the whole process. And if any idea is worth building actual software for, it then becomes a good reference for those projects. The crucial idea here is to NOT change the notebook over time, just add to it. Then there are no versioning issues. And when it gets too large to navigate efficiently, just break it apart and continue the thread in the next notebook. Of course in a collaborative environment this kind of thing is less useful, but for keeping track of individual work it can still be effective.

mlthoughts2018 8y ago
I am sympathetic to your perspective, especially because it's basically what I thought when I was in grad school too, and I totally know where you are coming from. But I wanted to highlight what I think are reasonable counterpoints to some of what you wrote: > "I can use them to track down the history of experiments and thought processes that led me to those experiments" But this is exactly what you can't do with notebooks, mainly because they are not well-suited for review. This is true even when working alone, but is amplified when working in a team because even experimental choices require code review. For example, I worked on a problem once where we needed to train a neural network for age prediction. Someone on the team very familiar with how to write this using Keras began writing up an ad hoc training script, meant to be a "playground" for changing parameters, re-running analysis, etc. But through code review, even before any experiment was run, we detected bugs in some of the code, mistakes in how the data was being loaded and data cleaning steps, and assumptions that had been made about parameters for less frequently used activation function parameters. By having extra reviewers look at it and review the intended experiment itself, as well as the code to implement it we saved a lot of time, gained from diversity of opinions about how to handle it, avoided needing to debug incorrect diagnostic or accuracy charts to backtrack to these bugs later. Code review doesn't always solve all of these problems, much the same way that it doesn't solve all the problems of straight software engineering. But it is clearly such a valuable tool that it almost always should be a baked-in, mandatory part of the process, even for greenfield experimental code. And the notebook makes this so hard that people don't do it, and when they do it, they don't benefit much from it. > "The crucial idea here is to NOT change the notebook over time, just add to it. Then there are no versioning issues." I wish it were that easy, and that effectively "undo" resolved all versioning problems, but that is extremely rare. In many cases, you need to run the same analysis notebook with a variety of different parameters, and each one corresponds to a separate experiment. Doing this by manually tweaking parameters, hoping you didn't make a typo, and then storing the output artifacts in a file called "run7_alpha3.0_learningrate_0.001_04242018.tar.gz" leads to awful problems of reproducibility. The other thing is that even if you only add to a notebook, you might be adding the use of new third-party libraries, or adding code that assumes some version of something has been implicitly upgraded in the background. So you'd still need a ground truth source of all your dependencies, like requirements.txt for Python, with possibly more information about the Python version itself (does your notebook code rely on stably ordered dictionaries starting in Python 3.6? How would someone coming to the notebook know that if they are using 3.5 and the results seem funny for no discernible reason?) -- and in general you'll need some overall environment management setup to ensure, for reproducibility, that other people can download the same data, setup the same language environment, get the right versions of the required libraries, and have other system dependencies if there are any (maybe they need a certain version of CUDNN installed? Or maybe you used TensorFlow built from source with extra optimization flags enabled?) The idea is to take a good "project hygiene" and software craftsmanship approach out of the box, to avoid situations when someone says, "why is this notebook broken on Bob's laptop? It works on Alice's Ubuntu server...". And to always design for code review and testability from the start, because they are huge productivity enhancers and the benefits compound over time. Even for research-oriented prototypes.

mlthoughts2018 8y ago

I am sympathetic to your perspective, especially because it's basically what I thought when I was in grad school too, and I totally know where you are coming from. But I wanted to highlight what I think are reasonable counterpoints to some of what you wrote: > "I can use them to track down the history of experiments and thought processes that led me to those experiments" But this is exactly what you can't do with notebooks, mainly because they are not well-suited for review. This is true even when working alone, but is amplified when working in a team because even experimental choices require code review. For example, I worked on a problem once where we needed to train a neural network for age prediction. Someone on the team very familiar with how to write this using Keras began writing up an ad hoc training script, meant to be a "playground" for changing parameters, re-running analysis, etc. But through code review, even before any experiment was run, we detected bugs in some of the code, mistakes in how the data was being loaded and data cleaning steps, and assumptions that had been made about parameters for less frequently used activation function parameters. By having extra reviewers look at it and review the intended experiment itself, as well as the code to implement it we saved a lot of time, gained from diversity of opinions about how to handle it, avoided needing to debug incorrect diagnostic or accuracy charts to backtrack to these bugs later. Code review doesn't always solve all of these problems, much the same way that it doesn't solve all the problems of straight software engineering. But it is clearly such a valuable tool that it almost always should be a baked-in, mandatory part of the process, even for greenfield experimental code. And the notebook makes this so hard that people don't do it, and when they do it, they don't benefit much from it. > "The crucial idea here is to NOT change the notebook over time, just add to it. Then there are no versioning issues." I wish it were that easy, and that effectively "undo" resolved all versioning problems, but that is extremely rare. In many cases, you need to run the same analysis notebook with a variety of different parameters, and each one corresponds to a separate experiment. Doing this by manually tweaking parameters, hoping you didn't make a typo, and then storing the output artifacts in a file called "run7_alpha3.0_learningrate_0.001_04242018.tar.gz" leads to awful problems of reproducibility. The other thing is that even if you only add to a notebook, you might be adding the use of new third-party libraries, or adding code that assumes some version of something has been implicitly upgraded in the background. So you'd still need a ground truth source of all your dependencies, like requirements.txt for Python, with possibly more information about the Python version itself (does your notebook code rely on stably ordered dictionaries starting in Python 3.6? How would someone coming to the notebook know that if they are using 3.5 and the results seem funny for no discernible reason?) -- and in general you'll need some overall environment management setup to ensure, for reproducibility, that other people can download the same data, setup the same language environment, get the right versions of the required libraries, and have other system dependencies if there are any (maybe they need a certain version of CUDNN installed? Or maybe you used TensorFlow built from source with extra optimization flags enabled?) The idea is to take a good "project hygiene" and software craftsmanship approach out of the box, to avoid situations when someone says, "why is this notebook broken on Bob's laptop? It works on Alice's Ubuntu server...". And to always design for code review and testability from the start, because they are huge productivity enhancers and the benefits compound over time. Even for research-oriented prototypes.