6 ms·

If interested, I spent some time on a comment thread a few days ago describing how my experience leads me to believe the Notebook environment (not all of Jupyte

by mlthoughts2018 8y ago

If interested, I spent some time on a comment thread a few days ago describing how my experience leads me to believe the Notebook environment (not all of Jupyter / IPython, just the Notebook part) is actually only appropriate for a tiny subset of pedagogical or throw-away situations, and should be avoided most of the time and avoided in most of the cases it’s marketed for (especially anything having to do with ‘reproducibility’ or ‘exploratory analysis’). < https://news.ycombinator.com/item?id=17202704 https://news.ycombinator.com/item?id=17202704 >

bunderbunder 8y ago
You're not wrong, though I think the value of pedagogical and throw-away situations in your work may not be a universal experience. For me, at least, pedagogic and throw-away situations aren't a tiny subset. They're most of what I do. It's exploratory work, figuring out how the data behaves, if the data behaves, where it needs to be cleaned, churning through great heaps of experiments and iterations before hitting on the ultimate plan, and putting together a presentation to help explain what I finally settled on to colleagues and stakeholders. Only after sinking a whole lot of sweat into that process do I go on to start building anything that we intend to keep. At which point, forget Jupyter notebooks, I'm typically not even working in Python anymore for that part of the job.
- na85 8y ago
  What are you working in at that later part of the job?
- wenc 8y ago
  > At which point, forget Jupyter notebooks, I'm typically not even working in Python anymore for that part of the job. This is what is typically done out there but I suggest it breaks the feedback loop between the scientist roles and the developer roles. In rapidly changing environments those feedback loops could be crucial. It's similar to what Wall Street folks did (still do?)--quants write models in Excel/VBA and pass them over to developers who would rewrite them in Java for production. There's a natural impedance mismatch, and back-and-forths are difficult. I think a better approach would be for data scientists to write somewhat production-ready code, send it to prod (with the help of devs), get feedback from the production environment as well as get a sense of what tricks are needed for prod, and then iterate on that code. It also helps to remove the insulation between data scientists and the real world.
  - bunderbunder 8y ago
    Well, emphasis on the pronouns there. I'm not doing the proverbial "throw it over the wall to engineering", I'm also writing the production version. I also dislike the "2 teams" approach. Even if you have separate roles for data scientists and software engineers, better to mix them onto a single team than force them to communicate across a partition. For me it's really down to efficiency. Writing somewhat production-ready code is more expensive and time-consuming than blithely hacking. In the early stages of a new project, I know that almost everything I'm doing will get thrown away. For the most interesting projects, there's even a decent chance that it will be a complete failure and everything gets thrown away. So, at that stage in the game, I'm inclined to say that any extra effort spent on production readiness is just a waste of time and money. Fail fast, YAGNI, etc.
    - wenc 8y ago
      Oh I understand, I'm one of the few people on my team who does devops + data engineering + data science (some people on my team only do 1 or 2, but not all 3). My point is more about the impedance mismatch between roles and code produced by each role, whether or not they are carried out by the same person. For instance, I find it difficult iterating between my own model code and production code, especially if the model code was conceived in an interactive notebook environment. I do agree that notebooks are good for writing throwaway code, but of n failed notebooks, typically there's one that we'd like to bring to production. That's typically the one notebook we'd want to be production ready. When I say production-readiness, I don't mean actually working in production boilerplate in the first iteration (maybe in later iterations...). I mean writing the code in a way that lends itself to easy productionization through observance of certain constraints, e.g. being cognizant of environment/scoping/global state/namespace conflicts, writing model code in modular units (functions or classes depending on the use case) rather than just imperative line-by-line code, etc. These tiny disciplines are almost effortless but can lower the friction of iterating between model and production. In data science work, the real proof of the pudding is in production, not in unit tests. Most people don't want to admit this but unit testing doesn't work as well in the mathematical modeling world as they do in the software development world -- much of the time our inputs aren't discrete/enumerable, and the state-space is large or infinite. So it's really important to be able to iterate between production and modeling. If I ever need to go back to my interactive environment to experiment and change the logic, there should be an easy path to flow that back into production. Right now notebook environments don't aid in that. I've observed OTOH that IDE environments do.
- mlthoughts2018 8y ago
  > “It's exploratory work, figuring out how the data behaves, if the data behaves, where it needs to be cleaned, churning through great heaps of experiments and iterations before hitting on the ultimate plan” The thing is though, you should be involving code reviewers even at this stage, to review both the statistical methodology you intend for your experiments, and also the source code you believe implements that methodology. (Even when working alone, but absolutely when part of a team). Instead of seeing the notebook as a big series of scratch-pad attempts to get something right, you should be using pull requests and code review as that scratch pad. Additionally, the functions, classes and modules you create to do the work of exploring data fidelity, cleaning pre-treatments, or parameter sweeps through sets of experiment-specific parameter bundles — all that should be written like proper, testable, well-designed code, that lives in separate libraries or packages to facilitate re-using it without reinventing the wheel or copy/pasting from some old notebook, etc. By that point, the notebook you’d use to explore data behavior or to invoke distributed training across a bunch of parameter values would be a tiny notebook that just imports everything it needs from properly maintained helper libraries you wrote. And the value of the notebook over the same code just living in an easy-to-review script starts to be extremely questionable.
wenc 8y ago
Agreed. Notebook environments are great for exploration, discovery and pedagogy. They aren't so good for productionizing code. We found this out the hard way when we tried to productionize ML code in Jupyter. We had to export to .py and add boilerplate. This works fine unless there is back and forth iteration between modeling and prod, which there invariably is; our data scientists had to make changes to the notebook and we had to redo our boilerplate, so the notebook code and production code were constantly out of sync. This could have been alleviated with automation -- but such automation is bespoke and hard to generalize. PyCharm has a Scientific Mode (similar to RStudio's IDE approach, where you are actually writing code in a text file but are able to statefully/interactively run code by pressing Ctrl-Enter on code blocks). Spyder, Matlab and a bunch of other IDEs implement this idea too. Unlike notebooks, this is, I feel, a good middle ground between interactive exploration and having production-ready code.
- mlthoughts2018 8y ago
  On that console / IDE point you made, IPython can still be quite good for that if you use the interactive shell. For example, I might make two shell tabs in tmux, and make one a small rectangle towards the bottom of the screen (holds my running IPython session), and a large rectangle above it (holds my Emacs where I’m editing source code). And I might have a third shell tab somewhere that detects any time source files are changed and re-runs unit tests.
  - wenc 8y ago
    Yes, that's definitely a possibility. Though it would be nice if it were more nicely integrated into an environment like RStudio where you can interactively set breakpoints, watch variables, etc. while still maintaining the interactivity. I do the tmux/vim too, but for exploratory work the experience is less well-integrated than it could be with an Rstudio-like IDE.
    - mlthoughts2018 8y ago
      I agree, and the IDE setups can be very valuable for certain use cases or certain preferences. The equivalent thing in the IPython shell approach, basically using souped-up pudb, is not quite as nicely interactive with the debugging cycle, since setting breakpoints, watchpoints, etc., is either a matter of editing them into the source code and re-running, or becoming a master of specifying them on the command line, both of which require stepping out of the tight iteration workflow slightly (but to be fair, they also offer more power than the preconfigured options availabe in the IDE debugger features).
joshuamorton 8y ago
I think most of the complaints you outline can be solved by 1. Having tooling support for source controlling a notebook without all the garbage generated json causing issues. 2. Having tooling support for testing 3. Having style and review guidelines for notebooked code In other words, treat a notebook as a first class citizen, and as a legitimate part of the workflow, and it stops feeling like a crufty, unreproducible, hacky mess.
- mlthoughts2018 8y ago
  1. and 2. could alleviate many of the problems, but people overlook how incompatible 1. and 2. are with any basic approach to testing. For example, notebooks inherently intermix units of display logic with units of implementation logic, but obviously these are separate concerns (in the spirit of e.g. Model-View-Controller), and you shouldn’t be writing “a module” (the notebook) that intermixes them & requires wacky coupling of display concerns for implementation questions. (This also extends to the formatting of non-code aspects of notebooks too, which should be reviewed as a separate concern... much like how in LaTeX, what content I write is a different concern than how it renders). Imagine the added strain on e.g. a bunch of pytest code you’ve already written for an underlying implementation library if you now require it to be used for also automatically testing display logic too. Finally, item 3. is also tricky because the point of a notebook is presentation, so the style guide might rightfully be different. Now if you have some implementation unit (some code block, function body, whatever) it’s suddenly a debate whether it should be styled for presentation or styled according to the team’s source code guidelines .. and you’d need more linting tools that can be surgically used on subsets of the notebook, which seems needlessly complicated in comparison to just factoring out units of implementation logic into a separate helper module in the first place. For these reasons, it is actually quite hard to treat a notebook as a first-class citizen in any sense beyond mere convention.. which is useless in the automated testing and review side of the issue.
  - joshuamorton 8y ago
    Generally speaking, the code you write in a notebook isn't too important for presentation, the code you write in a notebook creates something to be presented (a chart, an output value, a pandas dataframe). I'll grant that this isn't true 100% of the time, but I find its normally the case. So start with the assumption that you have magic tooling (which I know exists) that allows you to ignore diffs in generated output as far as code review and committing is concerned. The impl doesn't really matter. Then you also have magical tooling that allows you to write unit tests at the cell level, either treating a cell as a function container, and testing the single function within or treating a cell an "hermetic" function itself, where you configure globals, run the cell in the context of those globals, and see what the globals look like afterwords. That's why I'm saying you need dedicated tooling support. And you're still thinking in the realm of "oh I test this normally". No no, I mean truly dedicated libraries that hook into the notebook environment itself. You might structure your tests as pytests, but under the hood they're spinning up a notebook env and doing magic. >units of implementation logic into a separate helper module in the first place. Well sure, you need this anyway though. Any kind of shared infrastructure should be factored out into a module, not because of the notebook environment, but because that's good abstraction independent of how you're presenting your models.

bunderbunder 8y ago

You're not wrong, though I think the value of pedagogical and throw-away situations in your work may not be a universal experience. For me, at least, pedagogic and throw-away situations aren't a tiny subset. They're most of what I do. It's exploratory work, figuring out how the data behaves, if the data behaves, where it needs to be cleaned, churning through great heaps of experiments and iterations before hitting on the ultimate plan, and putting together a presentation to help explain what I finally settled on to colleagues and stakeholders. Only after sinking a whole lot of sweat into that process do I go on to start building anything that we intend to keep. At which point, forget Jupyter notebooks, I'm typically not even working in Python anymore for that part of the job.

na85 8y ago
What are you working in at that later part of the job?
wenc 8y ago
> At which point, forget Jupyter notebooks, I'm typically not even working in Python anymore for that part of the job. This is what is typically done out there but I suggest it breaks the feedback loop between the scientist roles and the developer roles. In rapidly changing environments those feedback loops could be crucial. It's similar to what Wall Street folks did (still do?)--quants write models in Excel/VBA and pass them over to developers who would rewrite them in Java for production. There's a natural impedance mismatch, and back-and-forths are difficult. I think a better approach would be for data scientists to write somewhat production-ready code, send it to prod (with the help of devs), get feedback from the production environment as well as get a sense of what tricks are needed for prod, and then iterate on that code. It also helps to remove the insulation between data scientists and the real world.
- bunderbunder 8y ago
  Well, emphasis on the pronouns there. I'm not doing the proverbial "throw it over the wall to engineering", I'm also writing the production version. I also dislike the "2 teams" approach. Even if you have separate roles for data scientists and software engineers, better to mix them onto a single team than force them to communicate across a partition. For me it's really down to efficiency. Writing somewhat production-ready code is more expensive and time-consuming than blithely hacking. In the early stages of a new project, I know that almost everything I'm doing will get thrown away. For the most interesting projects, there's even a decent chance that it will be a complete failure and everything gets thrown away. So, at that stage in the game, I'm inclined to say that any extra effort spent on production readiness is just a waste of time and money. Fail fast, YAGNI, etc.
  - wenc 8y ago
    Oh I understand, I'm one of the few people on my team who does devops + data engineering + data science (some people on my team only do 1 or 2, but not all 3). My point is more about the impedance mismatch between roles and code produced by each role, whether or not they are carried out by the same person. For instance, I find it difficult iterating between my own model code and production code, especially if the model code was conceived in an interactive notebook environment. I do agree that notebooks are good for writing throwaway code, but of n failed notebooks, typically there's one that we'd like to bring to production. That's typically the one notebook we'd want to be production ready. When I say production-readiness, I don't mean actually working in production boilerplate in the first iteration (maybe in later iterations...). I mean writing the code in a way that lends itself to easy productionization through observance of certain constraints, e.g. being cognizant of environment/scoping/global state/namespace conflicts, writing model code in modular units (functions or classes depending on the use case) rather than just imperative line-by-line code, etc. These tiny disciplines are almost effortless but can lower the friction of iterating between model and production. In data science work, the real proof of the pudding is in production, not in unit tests. Most people don't want to admit this but unit testing doesn't work as well in the mathematical modeling world as they do in the software development world -- much of the time our inputs aren't discrete/enumerable, and the state-space is large or infinite. So it's really important to be able to iterate between production and modeling. If I ever need to go back to my interactive environment to experiment and change the logic, there should be an easy path to flow that back into production. Right now notebook environments don't aid in that. I've observed OTOH that IDE environments do.
mlthoughts2018 8y ago
> “It's exploratory work, figuring out how the data behaves, if the data behaves, where it needs to be cleaned, churning through great heaps of experiments and iterations before hitting on the ultimate plan” The thing is though, you should be involving code reviewers even at this stage, to review both the statistical methodology you intend for your experiments, and also the source code you believe implements that methodology. (Even when working alone, but absolutely when part of a team). Instead of seeing the notebook as a big series of scratch-pad attempts to get something right, you should be using pull requests and code review as that scratch pad. Additionally, the functions, classes and modules you create to do the work of exploring data fidelity, cleaning pre-treatments, or parameter sweeps through sets of experiment-specific parameter bundles — all that should be written like proper, testable, well-designed code, that lives in separate libraries or packages to facilitate re-using it without reinventing the wheel or copy/pasting from some old notebook, etc. By that point, the notebook you’d use to explore data behavior or to invoke distributed training across a bunch of parameter values would be a tiny notebook that just imports everything it needs from properly maintained helper libraries you wrote. And the value of the notebook over the same code just living in an easy-to-review script starts to be extremely questionable.

wenc 8y ago

Agreed. Notebook environments are great for exploration, discovery and pedagogy. They aren't so good for productionizing code. We found this out the hard way when we tried to productionize ML code in Jupyter. We had to export to .py and add boilerplate. This works fine unless there is back and forth iteration between modeling and prod, which there invariably is; our data scientists had to make changes to the notebook and we had to redo our boilerplate, so the notebook code and production code were constantly out of sync. This could have been alleviated with automation -- but such automation is bespoke and hard to generalize. PyCharm has a Scientific Mode (similar to RStudio's IDE approach, where you are actually writing code in a text file but are able to statefully/interactively run code by pressing Ctrl-Enter on code blocks). Spyder, Matlab and a bunch of other IDEs implement this idea too. Unlike notebooks, this is, I feel, a good middle ground between interactive exploration and having production-ready code.

mlthoughts2018 8y ago
On that console / IDE point you made, IPython can still be quite good for that if you use the interactive shell. For example, I might make two shell tabs in tmux, and make one a small rectangle towards the bottom of the screen (holds my running IPython session), and a large rectangle above it (holds my Emacs where I’m editing source code). And I might have a third shell tab somewhere that detects any time source files are changed and re-runs unit tests.
- wenc 8y ago
  Yes, that's definitely a possibility. Though it would be nice if it were more nicely integrated into an environment like RStudio where you can interactively set breakpoints, watch variables, etc. while still maintaining the interactivity. I do the tmux/vim too, but for exploratory work the experience is less well-integrated than it could be with an Rstudio-like IDE.
  - mlthoughts2018 8y ago
    I agree, and the IDE setups can be very valuable for certain use cases or certain preferences. The equivalent thing in the IPython shell approach, basically using souped-up pudb, is not quite as nicely interactive with the debugging cycle, since setting breakpoints, watchpoints, etc., is either a matter of editing them into the source code and re-running, or becoming a master of specifying them on the command line, both of which require stepping out of the tight iteration workflow slightly (but to be fair, they also offer more power than the preconfigured options availabe in the IDE debugger features).

joshuamorton 8y ago

I think most of the complaints you outline can be solved by 1. Having tooling support for source controlling a notebook without all the garbage generated json causing issues. 2. Having tooling support for testing 3. Having style and review guidelines for notebooked code In other words, treat a notebook as a first class citizen, and as a legitimate part of the workflow, and it stops feeling like a crufty, unreproducible, hacky mess.

mlthoughts2018 8y ago
1. and 2. could alleviate many of the problems, but people overlook how incompatible 1. and 2. are with any basic approach to testing. For example, notebooks inherently intermix units of display logic with units of implementation logic, but obviously these are separate concerns (in the spirit of e.g. Model-View-Controller), and you shouldn’t be writing “a module” (the notebook) that intermixes them & requires wacky coupling of display concerns for implementation questions. (This also extends to the formatting of non-code aspects of notebooks too, which should be reviewed as a separate concern... much like how in LaTeX, what content I write is a different concern than how it renders). Imagine the added strain on e.g. a bunch of pytest code you’ve already written for an underlying implementation library if you now require it to be used for also automatically testing display logic too. Finally, item 3. is also tricky because the point of a notebook is presentation, so the style guide might rightfully be different. Now if you have some implementation unit (some code block, function body, whatever) it’s suddenly a debate whether it should be styled for presentation or styled according to the team’s source code guidelines .. and you’d need more linting tools that can be surgically used on subsets of the notebook, which seems needlessly complicated in comparison to just factoring out units of implementation logic into a separate helper module in the first place. For these reasons, it is actually quite hard to treat a notebook as a first-class citizen in any sense beyond mere convention.. which is useless in the automated testing and review side of the issue.
- joshuamorton 8y ago
  Generally speaking, the code you write in a notebook isn't too important for presentation, the code you write in a notebook creates something to be presented (a chart, an output value, a pandas dataframe). I'll grant that this isn't true 100% of the time, but I find its normally the case. So start with the assumption that you have magic tooling (which I know exists) that allows you to ignore diffs in generated output as far as code review and committing is concerned. The impl doesn't really matter. Then you also have magical tooling that allows you to write unit tests at the cell level, either treating a cell as a function container, and testing the single function within or treating a cell an "hermetic" function itself, where you configure globals, run the cell in the context of those globals, and see what the globals look like afterwords. That's why I'm saying you need dedicated tooling support. And you're still thinking in the realm of "oh I test this normally". No no, I mean truly dedicated libraries that hook into the notebook environment itself. You might structure your tests as pytests, but under the hood they're spinning up a notebook env and doing magic. >units of implementation logic into a separate helper module in the first place. Well sure, you need this anyway though. Any kind of shared infrastructure should be factored out into a module, not because of the notebook environment, but because that's good abstraction independent of how you're presenting your models.