5 ms·

> But if you need to continually be checking random samples of a dataframe, re-running modules, and seeing if your regular expressions correctly clean a dataset

> But if you need to continually be checking random samples of a dataframe, re-running modules, and seeing if your regular expressions correctly clean a dataset, you are using a notebook interface today, even if you bundle your code into a module at some point. Notebooks like jupyter and observable are not a good advance. The primary reason is that they are leading the community of data programmers away from git-based workflows. ("git" is a shorthand for whatever version-control here due to its dominance). Although it may seem like a strange claim, the primary reason "programmers" and "software engineers" use git is for correctness/debugging: the idea of not being to switch accurately between different states of your codebase to compare program behavior is insanity to a "programmer" / "software engineer". And it should be insanity also to data programmers. But due to the rise in popularity of notebooks, it is not. These things just do not encourage the use of version control to compare different states of the code. It's going to lead to many buggy analyses, and many data programmers missing the opportunity to learn how to write code to a high level of correctness.

jshen 5y ago
There is nothing inherent in notebooks that prevent the use of tools like git.
- nine_k 5y ago
  ...except the need to commit the current state to make things reproducible. And I'm afraid it's not just code in cells, but also the sequence of its execution, editing, execution again, etc, because the order of running the cells is arbitrary (whatever the user commands interactively), and the results of previous invocations form the dreaded arbitrarily mutable global state.
  - tomnipotent 5y ago
    > ...except the need to commit the current state to make things reproducible. This can definitely be a problem, but at least in the instances where I've worked with teams using Notebooks it hasn't been an issue. Usually data is either coming from networked storage (S3 etc.) as CSV/Parquet/HDF5, a database query with well-defined (enough) criterion, or API calls. > because the order of running the cells is arbitrary Most people I know have to restart their notebooks often enough to have to re-run all cells again anyway. You usually get into a good habit of making sure they're mostly "in order", or grouping common code into cells to minimize the number of cell executions and ctrl+enter's needed. > dreaded arbitrarily mutable global state It's definitely a double-edged sword, but the global state is a big part of what makes notebooks so useful. I have a repl-based environment, the ability to run arbitrary code, and global state to iteratively explore the domain of the problem I'm working on. That sword cuts me many times along the way, but they're (mostly) nicks. But it's usually worth the output, especially to the business. In all these instances, teams shared notebooks through git and most of the issues were simply understanding the context of their usage more than issues with their execution. I don't remember more than a few notebooks really being used by more than a few people anyway, so I've learned to consider them more as collaborative artifacts in an exploratory process to more mature, "packaged" code evolved from those notebooks.
  - beckingz 5y ago
    Are there any good workflows that depend on out of order execution outside the exploration phase? To use notebooks reproducibly, the whole thing should be cleared and run with some frequency. Global state shouldn't matter. There's also the nbconvert --to script jupyter function that can convert a notebook into a script.
  - xiphias2 5y ago
    For Jupyter sure, it's true, but not for Observable.
  - cxr 5y ago
    A couple of Brad Myers's students at CMU worked out a concept for lightweight versioning, with a particular focus on users of code notebooks. https://www.youtube.com/watch?v=rBrvT7vGleM https://www.youtube.com/watch?v=rBrvT7vGleM Unfortunately, they moved on to an evolution of the initial concept that's a lot less compelling/promising.
  - jshen 5y ago
    This has nothing to do with the ability to use a tool like git. Notebooks make sloppy practices easier, but all of these problems exist in non notebook code too. You have no idea how many times I’ve run into non notebook code that assumes a database or something is in a certain state with no clear explanation of how to get to that statement. Or, a library that requires a certain order of operations that is unclear.
beckingz 5y ago
This is a good point. On the other hand, developer productivity in notebook environments is so high that iterating your way to correct code may be faster than trying to plan it all out in the beginning. There is also the aspect of exploration. The hardest part of most data programmer projects is understanding the data and optimal wrangling.
- smohare 5y ago
  This is something I often wonder about. In my own experience the perceived productivity gains in prototyping often come with externalized technical debt that lead to an overall loss of productivity. Notebook-driven development tends to produce models that are too tightly coupled with an underlying data pipeline, as the workflow very much facilitates this. There’s a measured loss of productivity when a model and pipeline need to be decoupled, e.g., to split the pipeline or repurpose the model. Obviously coupling components is not unique to notebooks, but there just seems to be a culture in notebook-driven development to disproportionately push poorly designed code artifacts. The MLEs and data scientist I’ve worked with that write simple functional modules tend to introduce less technical debt, at the expense sometimes of taking longer to push out a demo prototype. All that said, I think the aforementioned issues are not truly intrinsic to a notebook-driven development, but rather an emerging culture that has rather aptly been called Potemkin data science: https://mcorrell.medium.com/potemkin-data-science-fba2b5ba5cc6 https://mcorrell.medium.com/potemkin-data-science-fba2b5ba5c...
- dash2 5y ago
  >On the other hand, developer productivity in notebook environments is so high that iterating your way to correct code may be faster than trying to plan it all out in the beginning. In science, what kills you isn't the mistake you see and correct. It's the mistake you don't see, that then gets published.
  - da39a3ee 5y ago
    Interesting, but not true. You clearly don't work in science: the main thing is getting publications. Mostly people don't pay that much attention to them, so if there are errors, (a) probably no one will notice and (b) you can always explain it away with some obfuscation.

jshen 5y ago

There is nothing inherent in notebooks that prevent the use of tools like git.

nine_k 5y ago
...except the need to commit the current state to make things reproducible. And I'm afraid it's not just code in cells, but also the sequence of its execution, editing, execution again, etc, because the order of running the cells is arbitrary (whatever the user commands interactively), and the results of previous invocations form the dreaded arbitrarily mutable global state.
- tomnipotent 5y ago
  > ...except the need to commit the current state to make things reproducible. This can definitely be a problem, but at least in the instances where I've worked with teams using Notebooks it hasn't been an issue. Usually data is either coming from networked storage (S3 etc.) as CSV/Parquet/HDF5, a database query with well-defined (enough) criterion, or API calls. > because the order of running the cells is arbitrary Most people I know have to restart their notebooks often enough to have to re-run all cells again anyway. You usually get into a good habit of making sure they're mostly "in order", or grouping common code into cells to minimize the number of cell executions and ctrl+enter's needed. > dreaded arbitrarily mutable global state It's definitely a double-edged sword, but the global state is a big part of what makes notebooks so useful. I have a repl-based environment, the ability to run arbitrary code, and global state to iteratively explore the domain of the problem I'm working on. That sword cuts me many times along the way, but they're (mostly) nicks. But it's usually worth the output, especially to the business. In all these instances, teams shared notebooks through git and most of the issues were simply understanding the context of their usage more than issues with their execution. I don't remember more than a few notebooks really being used by more than a few people anyway, so I've learned to consider them more as collaborative artifacts in an exploratory process to more mature, "packaged" code evolved from those notebooks.
- beckingz 5y ago
  Are there any good workflows that depend on out of order execution outside the exploration phase? To use notebooks reproducibly, the whole thing should be cleared and run with some frequency. Global state shouldn't matter. There's also the nbconvert --to script jupyter function that can convert a notebook into a script.
- xiphias2 5y ago
  For Jupyter sure, it's true, but not for Observable.
- cxr 5y ago
  A couple of Brad Myers's students at CMU worked out a concept for lightweight versioning, with a particular focus on users of code notebooks. https://www.youtube.com/watch?v=rBrvT7vGleM https://www.youtube.com/watch?v=rBrvT7vGleM Unfortunately, they moved on to an evolution of the initial concept that's a lot less compelling/promising.
- jshen 5y ago
  This has nothing to do with the ability to use a tool like git. Notebooks make sloppy practices easier, but all of these problems exist in non notebook code too. You have no idea how many times I’ve run into non notebook code that assumes a database or something is in a certain state with no clear explanation of how to get to that statement. Or, a library that requires a certain order of operations that is unclear.

beckingz 5y ago

This is a good point. On the other hand, developer productivity in notebook environments is so high that iterating your way to correct code may be faster than trying to plan it all out in the beginning. There is also the aspect of exploration. The hardest part of most data programmer projects is understanding the data and optimal wrangling.

smohare 5y ago
This is something I often wonder about. In my own experience the perceived productivity gains in prototyping often come with externalized technical debt that lead to an overall loss of productivity. Notebook-driven development tends to produce models that are too tightly coupled with an underlying data pipeline, as the workflow very much facilitates this. There’s a measured loss of productivity when a model and pipeline need to be decoupled, e.g., to split the pipeline or repurpose the model. Obviously coupling components is not unique to notebooks, but there just seems to be a culture in notebook-driven development to disproportionately push poorly designed code artifacts. The MLEs and data scientist I’ve worked with that write simple functional modules tend to introduce less technical debt, at the expense sometimes of taking longer to push out a demo prototype. All that said, I think the aforementioned issues are not truly intrinsic to a notebook-driven development, but rather an emerging culture that has rather aptly been called Potemkin data science: https://mcorrell.medium.com/potemkin-data-science-fba2b5ba5cc6 https://mcorrell.medium.com/potemkin-data-science-fba2b5ba5c...
dash2 5y ago
>On the other hand, developer productivity in notebook environments is so high that iterating your way to correct code may be faster than trying to plan it all out in the beginning. In science, what kills you isn't the mistake you see and correct. It's the mistake you don't see, that then gets published.
- da39a3ee 5y ago
  Interesting, but not true. You clearly don't work in science: the main thing is getting publications. Mostly people don't pay that much attention to them, so if there are errors, (a) probably no one will notice and (b) you can always explain it away with some obfuscation.