5 ms·

There is nothing inherent in notebooks that prevent the use of tools like git.

by jshen 5y ago

nine_k 5y ago
...except the need to commit the current state to make things reproducible. And I'm afraid it's not just code in cells, but also the sequence of its execution, editing, execution again, etc, because the order of running the cells is arbitrary (whatever the user commands interactively), and the results of previous invocations form the dreaded arbitrarily mutable global state.
- tomnipotent 5y ago
  > ...except the need to commit the current state to make things reproducible. This can definitely be a problem, but at least in the instances where I've worked with teams using Notebooks it hasn't been an issue. Usually data is either coming from networked storage (S3 etc.) as CSV/Parquet/HDF5, a database query with well-defined (enough) criterion, or API calls. > because the order of running the cells is arbitrary Most people I know have to restart their notebooks often enough to have to re-run all cells again anyway. You usually get into a good habit of making sure they're mostly "in order", or grouping common code into cells to minimize the number of cell executions and ctrl+enter's needed. > dreaded arbitrarily mutable global state It's definitely a double-edged sword, but the global state is a big part of what makes notebooks so useful. I have a repl-based environment, the ability to run arbitrary code, and global state to iteratively explore the domain of the problem I'm working on. That sword cuts me many times along the way, but they're (mostly) nicks. But it's usually worth the output, especially to the business. In all these instances, teams shared notebooks through git and most of the issues were simply understanding the context of their usage more than issues with their execution. I don't remember more than a few notebooks really being used by more than a few people anyway, so I've learned to consider them more as collaborative artifacts in an exploratory process to more mature, "packaged" code evolved from those notebooks.
- beckingz 5y ago
  Are there any good workflows that depend on out of order execution outside the exploration phase? To use notebooks reproducibly, the whole thing should be cleared and run with some frequency. Global state shouldn't matter. There's also the nbconvert --to script jupyter function that can convert a notebook into a script.
- xiphias2 5y ago
  For Jupyter sure, it's true, but not for Observable.
- cxr 5y ago
  A couple of Brad Myers's students at CMU worked out a concept for lightweight versioning, with a particular focus on users of code notebooks. https://www.youtube.com/watch?v=rBrvT7vGleM https://www.youtube.com/watch?v=rBrvT7vGleM Unfortunately, they moved on to an evolution of the initial concept that's a lot less compelling/promising.
- jshen 5y ago
  This has nothing to do with the ability to use a tool like git. Notebooks make sloppy practices easier, but all of these problems exist in non notebook code too. You have no idea how many times I’ve run into non notebook code that assumes a database or something is in a certain state with no clear explanation of how to get to that statement. Or, a library that requires a certain order of operations that is unclear.

There is nothing inherent in notebooks that prevent the use of tools like git.

by jshen 5y ago

There is nothing inherent in notebooks that prevent the use of tools like git.

...except the need to commit the current state to make things reproducible. And I'm afraid it's not just code in cells, but also the sequence of its execution, editing, execution again, etc, because the order of running the cells is arbitrary (whatever the user commands interactively), and the results of previous invocations form the dreaded arbitrarily mutable global state.

tomnipotent 5y ago
> ...except the need to commit the current state to make things reproducible. This can definitely be a problem, but at least in the instances where I've worked with teams using Notebooks it hasn't been an issue. Usually data is either coming from networked storage (S3 etc.) as CSV/Parquet/HDF5, a database query with well-defined (enough) criterion, or API calls. > because the order of running the cells is arbitrary Most people I know have to restart their notebooks often enough to have to re-run all cells again anyway. You usually get into a good habit of making sure they're mostly "in order", or grouping common code into cells to minimize the number of cell executions and ctrl+enter's needed. > dreaded arbitrarily mutable global state It's definitely a double-edged sword, but the global state is a big part of what makes notebooks so useful. I have a repl-based environment, the ability to run arbitrary code, and global state to iteratively explore the domain of the problem I'm working on. That sword cuts me many times along the way, but they're (mostly) nicks. But it's usually worth the output, especially to the business. In all these instances, teams shared notebooks through git and most of the issues were simply understanding the context of their usage more than issues with their execution. I don't remember more than a few notebooks really being used by more than a few people anyway, so I've learned to consider them more as collaborative artifacts in an exploratory process to more mature, "packaged" code evolved from those notebooks.
beckingz 5y ago
Are there any good workflows that depend on out of order execution outside the exploration phase? To use notebooks reproducibly, the whole thing should be cleared and run with some frequency. Global state shouldn't matter. There's also the nbconvert --to script jupyter function that can convert a notebook into a script.
xiphias2 5y ago
For Jupyter sure, it's true, but not for Observable.
cxr 5y ago
A couple of Brad Myers's students at CMU worked out a concept for lightweight versioning, with a particular focus on users of code notebooks. https://www.youtube.com/watch?v=rBrvT7vGleM https://www.youtube.com/watch?v=rBrvT7vGleM Unfortunately, they moved on to an evolution of the initial concept that's a lot less compelling/promising.
jshen 5y ago
This has nothing to do with the ability to use a tool like git. Notebooks make sloppy practices easier, but all of these problems exist in non notebook code too. You have no idea how many times I’ve run into non notebook code that assumes a database or something is in a certain state with no clear explanation of how to get to that statement. Or, a library that requires a certain order of operations that is unclear.