13 ms·

The Future of Notebooks: Lessons from JupyterCon

353 points by wcrichton 8y ago

jdonaldson 8y ago
"jupyter is the new bash" As much as I love Jupyter this is a bridge too far even for me. There's nothing about Jupyter that makes sense for non-interactive scripting. Although, I suppose it would help you do block chain calculations... which would also be a terrible idea.
- adw 8y ago
  Inline documentation and charting (which for data tasks is a honking huge deal). It's literate programming with an actual use-case. Very into this idea.
  - seanmcdirmid 8y ago
    None of that is what people use bash for, however.
    - adw 8y ago
      You'd be surprised. It's certainly not _all_ of what people use bash for, but there are a lot of sketchy ETL (and sampling, and small-scale analytics) jobs out there held together with bash and awk and hopes and dreams, and replacing those with something like this is a substantial win.
- akhilcacharya 8y ago
  I don't think the full talk for that is out yet but I think it could be brilliant. Literate bash scripts with built in metrics reporting that you can just pull out if necessary? Documentation and selective re-run? Sign me up!
- deleted 8y ago
  [deleted]
jakelarkin 8y ago
I understand why they became popular, but as a software engineer considering how they work, I am just full of disappointment. we're going to spend the next ten years re-inventing every single software engineering best practice for jupyter's weirdo environment.
- sipCom19 8y ago
  Testing, style checking, code coverage...
- gaius 8y ago
  In the mid-90s I was a Mech Eng undergrad using a program called MathCAD which provided a "notebook" interactive computation mixed with text environment by running as a Word plugin. In 2018 I use Jupyter and it's not clear where the progress has been. There are too many compromises trying to make it work in a web browser. For what Jupyter is for I find that RStudio or Spyder are infinitely superior. I do interactive exploring and transforming that into both documents and reusable and deployable code all in the one tool
  - semi-extrinsic 8y ago
    I too prefer RStudio/RMarkdown. Now we have the Reticulate package that allows running Python inside RMarkdown, for my purposes it's way better than Jupyter. It's just plain text so my editor and VCS and everything plays nicely. But perhaps the biggest strength is that you go through Pandoc, so you can just click a button and get the output as a Word file for sending to someone who's not a software dev, or you can get it as a LaTeX source file and extend it into a proper formal document like a journal paper, or you can do it as HTML where embedding Bokeh scripts and other interactive things Just Works. I've even made presentations with it going through the reveal.js framework, where you can put an interactive plot on one of your slides, and show people live "what happens when we change this parameter". That's still semi-witchcraft in 2018, but it's going to become a common thing (hopefully).
  - kgwgk 8y ago
    Unfortunately RStudio also makes compromises to make it work in a web browser. A native tool could be even better.
gaius 8y ago
When you consider that it is impossible to reuse notebooks in each other... Or unit test them... Or that it is not especially easy to version control them in any sort of branch/merge workflow... Jupyter Notebooks are much closer to Excel spreadsheets than they are to what most people would consider actual programs.
- hyperpallium 8y ago
  Spreadsheets have been insanely popular and useful. If the similarity is at all deep, there's opportunity in figuring out version control, unit testing, reusable packages etc, even if they seem hopelessly impossible on the surface.
- logicchains 8y ago
  >impossible to reuse notebooks in each other... Or unit test them... It's quite easy to unit test notebooks. Whenever you write a function you want to test, write some unit tests (or even just asserts) in the same cell it's defined, then any time the cell is run to define the function, the tests are run too. Likely you'll be doing some manual tests when you write a block of code cum function, and it's not hard to copy the input crafted from those manual tests to a unit test.
  - gaius 8y ago
    write some unit tests (or even just asserts) in the same cell it's defined As many people will do this as presently write tests for their Excel macros or formulas.
- CJefferson 8y ago
  Except Excel is in some ways much better designed -- it automatically recalculate everything if you change a cell, and makes it easy to link to other spreadsheets.
  - gaius 8y ago
    Yes there’s no concept of cell dependency in Jupyter. Often find myself just re-running the entire notebook!
keeptrying 8y ago
You should never use notebooks - period. My TL;DR: • Confusing for beginners • Encourage bad practices • Poor editor
- aportnoy 8y ago
  Why not?
  - keeptrying 8y ago
    My TL;DR: • Confusing for beginners • Encourage bad practices • Poor editor
- xc1222 8y ago
  You cannot criticize any currently hyped software here, the salaries of the HN community depend on hype.
aportnoy 8y ago
Most comments here express disbelief and disappointment in Jupyter from the software engineering point of view. What exactly is wrong with it? I use Jupyter daily and find no other Python environment more productive, be it scripts or IPython or IDEs. Granted, I work in scientific computing and use Python for data wrangling and stats. I find immense value in interactivity and iteration speed.
- gaius 8y ago
  How do you find it compares to Spyder?
  - aportnoy 8y ago
    Spyder is too clunky. I work with a number of remote compute clusters, and like the ability to fire up a jupyter server through ssh, enable port forwarding, and work with the notebook in my browser. If I need a file browser or I want to quickly view two notebooks side-by-side, Jupyter Lab is still more convenient than Spyder. Although it doesn't support all the features of the notebook, and you end up installing additional extensions managed through yarn and webpack (jeez).
- taeric 8y ago
  My hunch is the same disdain you probably have towards scientists using excel is how many software engineers feel about your using jupyter. It isn't that you aren't getting good results and doing good work. It is that there are many practices that were hammered out in software development that were completely tossed out the window. 4GL comes to mind as a comparison. Makes great demos. Is not likely to lead to good engineering.
  - aportnoy 8y ago
    I use Jupyter for two things: 1. Interactive computation and plotting, run data through a pipeline. 2. Quick prototyping and testing. Once a function/class is ready, it goes into a module, together with some unit tests. Nothing beats Jupyter for that second use case, 90% of the bugs are squashed through interaction and inspection in the notebook cells, not through unit tests.
grillorafael 8y ago
As many already said, I think it is a great tool for prototyping and data exploration but when it comes to moving code to production, for me it makes very little sense to use it. Netflix said that if the job breaks they can enter the notebook with the data and see what is wrong. For me it feels like they did development with 0 safe guards and if it breaks they check why. Instead of logging problems and dealing with edge cases in the code beforehand
- jre 8y ago
  I'm not sure exactly what Netflix means by "if the job breaks", but I've used jupyter notebook in production to do ML before. If you consider the notebook as a way to augment your logs with plots, it might make a bit more sense. Running a jupyter notebook is a nice way to generate a HTML report for a job. For a typical ML pipeline, you first plot some stats about the input data, then train some model, plot some training loss, a confusion matrix, some example of predictions, etc... If some job gives a strange result (maybe that's what they mean by break), having the notebook rendered as an HTML page with all the plot is a very effective way to do a first round of diagnostics. You can also start the notebook with the same parameters and 'run' through your report, which is a nice way to do interactive debugging. Also in this case, the notebook itself was quite small in terms of lines of code. All the functions were implemented in modules, so it's really like the notebook is your 20 lines 'main' function. So you need some discipline among your team.
agumonkey 8y ago
"jupyter is the new bash" has to be the most underwhelming hyperbole of the decade
- tree_of_item 8y ago
  Are you kidding? The shell is a massively important part of modern computers, and making it more expressive would be a huge win. You might be familiar with TermKit[0] which also tried this. [0]: https://github.com/unconed/TermKit https://github.com/unconed/TermKit
  - agumonkey 8y ago
    ${SHELL} is\ horrible\ and\ dated
eanzenberg 8y ago
I despise notebooks being used in production for the reasons given early in the article: you can execute code in any order so authors usually end up with spaghetti and long pages without clear flows. I would rather scientists use and learn the tools that have been developed over decades on collaborative code writing using version control. It helps integrate their solutions too.
- logicchains 8y ago
  This is a code style problem not a notebook problem: the same people would probably write spaghetti without clear control flow if writing a Python batch script instead of a Notebook.
  - eanzenberg 8y ago
    I agree, but tools like version control do help a ton to manage and mitigate these kinds of risks. Actually, it could be super super cool if someone figured out how to version control notebooks, including tracking of code execution flow.
stevesimmons 8y ago
I don't get the people here saying don't use Jupyter notebooks, or they are bad software engineering. So much Python development is trying snippets of code in a REPL as you introspect live objects, then once they're right pasting them into the IDE. All my Jupyter notebooks are like that, where my code starts as cells of a line or two, as I check each output. Then I coalesce them into a function (which avoid the problem of execution order that some people here mention). Then I may move those functions into a normal Python module and build up complex classes, add unit tests. Or if the goal is communicating with other people the results of running that code, I'll add explanatory text and mathematical formulas, break it up into sections, include references to papers and web links. I think what people miss here is many software development tasks, especially in data science and machine learning, are essentially exploratory data analysis. And the outputs are graphs, tables of data etc, designed to be read by humans. Jupyter's ease of use for iteration, and the ability to interleave documentation, formulas, graphs and tables with the code is a big win. In my daily work, I have my IDE and Jupyter open side by side. The Jupyter notebooks are version controlled like my .py modules. Over time, I expect editors like VSCode to edit Jupyter notebooks natively. And also more code editing to move from IDEs into the Jupyter (via integrating the Monaco editor). We will all benefit from that crosspollination, whether we are principally software engineers or data scientists.
- seanmcdirmid 8y ago
  > And also more code editing to move from IDEs into the Jupyter Jupyter, and all REPLS in general, rely on a concrete top-level execution context to provide useful feeedback. Much of the code written outside of data science doesn’t really have that, which is why you don’t see notebooks being used for software development very often.
  - vosper 8y ago
    I frequently use IPython or REPL.it for software development. It's not my main editor, and I don't write large chunks of my code there, but for rapidly prototyping a small subset of some task I find them invaluable. But you are right about notebooks - I don't find them particularly useful, except as a better REPL, and no-one I know uses them for "normal" (non Data Science) software development.
celias 8y ago
If Jupyter is not your cup of tea you might like Cauldron, the unnotebook - http://www.unnotebook.com http://www.unnotebook.com Interview with the author in May, 2017 - https://www.linkedin.com/pulse/why-cauldron-might-right-data-analysis-environment-you-dan-mayhew https://www.linkedin.com/pulse/why-cauldron-might-right-data...
xvilka 8y ago
It is a disaster that more than half of the research papers do not provide data and the code, required either to reproduce research (if it is a theoretical one) or reproduce results analysis. Wider adoption of JupyterLab (and alike) will help to solve this problem. Using PDF and paper for distributing research results feels like XIX century these days.
drej 8y ago
People complaining that notebooks are not good for software engineering are missing one important point – notebooks are very often used by people who are not software engineers and/or not doing software engineering work. (The point is otherwise valid, I have seen software engineering in notebooks and it was horrific.)
h4b4n3r0 8y ago
Even in its present state, Jupyter is strictly better than plain REPL, especially if you need visualization. It's like Python in general: it was not designed to write 10-100KLOC programs. Python was designed for scripting. Jupyter was designed for interactive data exploration, and to create a record of results which you can view without re-executing the cells. Is it error prone? Yes. But then so is REPL.
- LittlePeter 8y ago
  > Python was designed for scripting. Can you provide a citation?
  - h4b4n3r0 8y ago
    I don’t provide citations for such obvious things. If you’re interested, you can find the origin story as written by Guido himself in Python FAQ using any of the available search engines.
    - LittlePeter 8y ago
      I was not able to find it. What I was able to find is this: "Python is an interpreted, interactive, object-oriented programming language." https://docs.python.org/3/faq/general.html#what-is-python https://docs.python.org/3/faq/general.html#what-is-python Yes some people do write scripts in Python. Does that make Python a scripting language? Maybe in your definition of scripting language.
      - PurpleRamen 8y ago
        A bit below your section you will find https://docs.python.org/3/faq/general.html#why-was-python-created-in-the-first-place https://docs.python.org/3/faq/general.html#why-was-python-cr... Notable part: [..]We needed a better way to do system administration[..]
- stevesimmons 8y ago
  > Python was designed for scripting. It's far more than scripting, and for much bigger programs than 10k LOC... My company's Python codebase is 35m LOC/500k modules, with 25k commits per week and contributions from 2.5k developers per month. To my mind, Python has the opposite problem. I've been a Python programmer since 2000. Every few years, I think I should devote more time to other languages (first Java, then R, then Haskell, then JS, ...). But the Python ecosystem just keeps getting stronger. My current focus is data science and machine learning. In these domains, there are no good reasons to drop Python.
aluren 8y ago
It would seem that notebooks were specifically tailored for my use cases (being in academia and doing data analysis), in practice I found that I drift away from notebooks every time I try them and stick to a good old console (ipython or bash) instead. Here are a few things that irked me: -It's browser based. Yuck. This means you can wait upwards of ten seconds for a session to load up every time you want to start one. Also, generally speaking, I deeply dislike browser-based applications that have nothing to do with the internet. It just feels so clunky. My workflow gets interrupted as I have to play between windows and betwen tabs, and whenever I want to switch from a task to another I have to remind myself 'well this is my web browsing instance that also happens to be my notebook', it's not natural and gets in the way. By contrast, a terminal advertises itself as exactly what you want to use it for. It's also very snappy. -Following up on browser-based awkwardness, my god is this thing slow. Initializing kernels. Interrupting kernels. Restarting kernels. God save you if you want to import more than a couple libraries. Wait you made a mistake and want to stop and restart that one cell? Guess you just have to go through the whole kernel thing again I guess. So you launched something and it turns out it eats way more memory than expected? Well, 'sudo pkill -9 firefox' it is, do not pass go, do not collect autosave. It just does not play nice and lags behind every thought or action you want to take. Again, contrast with a plain terminal that just does the job and does it now. -Speaking of exiting, these things don't seem to be able to do so gracefully. If something hangs up you just have to kill everything if you want to resume working, instead of good old ctrl-C. The fact that killing everything also tends to kill the web browser you used to do actual web browsing is just icing on the cake. -Setting things up can be dodgy. Should your notebook fail to start, error messages are extremely cryptic. Documentation for troubleshooting is lackluster. No one ever seems to have any two identical issues on Stackoverflow, which is also plagued with WorksOnMyMachine (tm) syndrome. If you ever have to mess up with both Python 2 and 3, you may be in for a good time. -The notebook format in which sessions are saved is supposed to be human readable and editable. In practice I have never seen a single human read or edit any of these by hand, for the very simple reason they are horrible to read or edit by hand. Again, why do I have to load a file in a cryptic format in my web browser when I could just %paste my snippets off of my editor or even just use my IDE's builtin capacities (code runner on vscode, F9 in spyder, etc.)? Plus, compatibility issues between versions can make your notebooks suddenly unusable (and hard to salvage due to the aforementioned format issue). This doesn't happen with snippets pasted onto a terminal. -IDE integration has been sloppy in my experience. I've been told this may be subject to change, however. Note that all this criticism doesn't mean I don't respect the work of the people behind this - they've made a fantastic tool that just doesn't happens to fit my use cases, for now. Every once in a while I try to work with notebooks, find they're still unusuable, shrug and go back to my terminals. As the experience improves I may make the switch one day, who knows.
michihuber 8y ago
To everyone frustrated with notebooks, I urge you to check out nextjournal.com: - Notebooks are automatically versioned - You can reference and reuse parts of other notebooks immutably through something they call "Transclusions": https://nextjournal.com/nextjournal/transclusions https://nextjournal.com/nextjournal/transclusions - The technology-stacks underlying each notebook are immutable, meaning that notebooks work on any machine (no "hidden" dependencies). - You can explicitly reference other results instead of relying on global state, making execution order irrelevant. It's currently in private beta, but people are starting to use it in production. (EDIT: You can sign up using the code curryon2018) (Disclaimer: friends of mine are building this and I used to work on it in the past.)
enriquto 8y ago
I love jupyter notebooks, but unfortunately the code editor is unusable to me. I cannot stand the bizarre parenthesis auto-completion, and the autoindentation settings; and there's no easy way to remove them.
- Tistel 8y ago
  I feel the same way. One workaround is that some editors have a plugin. Emacs has a mode called ein that is fantastic. As you poke around getting the snippets to work, you can cut/paste into a different file/buffer. After you start the Jupiter notebook copy the login token, run: ein:notebooklist-login then ein:notebooklist-open and away you go.
  - mi_lk 8y ago
    Just be careful to use https://github.com/millejoh/emacs-ipython-notebook https://github.com/millejoh/emacs-ipython-notebook instead of the unmaintained https://github.com/tkf/emacs-ipython-notebook https://github.com/tkf/emacs-ipython-notebook. Somehow the latter comes up as my first search result.
megamindbrian2 8y ago
I wrote an importers for notebooks that laods and runs the cell based on keywords I'm the markdown.
gumby 8y ago
I found this most exciting: > While Jupyter notebooks have traditionally been a humans-only entrypoint into a program, researchers and companies alike are increasingly using notebooks for automation. It's part of a slow journey back the power of the Lispms and the Smalltalk environments.
ur-whale 8y ago
"On the education side of things, Jupyter is quickly gaining adoption in universities around America, particularly for data science courses. Conversely, data science is increasingly becoming students’ first exposure to programming and computer science" Am I the only one this two sentences worries deeply? Or is it just a rehash of the twenty year old "my first exposure to programming was Excel" (which, from what I'm reading is what the article is talking about : a glorified spreadsheet).
ausjke 8y ago
Been a bpython lover now I mainly use ipython. Used jupyter + ipython kernel once a while but for 99% my use cases, ipython is good enough(testing out code, checking out help info,etc). For anything you want to have graph(e.g. matlab), notebook shall serve you well.
dangjc 8y ago
I can’t recommend enough using Atom with the Hydrogen plugin. A Jupyter kernel runs in the back and you can execute a line or multiple lines of code at once just like the web notebooks. But you’re editing a plain code file, not an .ipynb, which is easy to check into git, move to inside a library, and get full IDE tooling on.
pyrce 8y ago
As the speaker for the scheduling notebook talk that got referenced here, I can probably give a few insights about how we're using notebooks. A lot of the risks and concerns brought up here were talked about in the session. The slides ended up on https://conferences.oreilly.com/jupyter/jup-ny/public/schedule/detail/68348 https://conferences.oreilly.com/jupyter/jup-ny/public/schedu... (and hopefully the talks themselves will get posted soon). In particular I referenced how we treat and emphasize notebooks as an integration tool which acts as a good place to combine actions with documentation, visuals, and output logs. There's a section on Integrating Notebook which outlines how we approach this. There's also a strong emphasis on pushing complexity and shared code into the repositories housing notebooks. Effectively you end up with a lot of same best-practices found in non-notebook development, and the same abuses that lead to unmaintainable code -- which sometimes is needed in the short term. So far we've had a lot of success with notebooks in production as parameterizable templates, or as a way to easily produce scheduled reports or machine learning experiments. Many users just provide the parameters while supporting teams provide the tested templates. Other users like being able to simply schedule their iterated work without needing to translate to another medium. One of the biggest wins though is gaining a shared interface for debugging, experimenting, and reusing code by having notebooks as the output artifacts of execution (even if it's just executing some other code elsewhere on behalf of the user). Papermill made a lot of this possible by separating input notebooks from output notebooks, and by being able to inject runtime values into those output notebooks.
- pyrce 8y ago
  We also are doing a blog series on how we're approaching notebooks at https://medium.com/netflix-techblog/notebook-innovation-591ee3221233 https://medium.com/netflix-techblog/notebook-innovation-591e... (the scheduling post is the second in the series). Also I'd recommend listening to talks from the conference with an open mind. There were a lot of great discussions and positive energy around notebooks. Hopefully these help describe a few of the new patterns and tools available in the notebook space.
- chuckcode 8y ago
  I really like notebooks as a way to share an analysis or data visualization. For me the biggest benefits are: - Integration. So nice to have access to the compute, data and libraries all in one place. There is a surprising amount of hassle moving data, setting up paths and libraries, etc. Notebooks almost act like for a container for data analysis rather than service. - Sharable and reproducible. My coworkers can reproduce and explore some new idea with almost no effort, especially important when their strengths are more ML or stats than devops. - Literate programming. It is really nice to have plots, markdown and code all it one place when deliverable is a report or analysis rather than code. Even with these benefits I do think the criticisms about software development are right on point. Notebooks are a step backward in terms of software engineering environment with none of the modern tooling, version control, testing frameworks etc. I think that the folks who dismiss notebooks as a platform though are missing some important benefits that have long been absent in current editors. Larger companies like Facebook and Google are already facing the reality that devops is a pain point even for sophisticated software engineers and have developed remote code editors like cider and nucleotide to try and enable bringing code development to the data and compute rather than doing it from the laptop. R has been working on a long time on integrating analysis results and code in a reproducible package with sweave/knitr and python now has pweave in a similar fashion. I hear all the issues with jupyter and I'm not particularly married to current form of notebooks. I do think though that the features of remote development, data visualization and support for literate/report programming and sharing ode are first class features that I'll continue to want in the future.
plg 8y ago
from Joel Grus, “I don’t like Notebooks” https://docs.google.com/presentation/d/1n2RlMdmv1p25Xy5thJUhkKGvjtV-dkAIsUXP-AL4ffI/ https://docs.google.com/presentation/d/1n2RlMdmv1p25Xy5thJUh...
abetlen 8y ago
As a pretty heavy notebook user (for computer vision and machine learning), I have to say that all of these new features are nice but they don't address the real problems with the current ecosystem. Namely, there's no good way to transition exploratory notebook code to python modules. We need linting and refactoring tools in the notebook and something better than `aimport` for moving code into separate `.py` files.
Eridrus 8y ago
I like notebooks. I like them for data work. I like them for prototyping things that many may consider software engineering. I still think this presentation has a lot of good points about where they could improve.

353 points by wcrichton 8y ago

"jupyter is the new bash" As much as I love Jupyter this is a bridge too far even for me. There's nothing about Jupyter that makes sense for non-interactive scripting. Although, I suppose it would help you do block chain calculations... which would also be a terrible idea.

adw 8y ago
Inline documentation and charting (which for data tasks is a honking huge deal). It's literate programming with an actual use-case. Very into this idea.
- seanmcdirmid 8y ago
  None of that is what people use bash for, however.
  - adw 8y ago
    You'd be surprised. It's certainly not _all_ of what people use bash for, but there are a lot of sketchy ETL (and sampling, and small-scale analytics) jobs out there held together with bash and awk and hopes and dreams, and replacing those with something like this is a substantial win.
akhilcacharya 8y ago
I don't think the full talk for that is out yet but I think it could be brilliant. Literate bash scripts with built in metrics reporting that you can just pull out if necessary? Documentation and selective re-run? Sign me up!
deleted 8y ago
[deleted]

jakelarkin 8y ago

I understand why they became popular, but as a software engineer considering how they work, I am just full of disappointment. we're going to spend the next ten years re-inventing every single software engineering best practice for jupyter's weirdo environment.

sipCom19 8y ago
Testing, style checking, code coverage...
gaius 8y ago
In the mid-90s I was a Mech Eng undergrad using a program called MathCAD which provided a "notebook" interactive computation mixed with text environment by running as a Word plugin. In 2018 I use Jupyter and it's not clear where the progress has been. There are too many compromises trying to make it work in a web browser. For what Jupyter is for I find that RStudio or Spyder are infinitely superior. I do interactive exploring and transforming that into both documents and reusable and deployable code all in the one tool
- semi-extrinsic 8y ago
  I too prefer RStudio/RMarkdown. Now we have the Reticulate package that allows running Python inside RMarkdown, for my purposes it's way better than Jupyter. It's just plain text so my editor and VCS and everything plays nicely. But perhaps the biggest strength is that you go through Pandoc, so you can just click a button and get the output as a Word file for sending to someone who's not a software dev, or you can get it as a LaTeX source file and extend it into a proper formal document like a journal paper, or you can do it as HTML where embedding Bokeh scripts and other interactive things Just Works. I've even made presentations with it going through the reveal.js framework, where you can put an interactive plot on one of your slides, and show people live "what happens when we change this parameter". That's still semi-witchcraft in 2018, but it's going to become a common thing (hopefully).
- kgwgk 8y ago
  Unfortunately RStudio also makes compromises to make it work in a web browser. A native tool could be even better.

gaius 8y ago

When you consider that it is impossible to reuse notebooks in each other... Or unit test them... Or that it is not especially easy to version control them in any sort of branch/merge workflow... Jupyter Notebooks are much closer to Excel spreadsheets than they are to what most people would consider actual programs.

hyperpallium 8y ago
Spreadsheets have been insanely popular and useful. If the similarity is at all deep, there's opportunity in figuring out version control, unit testing, reusable packages etc, even if they seem hopelessly impossible on the surface.
logicchains 8y ago
>impossible to reuse notebooks in each other... Or unit test them... It's quite easy to unit test notebooks. Whenever you write a function you want to test, write some unit tests (or even just asserts) in the same cell it's defined, then any time the cell is run to define the function, the tests are run too. Likely you'll be doing some manual tests when you write a block of code cum function, and it's not hard to copy the input crafted from those manual tests to a unit test.
- gaius 8y ago
  write some unit tests (or even just asserts) in the same cell it's defined As many people will do this as presently write tests for their Excel macros or formulas.
CJefferson 8y ago
Except Excel is in some ways much better designed -- it automatically recalculate everything if you change a cell, and makes it easy to link to other spreadsheets.
- gaius 8y ago
  Yes there’s no concept of cell dependency in Jupyter. Often find myself just re-running the entire notebook!

keeptrying 8y ago

You should never use notebooks - period. My TL;DR: • Confusing for beginners • Encourage bad practices • Poor editor

aportnoy 8y ago
Why not?
- keeptrying 8y ago
  My TL;DR: • Confusing for beginners • Encourage bad practices • Poor editor
xc1222 8y ago
You cannot criticize any currently hyped software here, the salaries of the HN community depend on hype.

aportnoy 8y ago

Most comments here express disbelief and disappointment in Jupyter from the software engineering point of view. What exactly is wrong with it? I use Jupyter daily and find no other Python environment more productive, be it scripts or IPython or IDEs. Granted, I work in scientific computing and use Python for data wrangling and stats. I find immense value in interactivity and iteration speed.

gaius 8y ago
How do you find it compares to Spyder?
- aportnoy 8y ago
  Spyder is too clunky. I work with a number of remote compute clusters, and like the ability to fire up a jupyter server through ssh, enable port forwarding, and work with the notebook in my browser. If I need a file browser or I want to quickly view two notebooks side-by-side, Jupyter Lab is still more convenient than Spyder. Although it doesn't support all the features of the notebook, and you end up installing additional extensions managed through yarn and webpack (jeez).
taeric 8y ago
My hunch is the same disdain you probably have towards scientists using excel is how many software engineers feel about your using jupyter. It isn't that you aren't getting good results and doing good work. It is that there are many practices that were hammered out in software development that were completely tossed out the window. 4GL comes to mind as a comparison. Makes great demos. Is not likely to lead to good engineering.
- aportnoy 8y ago
  I use Jupyter for two things: 1. Interactive computation and plotting, run data through a pipeline. 2. Quick prototyping and testing. Once a function/class is ready, it goes into a module, together with some unit tests. Nothing beats Jupyter for that second use case, 90% of the bugs are squashed through interaction and inspection in the notebook cells, not through unit tests.

grillorafael 8y ago

As many already said, I think it is a great tool for prototyping and data exploration but when it comes to moving code to production, for me it makes very little sense to use it. Netflix said that if the job breaks they can enter the notebook with the data and see what is wrong. For me it feels like they did development with 0 safe guards and if it breaks they check why. Instead of logging problems and dealing with edge cases in the code beforehand

jre 8y ago
I'm not sure exactly what Netflix means by "if the job breaks", but I've used jupyter notebook in production to do ML before. If you consider the notebook as a way to augment your logs with plots, it might make a bit more sense. Running a jupyter notebook is a nice way to generate a HTML report for a job. For a typical ML pipeline, you first plot some stats about the input data, then train some model, plot some training loss, a confusion matrix, some example of predictions, etc... If some job gives a strange result (maybe that's what they mean by break), having the notebook rendered as an HTML page with all the plot is a very effective way to do a first round of diagnostics. You can also start the notebook with the same parameters and 'run' through your report, which is a nice way to do interactive debugging. Also in this case, the notebook itself was quite small in terms of lines of code. All the functions were implemented in modules, so it's really like the notebook is your 20 lines 'main' function. So you need some discipline among your team.

agumonkey 8y ago

"jupyter is the new bash" has to be the most underwhelming hyperbole of the decade

tree_of_item 8y ago
Are you kidding? The shell is a massively important part of modern computers, and making it more expressive would be a huge win. You might be familiar with TermKit[0] which also tried this. [0]: https://github.com/unconed/TermKit https://github.com/unconed/TermKit
- agumonkey 8y ago
  ${SHELL} is\ horrible\ and\ dated

eanzenberg 8y ago

I despise notebooks being used in production for the reasons given early in the article: you can execute code in any order so authors usually end up with spaghetti and long pages without clear flows. I would rather scientists use and learn the tools that have been developed over decades on collaborative code writing using version control. It helps integrate their solutions too.

logicchains 8y ago
This is a code style problem not a notebook problem: the same people would probably write spaghetti without clear control flow if writing a Python batch script instead of a Notebook.
- eanzenberg 8y ago
  I agree, but tools like version control do help a ton to manage and mitigate these kinds of risks. Actually, it could be super super cool if someone figured out how to version control notebooks, including tracking of code execution flow.

stevesimmons 8y ago

I don't get the people here saying don't use Jupyter notebooks, or they are bad software engineering. So much Python development is trying snippets of code in a REPL as you introspect live objects, then once they're right pasting them into the IDE. All my Jupyter notebooks are like that, where my code starts as cells of a line or two, as I check each output. Then I coalesce them into a function (which avoid the problem of execution order that some people here mention). Then I may move those functions into a normal Python module and build up complex classes, add unit tests. Or if the goal is communicating with other people the results of running that code, I'll add explanatory text and mathematical formulas, break it up into sections, include references to papers and web links. I think what people miss here is many software development tasks, especially in data science and machine learning, are essentially exploratory data analysis. And the outputs are graphs, tables of data etc, designed to be read by humans. Jupyter's ease of use for iteration, and the ability to interleave documentation, formulas, graphs and tables with the code is a big win. In my daily work, I have my IDE and Jupyter open side by side. The Jupyter notebooks are version controlled like my .py modules. Over time, I expect editors like VSCode to edit Jupyter notebooks natively. And also more code editing to move from IDEs into the Jupyter (via integrating the Monaco editor). We will all benefit from that crosspollination, whether we are principally software engineers or data scientists.

seanmcdirmid 8y ago
> And also more code editing to move from IDEs into the Jupyter Jupyter, and all REPLS in general, rely on a concrete top-level execution context to provide useful feeedback. Much of the code written outside of data science doesn’t really have that, which is why you don’t see notebooks being used for software development very often.
- vosper 8y ago
  I frequently use IPython or REPL.it for software development. It's not my main editor, and I don't write large chunks of my code there, but for rapidly prototyping a small subset of some task I find them invaluable. But you are right about notebooks - I don't find them particularly useful, except as a better REPL, and no-one I know uses them for "normal" (non Data Science) software development.

celias 8y ago

If Jupyter is not your cup of tea you might like Cauldron, the unnotebook - http://www.unnotebook.com http://www.unnotebook.com Interview with the author in May, 2017 - https://www.linkedin.com/pulse/why-cauldron-might-right-data-analysis-environment-you-dan-mayhew https://www.linkedin.com/pulse/why-cauldron-might-right-data...

xvilka 8y ago

It is a disaster that more than half of the research papers do not provide data and the code, required either to reproduce research (if it is a theoretical one) or reproduce results analysis. Wider adoption of JupyterLab (and alike) will help to solve this problem. Using PDF and paper for distributing research results feels like XIX century these days.

drej 8y ago

People complaining that notebooks are not good for software engineering are missing one important point – notebooks are very often used by people who are not software engineers and/or not doing software engineering work. (The point is otherwise valid, I have seen software engineering in notebooks and it was horrific.)

h4b4n3r0 8y ago

Even in its present state, Jupyter is strictly better than plain REPL, especially if you need visualization. It's like Python in general: it was not designed to write 10-100KLOC programs. Python was designed for scripting. Jupyter was designed for interactive data exploration, and to create a record of results which you can view without re-executing the cells. Is it error prone? Yes. But then so is REPL.

LittlePeter 8y ago
> Python was designed for scripting. Can you provide a citation?
- h4b4n3r0 8y ago
  I don’t provide citations for such obvious things. If you’re interested, you can find the origin story as written by Guido himself in Python FAQ using any of the available search engines.
  - LittlePeter 8y ago
    I was not able to find it. What I was able to find is this: "Python is an interpreted, interactive, object-oriented programming language." https://docs.python.org/3/faq/general.html#what-is-python https://docs.python.org/3/faq/general.html#what-is-python Yes some people do write scripts in Python. Does that make Python a scripting language? Maybe in your definition of scripting language.
    - PurpleRamen 8y ago
      A bit below your section you will find https://docs.python.org/3/faq/general.html#why-was-python-created-in-the-first-place https://docs.python.org/3/faq/general.html#why-was-python-cr... Notable part: [..]We needed a better way to do system administration[..]
stevesimmons 8y ago
> Python was designed for scripting. It's far more than scripting, and for much bigger programs than 10k LOC... My company's Python codebase is 35m LOC/500k modules, with 25k commits per week and contributions from 2.5k developers per month. To my mind, Python has the opposite problem. I've been a Python programmer since 2000. Every few years, I think I should devote more time to other languages (first Java, then R, then Haskell, then JS, ...). But the Python ecosystem just keeps getting stronger. My current focus is data science and machine learning. In these domains, there are no good reasons to drop Python.

aluren 8y ago

It would seem that notebooks were specifically tailored for my use cases (being in academia and doing data analysis), in practice I found that I drift away from notebooks every time I try them and stick to a good old console (ipython or bash) instead. Here are a few things that irked me: -It's browser based. Yuck. This means you can wait upwards of ten seconds for a session to load up every time you want to start one. Also, generally speaking, I deeply dislike browser-based applications that have nothing to do with the internet. It just feels so clunky. My workflow gets interrupted as I have to play between windows and betwen tabs, and whenever I want to switch from a task to another I have to remind myself 'well this is my web browsing instance that also happens to be my notebook', it's not natural and gets in the way. By contrast, a terminal advertises itself as exactly what you want to use it for. It's also very snappy. -Following up on browser-based awkwardness, my god is this thing slow. Initializing kernels. Interrupting kernels. Restarting kernels. God save you if you want to import more than a couple libraries. Wait you made a mistake and want to stop and restart that one cell? Guess you just have to go through the whole kernel thing again I guess. So you launched something and it turns out it eats way more memory than expected? Well, 'sudo pkill -9 firefox' it is, do not pass go, do not collect autosave. It just does not play nice and lags behind every thought or action you want to take. Again, contrast with a plain terminal that just does the job and does it now. -Speaking of exiting, these things don't seem to be able to do so gracefully. If something hangs up you just have to kill everything if you want to resume working, instead of good old ctrl-C. The fact that killing everything also tends to kill the web browser you used to do actual web browsing is just icing on the cake. -Setting things up can be dodgy. Should your notebook fail to start, error messages are extremely cryptic. Documentation for troubleshooting is lackluster. No one ever seems to have any two identical issues on Stackoverflow, which is also plagued with WorksOnMyMachine (tm) syndrome. If you ever have to mess up with both Python 2 and 3, you may be in for a good time. -The notebook format in which sessions are saved is supposed to be human readable and editable. In practice I have never seen a single human read or edit any of these by hand, for the very simple reason they are horrible to read or edit by hand. Again, why do I have to load a file in a cryptic format in my web browser when I could just %paste my snippets off of my editor or even just use my IDE's builtin capacities (code runner on vscode, F9 in spyder, etc.)? Plus, compatibility issues between versions can make your notebooks suddenly unusable (and hard to salvage due to the aforementioned format issue). This doesn't happen with snippets pasted onto a terminal. -IDE integration has been sloppy in my experience. I've been told this may be subject to change, however. Note that all this criticism doesn't mean I don't respect the work of the people behind this - they've made a fantastic tool that just doesn't happens to fit my use cases, for now. Every once in a while I try to work with notebooks, find they're still unusuable, shrug and go back to my terminals. As the experience improves I may make the switch one day, who knows.

michihuber 8y ago

To everyone frustrated with notebooks, I urge you to check out nextjournal.com: - Notebooks are automatically versioned - You can reference and reuse parts of other notebooks immutably through something they call "Transclusions": https://nextjournal.com/nextjournal/transclusions https://nextjournal.com/nextjournal/transclusions - The technology-stacks underlying each notebook are immutable, meaning that notebooks work on any machine (no "hidden" dependencies). - You can explicitly reference other results instead of relying on global state, making execution order irrelevant. It's currently in private beta, but people are starting to use it in production. (EDIT: You can sign up using the code curryon2018) (Disclaimer: friends of mine are building this and I used to work on it in the past.)

enriquto 8y ago

I love jupyter notebooks, but unfortunately the code editor is unusable to me. I cannot stand the bizarre parenthesis auto-completion, and the autoindentation settings; and there's no easy way to remove them.

Tistel 8y ago
I feel the same way. One workaround is that some editors have a plugin. Emacs has a mode called ein that is fantastic. As you poke around getting the snippets to work, you can cut/paste into a different file/buffer. After you start the Jupiter notebook copy the login token, run: ein:notebooklist-login then ein:notebooklist-open and away you go.
- mi_lk 8y ago
  Just be careful to use https://github.com/millejoh/emacs-ipython-notebook https://github.com/millejoh/emacs-ipython-notebook instead of the unmaintained https://github.com/tkf/emacs-ipython-notebook https://github.com/tkf/emacs-ipython-notebook. Somehow the latter comes up as my first search result.

megamindbrian2 8y ago

I wrote an importers for notebooks that laods and runs the cell based on keywords I'm the markdown.

gumby 8y ago

I found this most exciting: > While Jupyter notebooks have traditionally been a humans-only entrypoint into a program, researchers and companies alike are increasingly using notebooks for automation. It's part of a slow journey back the power of the Lispms and the Smalltalk environments.

ur-whale 8y ago

"On the education side of things, Jupyter is quickly gaining adoption in universities around America, particularly for data science courses. Conversely, data science is increasingly becoming students’ first exposure to programming and computer science" Am I the only one this two sentences worries deeply? Or is it just a rehash of the twenty year old "my first exposure to programming was Excel" (which, from what I'm reading is what the article is talking about : a glorified spreadsheet).

ausjke 8y ago

Been a bpython lover now I mainly use ipython. Used jupyter + ipython kernel once a while but for 99% my use cases, ipython is good enough(testing out code, checking out help info,etc). For anything you want to have graph(e.g. matlab), notebook shall serve you well.

dangjc 8y ago

I can’t recommend enough using Atom with the Hydrogen plugin. A Jupyter kernel runs in the back and you can execute a line or multiple lines of code at once just like the web notebooks. But you’re editing a plain code file, not an .ipynb, which is easy to check into git, move to inside a library, and get full IDE tooling on.

pyrce 8y ago

As the speaker for the scheduling notebook talk that got referenced here, I can probably give a few insights about how we're using notebooks. A lot of the risks and concerns brought up here were talked about in the session. The slides ended up on https://conferences.oreilly.com/jupyter/jup-ny/public/schedule/detail/68348 https://conferences.oreilly.com/jupyter/jup-ny/public/schedu... (and hopefully the talks themselves will get posted soon). In particular I referenced how we treat and emphasize notebooks as an integration tool which acts as a good place to combine actions with documentation, visuals, and output logs. There's a section on Integrating Notebook which outlines how we approach this. There's also a strong emphasis on pushing complexity and shared code into the repositories housing notebooks. Effectively you end up with a lot of same best-practices found in non-notebook development, and the same abuses that lead to unmaintainable code -- which sometimes is needed in the short term. So far we've had a lot of success with notebooks in production as parameterizable templates, or as a way to easily produce scheduled reports or machine learning experiments. Many users just provide the parameters while supporting teams provide the tested templates. Other users like being able to simply schedule their iterated work without needing to translate to another medium. One of the biggest wins though is gaining a shared interface for debugging, experimenting, and reusing code by having notebooks as the output artifacts of execution (even if it's just executing some other code elsewhere on behalf of the user). Papermill made a lot of this possible by separating input notebooks from output notebooks, and by being able to inject runtime values into those output notebooks.

pyrce 8y ago
We also are doing a blog series on how we're approaching notebooks at https://medium.com/netflix-techblog/notebook-innovation-591ee3221233 https://medium.com/netflix-techblog/notebook-innovation-591e... (the scheduling post is the second in the series). Also I'd recommend listening to talks from the conference with an open mind. There were a lot of great discussions and positive energy around notebooks. Hopefully these help describe a few of the new patterns and tools available in the notebook space.
chuckcode 8y ago
I really like notebooks as a way to share an analysis or data visualization. For me the biggest benefits are: - Integration. So nice to have access to the compute, data and libraries all in one place. There is a surprising amount of hassle moving data, setting up paths and libraries, etc. Notebooks almost act like for a container for data analysis rather than service. - Sharable and reproducible. My coworkers can reproduce and explore some new idea with almost no effort, especially important when their strengths are more ML or stats than devops. - Literate programming. It is really nice to have plots, markdown and code all it one place when deliverable is a report or analysis rather than code. Even with these benefits I do think the criticisms about software development are right on point. Notebooks are a step backward in terms of software engineering environment with none of the modern tooling, version control, testing frameworks etc. I think that the folks who dismiss notebooks as a platform though are missing some important benefits that have long been absent in current editors. Larger companies like Facebook and Google are already facing the reality that devops is a pain point even for sophisticated software engineers and have developed remote code editors like cider and nucleotide to try and enable bringing code development to the data and compute rather than doing it from the laptop. R has been working on a long time on integrating analysis results and code in a reproducible package with sweave/knitr and python now has pweave in a similar fashion. I hear all the issues with jupyter and I'm not particularly married to current form of notebooks. I do think though that the features of remote development, data visualization and support for literate/report programming and sharing ode are first class features that I'll continue to want in the future.

plg 8y ago

from Joel Grus, “I don’t like Notebooks” https://docs.google.com/presentation/d/1n2RlMdmv1p25Xy5thJUhkKGvjtV-dkAIsUXP-AL4ffI/ https://docs.google.com/presentation/d/1n2RlMdmv1p25Xy5thJUh...

abetlen 8y ago

As a pretty heavy notebook user (for computer vision and machine learning), I have to say that all of these new features are nice but they don't address the real problems with the current ecosystem. Namely, there's no good way to transition exploratory notebook code to python modules. We need linting and refactoring tools in the notebook and something better than `aimport` for moving code into separate `.py` files.

Eridrus 8y ago

I like notebooks. I like them for data work. I like them for prototyping things that many may consider software engineering. I still think this presentation has a lot of good points about where they could improve.