14 ms·

What’s wrong with computational notebooks?

377 points by ashort11 7y ago

hprotagonist 7y ago
See also Joel Grus' talk, "I Don't Like Notebooks": https://www.youtube.com/watch?v=7jiPeIFXb6U https://www.youtube.com/watch?v=7jiPeIFXb6U slides: https://docs.google.com/presentation/d/1n2RlMdmv1p25Xy5thJUhkKGvjtV-dkAIsUXP-AL4ffI/edit https://docs.google.com/presentation/d/1n2RlMdmv1p25Xy5thJUh...
- funklute 7y ago
  Superb talk! It's worth noting that a lot of the issues he brings up, ultimately stem from the format in which Jupyter notebooks are stored. R notebooks, with their plain-text stored format as well as code-chunk parameters, solve some, but not all, of these problems.
  - hprotagonist 7y ago
    https://github.com/mwouts/jupytext https://github.com/mwouts/jupytext doesn’t solve the state management testing or tooling issues though, but commits are slightly less awful.
    - funklute 7y ago
      Yea, when I first found that extension, I was pretty excited about it. But it ultimately is not a first class citizen in the way it is for R notebooks, so I simply don't feel as comfortable using this as I might otherwise have been.
nl 7y ago
The FastAI people have been working on a lot of these issues with their NBDev too: https://www.fast.ai/2019/12/02/nbdev/ https://www.fast.ai/2019/12/02/nbdev/
- jph00 7y ago
  nbdev deals with the version control and library of code issues pretty effectively.
- zneveu 7y ago
  Came here to post the same thing. Nbdev helps fill in the strengths that IDEs are traditionally good at. Even if you don't use the full nbdev library and templates, the work flow makes sense. Write code in Jupyter, export to a python library, and you can use it everywhere else after that.
wrnr 7y ago
Trying my best to solve some of these: https://github.com/wrnrlr/foxtrot https://github.com/wrnrlr/foxtrot
moultano 7y ago
I want a notebook where causality can only flow forward through the cells. I hate notebook time-loops where a variable from a deleted cell can still be in scope. 1. Checkpoint the interpreter state after every cell execution. 2. If I edit a cell, roll back to the previous checkpoint and let execution follow from there. I can't tell you how many times I've seen accidental persistence of dead state waste hours of people's time.
- johnc1231 7y ago
  I haven't dug into it myself, but Netflix makes something called Polynote that is supposed to add some awareness of the sequence of the cells to combat this
- ivalm 7y ago
  If the interpreter state contains large variables checkpointing might not be viable (eg I have dataframes that are 100s of GB/large fractions of total available memory, reading/writing from hard drive all the time would be relatively slow. If you can save deltas I guess it wouldn't be too space inefficient but I imagine still slow). At the same time, I do like the idea of an append only notebook where you can: 1. Only run cells in sequential order 2. Only edit cells that are below the most recently ran cell. Thankfully you can enforce it through code practice and the notebook is relatively guaranteed to be "run all"-able. You will need to refactor it after the initial dirty run, but at least it's easy to reason about.
  - roblabla 7y ago
    Assuming the data isn't changed, thanks to CoW forking wouldn't cause any extra memory usage. If only a subset of data is changed, same thing - only the changed cells will take extra space. The problem only occurs when the whole variable changes - in which case yeah, you're SOL. I wonder what the usage patterns are for such datasets?
    - ivalm 7y ago
      Personal experience: when first looking at the data I often do lots of map /reduce style operations which might transform large portions of the dataframe. Question, if you use CoW then presumably your variable blocks are no longer contiguous, wouldn't this really slow down vector operations?
stared 7y ago
(A frequent Jupyter Notebook user here. For data exploration, and teaching deep learning - then Colab is indispensable.) The main question is: what are the alternatives, for data exploration (and sharing its results). Similarly, for data science tool demos, Notebooks shine. IMHO the problem is not in the notebooks, but in how they are being used (i.e. the workflow). By writing scripts in py files, and using notebooks only to show their results (processed data, charts, etc) we get the best of both worlds. The only build-in problem with Jupyter Notebooks is JSON, mixing input and output (and making it pain to work with version control). But here RMarkdown (and a few other alternatives) work well.
- fhennig 7y ago
  Yes, the article mentions users copy pasting snippets from their personal "library". Well, that could just be made into an actual library of functions to call. I'm currently at uni enrolled in an AI/ML degree, and there are a lot of people with no previous exposure to programming. It's just that most people don't know that these things are possible, don't want to learn another tool (IDE) and are not interested in longevity of the code, just in the results. This shouldn't sound like me complaining, I totally understand. I think a lot of the stuff could be solved with just better tooling, but a familiarity with software development is definitely helpful. Also a while back streamlit (https://www.streamlit.io/ https://www.streamlit.io/) was here on HN and since then I've been meaning to try it. I think this could be a good approach to bring together the best of both worlds.
  - sillysaurusx 7y ago
    most people don't know that these things are possible, don't want to learn another tool (IDE) and are not interested in longevity of the code, just in the results. This approach is arguably more effective than wasting time trying to refactor everything into a library of functions. Programming-as-crafting needs to be more of a thing. Not everything is written to be long-lasting. Even HN ushered the ugly code into hook functions that weren't shipped with the main codebase.
- karlicoss 7y ago
rabryan35 7y ago
No mention of https://observablehq.com https://observablehq.com notebooks? They’re the best I’ve found in the “Share and collaborate” and “As products” category. JupyterLab is still pretty great for exploratory stuff, but visualization possibilities in observable are incredible.
- azhenley 7y ago
  No one we interviewed or surveyed mentioned it.
- goatlover 7y ago
  Problem is that Javascript doesn't have the scientific computing ecosystem that Python, R and Julia have. Jupyter supports those languages and any others that people write kernels for. And you can also execute bash, JS, CSS and HTML directly in python notebooks with magic commands.
- tomgp 7y ago
  Agreed, Observable fixes a lot of the problems I've had with other notebooks. It can still be fiddly for code over a certain size/ complexity but the ability to import from npm modules goes a long way to fixing the problem. The user base seems to be predominantly drawn from the visualisation side of things + the fact that it's javascript may limit its uptake in science/maths areas. Aside: I've felt for a while that JS is really missing decent maths/stats libraries, any suggestions?
- btbuildem 7y ago
  They're a walled garden and to be honest I kinda hate them. Bostock had lots of accessible D3 examples before, now it's all on that "platform".. sure it's slick, but overall loss for the ecosystem IMO.
evrydayhustling 7y ago
This is a solid list. It will be even better if juxtaposed with current efforts to solve each of these problems - every DS I know is addressing at least 2-3 of these with some pet tools in their own environment. For example, we use Panel and Holoviews to make data exploration much easier. I have a feeling the ecosystem would improve faster if we had an index of (partial) solutions aligned with this problem set. One category left out of the list: testing of data pipelines (c.f. great expectations).
azhenley 7y ago
Co-author of the study here. Let me know if you have any questions or how you overcome some of the problems we identified!
- autokad 7y ago
  As a data scientist, I used all of the notebooks and didn't find any of the problems listed with databricks. I don't get to use it in my current role, miss it a lot.
- snapetom 7y ago
  I just wanted to say thank you. Many of the points in your study strikes a nerve. Part of my responsibility at my last job was to introduce good software engineering practices. What happens? The data scientists go rogue and start running notebooks left and right. How do they productionize their work? Well, they don't. They were academics. All they know is that the models ran fine in their notebooks on their laptops. Meanwhile, we didn't have anyone that was devoted full time on model productionization. Sharing data? They had enough problems sharing their notebooks.
  - ssivark 7y ago
    I just happened to be reading Peter Naur's "Programming as theory building" recently. It strikes me that taking its theme even a little seriously helps understand why notebooks are so popular. Notebooks happen to be convenient tools for exploring a new domain (interactively). Irrespective of how much software purists might complain, conventional software engineering provides very few tools/solutions/practices for that process. The wretched state of interactive debugging (in most languages) is a simple example. As someone who spends a substantial amount of time working with both modes (writing research code in Jupyter notebooks, and writing production code as python modules), notebooks scratch certain itches that IDEs typically don't even come close to. (Some recent progress on add-ons in Javascript-based editors is potentially interesting, because that might help marry the strengths of the two) In my experience, in the evolution of code from Jupyter notebooks to repositories of production code as part of any project, there comes a "right time" to switch from the former to the latter. And this can typically only be learned with experience.
ktpsns 7y ago
Obviously it's pretty hard to make general criticism of the Notebook GUI. This is especially without comparing to a specific other user interface for data scientists, such as a traditional REPL terminal, or some other command line tools? The Python world gives a good example about the sheer complexity of a notebook infrastructure. The is IPython, there is Jupyter Notebook, JupyterLab. There is even stuff like the SageMathCloud (nowadays called CoCalc) which is basically a web GUI to a VPS combining command lines and various notebooks. And hell, most of these web based interfaces try to make sharing easy. Mabye we should start comparing these (mostly OSS) tools to the traditional notebook GUIs of Matlab and Mathematica, something we used in the 90s and 2000s. From my feeling, they were more robust, could handle large data better, but they lack all the tooling we get for free in the web.
- nl 7y ago
  Jupyter Notebook/Jupyter Lab has replaced IPython as the notebook front end. I suspect 90%+ of Python Notebook work is done in Jupyter/Jupyter Lab (or things built on it like Google Collab/Kaggle Kernels). traditional notebook GUIs of Matlab and Mathematica, something we used in the 90s and 2000s. From my feeling, they were more robust, could handle large data better I've done 10s of terabyte analysis on Jupyter (Spark backend) and I personally know people doing petabyte work on it so this seems doubtful.
  - ktpsns 7y ago
    You probably were careful enough to understand the limits of the Jupyter server and client (frontend). It's easy to screw up a terminal application in data science when dumping a large array. Many REPLs cannot handle this properly (and CTRL+C won't work). It's easy to test this: What does your favourite notebook do when you call some command such as (pseudocode/python here) print(range(int(1e7))) # or 1e8 In this particular example, the python CLI seems to handle keyboard interrupts fine when the terminal (or RAM) is flooded.
rcar 7y ago
I'm someone who has been programming for a very long time and has been using notebooks for a reasonably long time (and almost always starts projects with them), my feeling is that they are a bit like C in that they make it easy to accidentally shoot yourself in the foot if you aren't careful. I always strive to end up with a notebook that can be "Run All" from a fresh clone, and I'd say that I'm successful with that maybe 60-70% of the time, and am close enough that I can fix it in the remainder. As the article (and the many others like it that have frequently cropped up as soon as IPython Notebooks first started ramping up in popularity) points out though, a lot of newer users don't have the discipline to ensure that they're not jumping around too much. It's not a problem for them in the immediate term since they know how the state ought to work, but then it becomes a mess when they try to share it with someone else (or to run it themselves again 3 months later). The challenge though is that the data analysis workflows that it allows are unbeatable by any other tools I've tried. In the end, it may just be that it's the worst form of data programming except for all of the others that have been tried.
tbenst 7y ago
I think Atom’s hydrogen and VSCode’s python are best-in-class Jupyter clients that achieve everything Jupyter Lab set out to do with more and better features. I develop scripts that function top to bottom with a notebook side-by-side that on a keyboard stroke executes code blocks from my script in the notebook.
- pqs 7y ago
  For those that do not know it, vscode is great for Jupyter Notebooks. https://code.visualstudio.com/docs/python/jupyter-support https://code.visualstudio.com/docs/python/jupyter-support
rb808 7y ago
I tried to encourage our team to use notebooks, however everyone prefers using PyCharm and git for sharing code. We dont have much visualization, which might be the reason, but I was surprised just how many people just hated it.
- mistrial9 7y ago
  Notebooks are not so much for writing programs or collections of functions.. they are better for a style of "code plus explanation" .. add flexible inline charting for data itself
- ngcc_hk 7y ago
  Are you using oo? Still not sure how to “explain” an oo system once sophisticated enough. Just better than go-to everywhere but not much. Of course a trigger based system (gui, system) also have the same issue. This code + explanation would not work I guess.
  - pjmlp 7y ago
    > Still not sure how to “explain” an oo system once sophisticated enough. With a couple of UML diagrams, still the best option.
omarhaneef 7y ago
Good list. Their observations bring to mind the benefits of watching people program on YouTube or video where you learn a style of working you may not even have considered. However there is one other issue that is not on the list: because a notebook is meant to be read or shared, I always feel like my work is public and feel less inclined to play around and just take a look at things. When I do “transfer” my work to a notebook, it’s only surprising or interesting things that suppress the discovery process.
commandlinefan 7y ago
I don’t get the popularity of these things - I think you have to have started out with them to like them.
- bloaf 7y ago
  I learned to like them in Mathematica, in many ways Jupyter is just a pale imitation of the excellent system built at Wolfram.
thiagomgd 7y ago
Small note: why post an image with the pain points if I need to check the list below to understand what's written?
rossdavidh 7y ago
I think Computational Notebooks are a great idea, and yet I have the feeling that we are in the process of seeing them overapplied. They are wonderful for certain situations, and teaching or demonstrating code to others is right in its sweet spot. I get the impression that people are creeping in the direction of trying to do everything with one tool, which sounds like it would end up in the same swamp that Eclipse went into. Sometimes, you need to use different tools for different tasks, and not everything should integrate. Just my opinion.
xixixao 7y ago
This is a great list, and totally matches my experience. I also agree this is solvable with tooling. A) VS Code / IDE needs to be the primary editor B) Results are not stored with source C) Export (build) allows packaging for whatever platform. Python notebooks especially also use some crazy mutable APIs. In general notebooks align with other code written by people who aren’t usually software engineers building production systems. They’re much more about getting things done, APIs and tools are less questioned, a lot of pain is swallowed because PhDs have plenty of time to write a few lines of code. I don’t want to sound disparaging towards these people, it’s just a different set of tradeoffs from writing production grade software.
- laichzeit0 7y ago
  Logging, monitoring, security, versioning, etc. These are things that most often get ignored due to ignorance or inexperience, but are required for production grade software.
wwarner 7y ago
I have been heads down in jupyter for the past couple of weeks and I finally realized I just DO NOT LIKE IT AT ALL! Cracks started appearing and then suddenly there was an avalanche of disappointment. The first crack -- it's almost impossible to build a nice presentation in Jupyter, because you always have to show your code and its stderr. I imported all the TeX goodness, and it looked pretty nice, but I couldn't show the output without showing the TeX code. Importing the TeX interpreter is quite non-standard and means that my notebook doesn't play well with the public servers. I also got burned by some kind of permissions issue, so that all my charts ended up being invisible to read-only users. The second crack -- I can only look at the code from within my own jupyter server. The source is buried in a very noisy json format. The third crack -- Who wants to write code in the impoverished browser based editor provided? How many times have I deleted a closing brace that was automatically inserted incorrectly? How can I do a global search and replace? The fourth crack -- I can't test my code unless I include all the tests in the notebook! I'm complaining. I realize that I don't have anything constructive to offer, and I'm really a beginner. However, I think some of my disappointment is justified, as I think it was reasonable to assume that I could build my notebooks to be next level presentations.
- mikepurvis 7y ago
  This all sounds very familiar to me. I'm at a robotics company; we had some experimental infrastructure built up around processing ROS bag files via notebooks, and it just eventually became like pulling teeth. Stuff would get cut and pasted between notebooks, or moved out to helper modules which then had versioning and permissions chaos. Each bag needed its own notebook/interpreter instance because there's no way to rerun a notebook on new data, but then the server would explode because of these massive Python processes hanging around with half-processed data state still in them. In the end we dumped it all and turned the good parts into a sane CLI tool which ingests data and dumps out Bokeh plots. At some point we'll throw a Jenkins front end on it, but the current approach seems to be working fine.
- bb88 7y ago
  So once your code gets large enough that it doesn't fit neatly within a jupyter notebook, it's time to split the code out into another package, and then import it into your notebook. The benefit here is that now your code and be used inside the jupyter notebook, and also inside a webserver say.
tdhttt 7y ago
Previous discussion: General: https://news.ycombinator.com/item?id=18336202 https://news.ycombinator.com/item?id=18336202 Version Control: https://news.ycombinator.com/item?id=21661013 https://news.ycombinator.com/item?id=21661013
LifeIsBio 7y ago
I love jupyter notebooks. Without them, I wouldn’t have been half as productive as I was during my PhD. Here’s a post I wrote just a few weeks ago describing some of the conventions that I established for myself over the course of 5 years: https://jessimekirk.com/blog/notebook_rules/ https://jessimekirk.com/blog/notebook_rules/ I suspect that a lot of the conventions I describe help mitigate problems described here, some of which should be strictly or optionally enforced by the notebook instead of the user. (The site’s very much a work in progress, so expect to see odd and broken things if you go poking around.)
- jboynyc 7y ago
  Thanks for this! Didn't know about the watermark extension, that looks useful. I just started working with the Guix kernel for more easily reproducible and reusable notebooks. I suppose that's an alternative to using a conda environment. See here for an example: https://gist.github.com/jboynyc/5d0319f33e71427aa42a98c1a3a915cb https://gist.github.com/jboynyc/5d0319f33e71427aa42a98c1a3a9...
bryanhpchiang 7y ago
I interned @ Google AI last summer; used notebooks nearly everyday. Estimated productivity gain is 3-5x. Biggest tip I have is to turn auto reload on, then write the bulk of your code as modular functions and call functions within your notebooks. Keeps the notebook tidy and it’s easier to push your code this way. It’s also easier for sharing since most people viewing your notebooks (mentors, people outside your team) are interested in results/artifacts such as metrics, generated text, images, audio, which notebooks display well (not your code).
- randomsearch 7y ago
  Beware perceived gains that (1) benefit you at the detriment of others or (2) have hidden costs exposed at a later stage.
- tastyminerals 7y ago
  Yes, notebooks are good for studying.
breatheoften 7y ago
The reality is — notebooks are and need to be developed as an app platform ... In order to do notebooks properly — you need: 1. discovery (Ideally static discovery) of all the state the notebook needs, and the bulk of state the notebook will/could manipulate during its execution. Your container needs to intercept the filesystem and the networking apis that will be invoked so that a determination of the state that results from these operations can be observed by the runtime and shimmed appropriately for reproducibility and for performance optimization 2. The notebook (and the runtime inferred model of all the required inputs) needs to be repo stable — I Should be able to write a notebook app that reads from the file system on my development host, deploy it somewhere, and the runtime should take care that wherever that however that post deployment file system read is implemented matches my local development semantics 3. Pplatform level dependency graph needs to exist to model re-execution requirements automatically — incorporating code changes and external state Apple could build this And “notebook-os” would be the correct conceptual framework for it ... anything less is always going to leave us severely wanting
andrew_n 7y ago
I used Mathematica’s notebook interface quite heavily 15-20 years ago; Jupyter’s interface is a clone of that in many ways. At the time, my workflow was to use two different notebooks for everything: foo.nb and foo-scratch.nb. I’d get things working a piece at a time in foo-scratch.nb, not caring at all how it looked, not having to worry about leaving extra output or dead ends of explorations lying around; then the refined cells would be copied over to foo.nb, which would get pristine presentation, and which I could run top-to-bottom. This workflow worked pretty well for me: very clean reproducible output, with the ability to easily refer back to all the steps of how I’d derived something, along with copious detailed private notes. I never had to use it but I’m pretty sure each cell even had its modification time stored in the metadata in case I wanted to view a chronological history.
- zneveu 7y ago
  I make a "scratch pad" section of my notebook and work on ideas there. Then once I've pieced together a function line by line and tested it a bit I move it up to where it should be in the chronological order of the notebook. Kind of like your two notebook system but makes copying easier in Jupyter.
  - EForEndeavour 7y ago
    I do the same, though it feels dangerous because both the good-copy and scratch sections share the same kernel. JupyterLab works on .ipynb files, and makes it way easier to copy (or drag and drop) cells between different notebooks. One of these days, I plan to switch to JupyterLab to get a sense of what else it offers above Jupyter Notebook.
comment_guy 7y ago
I don't get why anyone one who knows how to use an IDE would ever use a notebook, the coding experience is garbage in comparison. I understand they started as a way to get STEM kids coding quick, but now they are like a standard in data analysis and data science, with those people needing experienced devs to translate the notebook into production code. This just drives the silo walls up higher.
- grp000 7y ago
  Doing data science in an IDE would be terrible. With a notebook, you get the chance to load the data, view it, clean it where needed, view it again, analyze it, model it and do anything else you need to it. An IDE means that you can't use the previous output to guide your next operation in a direct fashion like you can with a notebook.
  - bllguo 7y ago
    as a counterpoint, plenty of R folks are pretty happy doing all of that in Rstudio
    - kyllo 7y ago
      I'm an R folk, and I'm even happy doing all of that in Emacs!
    - meztez 7y ago
      It is interesting to see this discussion about notebooks while I'm thinking about all the RStudio users who do all their work inside the IDE and are pretty happy. Notebooks seem like such an inferior tool to me. I'm also extremely bias.
    - arminiusreturns 7y ago
      Also lots of emacs users of org-mode as an awesome notebook.
  - tastroder 7y ago
    That kind of depends on your process. In many cases pdb (or the debugging interface in your IDE of choice) works just fine for that. It's certainly not "terrible". After the exploration and preprocessing stage I personally don't see much benefit of the notebook model, training/evaluation and any meaningful visualization takes forever anyway, that means I need to cache and persist intermittent results. With that it doesn't really matter all too much if I work on it in vim&pdb, an IDE, or Jupyter.
bloaf 7y ago
So literally all of these complaints are about their particular implementations of notebooks, not the concept of computational notebooks in general, or are all computational notebooks destined to have unstable kernels? In my mind, notebooks should be married to a functional style of programming, where you use the notebook's markup to thoroughly explain and document your functions. Below your "function definition" section, you keep a "trying things out" section where you actually plug the data into your functions for debugging/visualizations. You can't shoot yourself in the foot with variables because all the work is done in your function's lexical scope. You can shoot yourself in the foot with stale function definitions, but a good notebook interface gives you the ability to clear function definitions and run groups of cells, so you can make sure you always run your functions in a group that starts with a "clear function definitions" cell. When you are done, you just cut the "trying things out" section into a second notebook which references the functions in the first and viola, you've got a very well documented library of functions, and a new work notebook where you can freely polish your visualizations/whatever.
pottertheotter 7y ago
I use Jupyter Lab with Python every day. It's where I do my initial data exploration and cleaning. Jupyter Lab is not perfect, but most of these findings seem like they are more issues of inexperience with technology and programming, not computational notebooks.
eanzenberg 7y ago
Notebooks are sort of like democracy, its the worst form of government except all the others. You need to pick the best tool for the job, and often times in machine learning that tool is a notebook.
ngcc_hk 7y ago
What is wrong with life? Many but let us appreciate how to use it more instead of seemingly criticise it. The world is so much better with you alive. So is the founded tool of computational notebook. Not sure I read it covered R notebook which is really good to share info and analyst. Just wonder how to use it better. Of course they can always improve on it. But I would promote more expansion - How about a lisp notebook, a clojure notebook, a js notebook and a forth notebook. The real problem is can you have oo notebook ... it is more “serial” and graphic and data. But not for the “messy” class or trigger Based system. Hence if I may, the real problem is the scoping. It is so hard to visualise a live oo system. Unlike a live functional or even a stack based system. It is not life that is the problem. Even useless life has its use, as long as it is alive. But if it is not reaching there an alternative may have to think about. Just like we cannot be there we send in our voyagers outside solar system. Be long and prosper.
anonsivalley652 7y ago
There are these and other problems with CNs: 0. They try to be "be-all, end-all" proprietary container documents, so they lack generality, compatibility and embeddability. It would be better if live code try-out snippets were self-contained and embeddable in other documents: HTML, other software, maybe PDF, LaTex or literate programming formats. Maybe there should be standard, versioned interpreters for each kind of programming language in WebAssembly and cached for offline usage by the browser for inclusion in documentation, papers, etc.? 1. For prototyping, it is better to have try-out live code (and/or REPLs with undo) for prototyping like what is Xcode/iOS Playgrounds for Swift or ReInteract was for Python. 2. Computational notebook software, that I've seen, are terrible, complex, fragile and messy to install. The ones I've seen make TeXLive look effortless by comparison. 3. Beyond replicability what goal(s) are CN really trying to solve? 3.0. For replicability itself, why not have a GitLab/BitBucket/GitHub repo for code and a Docker/Vagrant container one-liner that grabs the latest source when built? Without a clear, consistent and simple build process, there is no replicability, only wasted time, headaches and fragile/messy results. 3.1. Are CNs "hammers" for "nails" that don't exist?
- anchpop 7y ago
  > Maybe there should be standard, versioned interpreters for each kind of programming language in WebAssembly and cached for offline usage by the browser for inclusion in documentation, papers, etc. This would be incredible. Even better, the output from the code (like graphs) should be able to be embedded in the paper. You have no idea how many papers have errors in the code that generated the graphs/statistics/etc. and nobody can tell because the authors rarely release the data, let alone the source
  - anonsivalley652 7y ago
    For WASM, there ought to be a package-management/registry mechanism for installation (unless there is already? It might get complicated, but would seem a good idea to reuse code/plugins.)... or as below, there ought to be some caching priority mechanism. Then for HTML assets (and CSS ones too), perhaps a hint on asset-linking tags (a, script, link, img, audio, video, etc.) there ought to an offline-priority attribute to help the browser decide what to throw away when clearing cache the regular way or evicting items from the cache, while being able to leave some things deemed vital when not nuking the entire cache. Yes, websites could be goofy and game caching mechanisms, marking everything "vital" like for 0-pixel image cookies but I'm sure someone would make an "RBL" (real-time blackhole list) system of which priorities on which websites to ignore. Related aside: There's a lot of common frameworks, libraries and bits that could be cached user-side, with the trick either to a) herding web devs to de-fragment their CDNs, which could create SPoF's or b) changing the standard allowing multiple SRCs or HREFs for high-availability/less bitrot to preserve both choice and encourage de-duplication of common assets. [0] 0. https://html.spec.whatwg.org/multipage/links.html#attr-hyperlink-href https://html.spec.whatwg.org/multipage/links.html#attr-hyper...
juskrey 7y ago
Why no Mathematica?
- enriquto 7y ago
  It's not even free software.
  - pjmlp 7y ago
    Developers need to pay bills somehow.
    - kragen 7y ago
      "Free software" is about freedom, not price. Putting the research that is your life's work inside of proprietary software that can be taken away from you, forever, at any time — that seems foolish.
      - pjmlp 7y ago
        Copyleft derived "free software" can also be taken away from me.
        kragen 7y ago
        Only if you violate the license. By the way, I'd be interested in your thoughts on https://news.ycombinator.com/item?id=22083468 https://news.ycombinator.com/item?id=22083468.
- fsh 7y ago
  Mathematica is great for symbolic mathematics and terrible for anything else. The awful control flow syntax makes reading longer scripts pretty much impossible. Plotting is very clunky and by default produces output files that are essentially unreadable. Of course one can somehow work around these issues, but it is much easier (and free) to just use python.
cwyers 7y ago
It is interesting to me how this talks about "computational notebooks" but it seems to be about Jupyter and derivatives thereof -- RMarkdown notebooks run inside of the RStudio IDE, and they don't use the term 'kernels' like Jupyter does.
ageofwant 7y ago
For emacs org-mode users this https://github.com/dzop/emacs-jupyter/blob/master/README.org https://github.com/dzop/emacs-jupyter/blob/master/README.org is worth looking into.
fulafel 7y ago
What are the currently available CI options for notebooks? You'd think this would be one of the first tools people would need to make sure notebooks are reproducible, but there seems to be little sign of CI usage.
- amirathi 7y ago
  Checkout treon[1], open source testing framework for Jupyter notebooks. Since it runs via CLI you can hook it up to any CI platform of your choice. Disclaimer: I wrote large part of treon. [1] https://github.com/reviewNB/treon https://github.com/reviewNB/treon
enriquto 7y ago
Jupyter notebooks are great for many purposes. They have, however, two really tragic shortcomings: 1. They are stored by default in stupid json files instead of plain source code with comments. 2. The text editing interface inside the browser is horrific and very difficult to normalize (e.g., disable "smart" closing of parentheses, disable the capture of classic unix copy-pasting, etc).
sunaden 7y ago
I work at https://www.deepnote.com/ https://www.deepnote.com/, we are trying to tackle some of the pains mentioned in the article (setup, collaboration, IDE features like auto-complete or linting). We are still early access, but if you are interested in an invite just let me know. My email is filip at deepnote dot com.
- serendipityisme 7y ago
  Deepnote seems quite interesting, but as a cheapskate grad student, I'm compelled to ask. If this information isn't private, what sort of business model do you use? I take it you'll have a SaaS subscription model? I see it's free to use now, but how does your company plan to make money (especially taking into account the cost of the cloud hosting Deepnote requires)?
  - sunaden 7y ago
    Hey, thanks for the question. Our goal right now is to build the most amazing data science notebook. We need a lot of feedback to get there, that's why we are keeping it free. But since the servers also cost us something, we haven't opened up Deepnote to the public just yet. Once in GA, we know we can support students on a free tier almost indefinitely (it doesn't really cost that much) while offering more advanced features on a subscription model for teams and enterprises.
archi42 7y ago
As a computer scientist/software engineer, please allow me the question: Why would I prefer a notebook over e.g. equivalent python script(s) in a git? I first saw jupyter notebooks when my sister (physicist, non-programmer) used it for analyzing economical data with pandas. Run-time for the full data set was half a day (and IMHO for that analysis SQL would have been better suited). I understand that as a non-programmer it looks alluring, but once the language proficiency is build up, why not use an IDE and run the code on a shell?
- fsh 7y ago
  The main reason is that one has to fiddle with the code a lot and re-running the whole thing is much too slow. A common example is that a huge text file containing experimental data gets parsed in the beginning. Then you have to explore the data step-by-step using all kinds of visualization and analysis such as Fourier transforms, curve fitting, etc. If you simply put everything into one giant python script, for every step you have to re-run the entire thing which takes forever. Of course you can speed things up by writing intermediate results to disk, but this adds tons of boilerplate code and is quite error prone. One alternative would be to write individual scripts for each step and read them into an interactive REPL shell. However, then you still have to somehow record the proper execution order if you ever want to repeat the analysis.
- PeterisP 7y ago
  The key factor is iteration speed. If step A takes 5 minutes (and 5 minutes is a very short time) and I want to experiment on step B, then I don't want to rerun step A each time while I'm writing and running code that helps me understand what step B is going to be; I'd want that to be interactive and immediate, not have each rerun take 5 minutes. Storing/loading to disk is not a good option because all the data that needs to be stored is not yet determined until the exploration is finished; If I write code to save/load A, then I need to change (and test) it after I'm done with B and now want to experiment with C, and it all becomes even more complicated when I need to add an extra step and data field to step A and rerun everything. Deciding what data should be stored in what format is something that you can do in 'productionizing' the code after you've done the exploratory analysis. REPL is not a good option because it's not convenient to save and replicate the code that got you to the current REPL state. The other aspect is that visually 'debugging' intermediary data through various plots is not conveniently possible in IDEs. I could generate some picture files in a folder or possibly an HTML 'dashboard' to see the results of my most recent run but that takes extra code and effort, and the results aren't immediately in my face like in a notebook.
fsh 7y ago
I quite like the Spyder approach: Pure python code that is segmented into cells by inserting a special comment line. The cells can then be individually executed in an ipython shell, or the entire script can be run with the regular python interpreter. This makes it easy to tweak the individual parts without having to re-run everything. In contrast to jupyter notebooks you still end up with a valid python script that can be easily version controlled. I just wish that I could use vim instead of the Spyder IDE.
kdamica 7y ago
Streamlit is imo the best alternative. I was a beta tester and I found that it encouraged good coding practice without sacrificing too much functionality. I highly recommend that other data scientists check it out. Streamlit.io
tardenoisean 7y ago
https://datalore.io https://datalore.io has (1) a reactive Datalore kernel that solves the reproducibility problem. It recalculates the code automatically when something is changed, and recalculates only the changed and the dependent code; (2) good completion; (3) online collaboration; (4) read-only sharing; (5) publishing; (6) sensitive data can be saved in .private directory that is not exposed when the notebook is shared with read-only access
- LeanderK 7y ago
  it seems it's cloud-based. Fun for playing around but not suitable for real work (at least for me). I can't just upload random data to some cloud service to work with it, also I can't upload data if it's too big. Often the data that's valuable is very sensitive.
tastyminerals 7y ago
Notebooks are bad and unreliable. You are repeating your code all the time, you are limited to work with smaller datasets. If you are into visual data analysis use Orange or other similar data mining tools. We allow usage of notebooks only for presentation purposes.
randomsearch 7y ago
For those asking “what’s the alternative”, RStudio and Matlab already solved the design problem (though they could be better executed).
VvR-Ox 7y ago
I would love to see some gifted people using this info to further improve tools like nteract[0]. It already eases a lot of pain you may have in comparison when setting up jupyter notebook without the knowledge of a software developer. [0]: https://nteract.io/ https://nteract.io/
Evidlo 7y ago
I wrote a plugin for ipython that some people might find useful: https://github.com/uiuc-sine/ipython-cells https://github.com/uiuc-sine/ipython-cells It lets you do linear execution of blocks like in Jupyter, but in a normal .py file. Obviously more lightweight than Jupyter and you get to use your regular editor.
Rainymood 7y ago
People love jupyter notebooks for the same reason people love Excel. This is intended to be a Zen-like Koan, so take it as you will.
kriro 7y ago
Direct link to the preprint: http://web.eecs.utk.edu/~azh/pubs/Chattopadhyay2020CHI_NotebookPainpoints.pdf http://web.eecs.utk.edu/~azh/pubs/Chattopadhyay2020CHI_Noteb... Interesting study, I like the mixed-method approach. A quick glance at the industry of the participants suggests that there might be a bias towards structural data (which I think is actually acceptable as that makes up a huge chunk of the non-academic ML-Notebook work) Edit: The authors acknowledge this in the "Limitations".
JosephRedfern 7y ago
I think it's easy to do notebooks wrong, but possible to do them right. I try and do quick prototyping in notebook cells before moving it off to a separate .py file, and avoid keeping any code that does anything other than visualisation or parameter setting inside a cell long-term. That way, if you need to run something "in production" (whatever that means in your context), you don't end up having to pick apart and re-write your code -- you just import the .py file you wrote along the way. For me, notebooks are a super handy way of visualising and sharing results during meetings, and it's difficult to imagine a more convenient alternative.
kriro 7y ago
I'm curious. How do people protocol their experiments? When I started, I used to just keep the cells but that lead to very long and impossible to parse Jupyter notebooks. I have since opted for keeping a journal.txt file in Atom where I write down hpyerparameter configurations, epochs run and results (for ML). But that feels a bit awkward as well.
procrastinatus 7y ago
I’m surprised no one has mentioned what I see as the biggest failings of notebooks: poor handling of connection loss / re-connection. The kernel will continue to run, but a connection hiccup will often make the notebook UI stop updating (and lose any kernel output).
ivan_ah 7y ago
One thing that I find to be incredibly useful is the keyboard shortcut `00` (press zero twice while focus is outside of a cell), which will restart the kernel, clear all output and re-run the whole notebook. This way I'm sure that "library code" that I'm editing in parallel in a real text editor is up to date in the notebook and also solves the limits the confusion due to run-out-of-oder problems. The overall workflow is something like this: 1. explore using thing.<TAB>, thing?, and %psource thing 2. edit draft code chunk or function 3. when chunk 80% done; move it to a module and replace it with an import statement 4. press 00 to re-run everything, then GOTO step 1 The key to preserving sanity is step 3—as soon as the exploration phase is done, move to a real text editor (and start adding tests). Don't try to do big chunks of software development in the notebook. You wouldn't write an entire program in the REPL, would you? Sometimes I keep around the notebook as a record for failed explorations or as a "test harness" for the code, but most of the time it's throwoutable since all the useful bits have moved into a normal python module/script under version control.
- aldanor 7y ago
  Another useful tip which doesn't require always doing '00': when editing the library code, import things like this: import mylib; importlib.reload(mylib); from mylib import foo Then in most cases except some very entangled ones, you can simply rerun this cell without having to restart the kernel (especially if it requires reloading all the data).
desmond373 7y ago
My main use for notebooks is a simple way to constantly hold a whole large dataset in memory. That way if I want to try some feature reduction or remove some bad result, I can just do that and not wait 10 minutes for my slow PC to rerun my import code. I feel like an easy way to do that in base python would draw me away from notebooks.
teekert 7y ago
I don't know what this document is meant to do but you will have to take my Jupyter-lab instance from my dead cold hands. I love notebooks, I work fast, line by line I execute commands and I immediately see the output (dataframes or graphs). For complex code I have an editor open (in jupyter-lab or vscode) for some functions and classes. But the main developing is done in the notebook, anything that ends in a module start in my notebooks. As a biologist that learned to program after 30 I just don't understand how you can develop data processing code without such a close handle on dataframes and without checking in graphs/visualizations if your code does what you expect. I don't see how I would do that in pure vscode of other IDEs. I also don't understand this sentence: "Once the data is loaded, it then has to be cleaned, which participants complained is a repetitive and time consuming task that involves copying and pasting code from their personal "library" of commonly used functions." What is the alternative? Not cleaning the code? And why copy and paste when you can perfectly fine have your own shareable module on the side? I guess most notebook users do some kind of hybrid development.
- grenoire 7y ago
  Good point on the last one; I think we have to 'educate' researchers on the fact that they can also write their own libraries and frameworks, and they should. Even basic data manipulation utilities can be made into Python modules and distributed at ease. If something is tedious, there definitely is a way to make it less so.
voldacar 7y ago
Performance is another pain point, at least for jupyter
ospohngellert 7y ago
I think that all the pain points of the article are a result of not using notebooks for their purpose. In my opinion, notebooks are good for: 1. POC/MVP: Showing that what you want to do will work before making a full structure. 2. Creating PDF/HTML documents with code and output. 3. Exploratory data analysis and visualization. I think many of the data scientists in the article go well beyond what a notebook is. A notebook is where you start, but should never be a production tool.
jupp0r 7y ago
Why not teach data scientists how to write software effectively? Those are smart people, it’s not like using version control, writing unit tests and extracting common code into libraries is rocket science.
zneveu 7y ago
One idea for a pain point not mentioned: better variable persistence. If I declare a variable, then delete the cell I declared it in, the variable persists. I've had this cause issues because if I use the deleted variable by accident, it will work fine right up until a kernel restart.
boomersooner 7y ago
This is akin to reviewing how well a screwdriver drives nails. Yes, it has problems. That doesn't mean it's a bad tool - you're just not using it right. Does it require discipline? Yes, but so does the screwdriver. That being said, I think jupyter specifically has some legacy issues around format, and I prefer R markdown. As much as I love pycharm, it's never going to do more than replicate the notebook experience. IMHO, the main author publishes on code UI/UX, the title seems more like click bait. Not sure why it's so upvoted.
bobbylarrybobby 7y ago
The problem of notebooks has been solved by the Python extension in Visual Studio Code (and some other editors too, although VS Code is the one I'm most familiar with). Editing an ordinary Python file, if you insert the comment "# %%", you turn everything between that comment and the next "# %%" (or the end of the file) into a code cell that can be submitted to the ipython kernel, just as in a Jupyter notebook. The editor splits into two halves, the left half your Python file and the right half the Jupyter notebook window with submitted code and formatted output (e.g., DataFrames look pretty, plots display normally, etc.). When you're done running everything, you can export the result as a Jupyter notebook. Because you're editing an ordinary Python file, standard features like version control and importing the file you're editing into other files (you cannot normally import .ipynb files IIRC) work normally. And of course since VS Code is a real editor/IDE, you can double click a file and have it open right up (no resorting to a Terminal to start your Jupyter session) and you get syntax themes, a built in Terminal, a git UI, code snippets, documentation on hover, vim mode if that's your thing, etc. The only downside I've found is that the Python extension doesn't incorporate ipython's autocomplete in its own autocompletion, but that's a small price to pay for getting to treat .py files as notebooks.
bart_spoon 7y ago
I'm not sure I understand the issue about the user repeatedly tweaking parameters for their data visualization. If anything, that is a reason notebooks are so nice. The repeated tweaks are due to the notebook format, its because that's an inherent part of the data visualization process, where the end result of a particular parameter choice is hard to predict how it will look with a given data set. So the same process would occur whether one was using a notebook or a script, but with a script it becomes much more cumbersome to actually see the result. In a notebook, the parameter tweaking for a data visualization is immediately followed by the result. I definitely agree with most of the other points though.
dkleissas 7y ago
At Gigantum, we're trying to solve some of these issues too. A Gigantum Project lets you run Jupyter or RStudio in a container that is managed for you. Everything is automatically versioned so you can sort out exactly what was run, by who, and when. https://gigantum.com https://gigantum.com

377 points by ashort11 7y ago

See also Joel Grus' talk, "I Don't Like Notebooks": https://www.youtube.com/watch?v=7jiPeIFXb6U https://www.youtube.com/watch?v=7jiPeIFXb6U slides: https://docs.google.com/presentation/d/1n2RlMdmv1p25Xy5thJUhkKGvjtV-dkAIsUXP-AL4ffI/edit https://docs.google.com/presentation/d/1n2RlMdmv1p25Xy5thJUh...

funklute 7y ago
Superb talk! It's worth noting that a lot of the issues he brings up, ultimately stem from the format in which Jupyter notebooks are stored. R notebooks, with their plain-text stored format as well as code-chunk parameters, solve some, but not all, of these problems.
- hprotagonist 7y ago
  https://github.com/mwouts/jupytext https://github.com/mwouts/jupytext doesn’t solve the state management testing or tooling issues though, but commits are slightly less awful.
  - funklute 7y ago
    Yea, when I first found that extension, I was pretty excited about it. But it ultimately is not a first class citizen in the way it is for R notebooks, so I simply don't feel as comfortable using this as I might otherwise have been.

nl 7y ago

The FastAI people have been working on a lot of these issues with their NBDev too: https://www.fast.ai/2019/12/02/nbdev/ https://www.fast.ai/2019/12/02/nbdev/

jph00 7y ago
nbdev deals with the version control and library of code issues pretty effectively.
zneveu 7y ago
Came here to post the same thing. Nbdev helps fill in the strengths that IDEs are traditionally good at. Even if you don't use the full nbdev library and templates, the work flow makes sense. Write code in Jupyter, export to a python library, and you can use it everywhere else after that.

wrnr 7y ago

Trying my best to solve some of these: https://github.com/wrnrlr/foxtrot https://github.com/wrnrlr/foxtrot

moultano 7y ago

I want a notebook where causality can only flow forward through the cells. I hate notebook time-loops where a variable from a deleted cell can still be in scope. 1. Checkpoint the interpreter state after every cell execution. 2. If I edit a cell, roll back to the previous checkpoint and let execution follow from there. I can't tell you how many times I've seen accidental persistence of dead state waste hours of people's time.

johnc1231 7y ago
I haven't dug into it myself, but Netflix makes something called Polynote that is supposed to add some awareness of the sequence of the cells to combat this
ivalm 7y ago
If the interpreter state contains large variables checkpointing might not be viable (eg I have dataframes that are 100s of GB/large fractions of total available memory, reading/writing from hard drive all the time would be relatively slow. If you can save deltas I guess it wouldn't be too space inefficient but I imagine still slow). At the same time, I do like the idea of an append only notebook where you can: 1. Only run cells in sequential order 2. Only edit cells that are below the most recently ran cell. Thankfully you can enforce it through code practice and the notebook is relatively guaranteed to be "run all"-able. You will need to refactor it after the initial dirty run, but at least it's easy to reason about.
- roblabla 7y ago
  Assuming the data isn't changed, thanks to CoW forking wouldn't cause any extra memory usage. If only a subset of data is changed, same thing - only the changed cells will take extra space. The problem only occurs when the whole variable changes - in which case yeah, you're SOL. I wonder what the usage patterns are for such datasets?
  - ivalm 7y ago
    Personal experience: when first looking at the data I often do lots of map /reduce style operations which might transform large portions of the dataframe. Question, if you use CoW then presumably your variable blocks are no longer contiguous, wouldn't this really slow down vector operations?

stared 7y ago

(A frequent Jupyter Notebook user here. For data exploration, and teaching deep learning - then Colab is indispensable.) The main question is: what are the alternatives, for data exploration (and sharing its results). Similarly, for data science tool demos, Notebooks shine. IMHO the problem is not in the notebooks, but in how they are being used (i.e. the workflow). By writing scripts in py files, and using notebooks only to show their results (processed data, charts, etc) we get the best of both worlds. The only build-in problem with Jupyter Notebooks is JSON, mixing input and output (and making it pain to work with version control). But here RMarkdown (and a few other alternatives) work well.

fhennig 7y ago
Yes, the article mentions users copy pasting snippets from their personal "library". Well, that could just be made into an actual library of functions to call. I'm currently at uni enrolled in an AI/ML degree, and there are a lot of people with no previous exposure to programming. It's just that most people don't know that these things are possible, don't want to learn another tool (IDE) and are not interested in longevity of the code, just in the results. This shouldn't sound like me complaining, I totally understand. I think a lot of the stuff could be solved with just better tooling, but a familiarity with software development is definitely helpful. Also a while back streamlit (https://www.streamlit.io/ https://www.streamlit.io/) was here on HN and since then I've been meaning to try it. I think this could be a good approach to bring together the best of both worlds.
- sillysaurusx 7y ago
  most people don't know that these things are possible, don't want to learn another tool (IDE) and are not interested in longevity of the code, just in the results. This approach is arguably more effective than wasting time trying to refactor everything into a library of functions. Programming-as-crafting needs to be more of a thing. Not everything is written to be long-lasting. Even HN ushered the ugly code into hook functions that weren't shipped with the main codebase.
karlicoss 7y ago

rabryan35 7y ago

No mention of https://observablehq.com https://observablehq.com notebooks? They’re the best I’ve found in the “Share and collaborate” and “As products” category. JupyterLab is still pretty great for exploratory stuff, but visualization possibilities in observable are incredible.

azhenley 7y ago
No one we interviewed or surveyed mentioned it.
goatlover 7y ago
Problem is that Javascript doesn't have the scientific computing ecosystem that Python, R and Julia have. Jupyter supports those languages and any others that people write kernels for. And you can also execute bash, JS, CSS and HTML directly in python notebooks with magic commands.
tomgp 7y ago
Agreed, Observable fixes a lot of the problems I've had with other notebooks. It can still be fiddly for code over a certain size/ complexity but the ability to import from npm modules goes a long way to fixing the problem. The user base seems to be predominantly drawn from the visualisation side of things + the fact that it's javascript may limit its uptake in science/maths areas. Aside: I've felt for a while that JS is really missing decent maths/stats libraries, any suggestions?
btbuildem 7y ago
They're a walled garden and to be honest I kinda hate them. Bostock had lots of accessible D3 examples before, now it's all on that "platform".. sure it's slick, but overall loss for the ecosystem IMO.

evrydayhustling 7y ago

This is a solid list. It will be even better if juxtaposed with current efforts to solve each of these problems - every DS I know is addressing at least 2-3 of these with some pet tools in their own environment. For example, we use Panel and Holoviews to make data exploration much easier. I have a feeling the ecosystem would improve faster if we had an index of (partial) solutions aligned with this problem set. One category left out of the list: testing of data pipelines (c.f. great expectations).

azhenley 7y ago

Co-author of the study here. Let me know if you have any questions or how you overcome some of the problems we identified!

autokad 7y ago
As a data scientist, I used all of the notebooks and didn't find any of the problems listed with databricks. I don't get to use it in my current role, miss it a lot.
snapetom 7y ago
I just wanted to say thank you. Many of the points in your study strikes a nerve. Part of my responsibility at my last job was to introduce good software engineering practices. What happens? The data scientists go rogue and start running notebooks left and right. How do they productionize their work? Well, they don't. They were academics. All they know is that the models ran fine in their notebooks on their laptops. Meanwhile, we didn't have anyone that was devoted full time on model productionization. Sharing data? They had enough problems sharing their notebooks.
- ssivark 7y ago
  I just happened to be reading Peter Naur's "Programming as theory building" recently. It strikes me that taking its theme even a little seriously helps understand why notebooks are so popular. Notebooks happen to be convenient tools for exploring a new domain (interactively). Irrespective of how much software purists might complain, conventional software engineering provides very few tools/solutions/practices for that process. The wretched state of interactive debugging (in most languages) is a simple example. As someone who spends a substantial amount of time working with both modes (writing research code in Jupyter notebooks, and writing production code as python modules), notebooks scratch certain itches that IDEs typically don't even come close to. (Some recent progress on add-ons in Javascript-based editors is potentially interesting, because that might help marry the strengths of the two) In my experience, in the evolution of code from Jupyter notebooks to repositories of production code as part of any project, there comes a "right time" to switch from the former to the latter. And this can typically only be learned with experience.

ktpsns 7y ago

Obviously it's pretty hard to make general criticism of the Notebook GUI. This is especially without comparing to a specific other user interface for data scientists, such as a traditional REPL terminal, or some other command line tools? The Python world gives a good example about the sheer complexity of a notebook infrastructure. The is IPython, there is Jupyter Notebook, JupyterLab. There is even stuff like the SageMathCloud (nowadays called CoCalc) which is basically a web GUI to a VPS combining command lines and various notebooks. And hell, most of these web based interfaces try to make sharing easy. Mabye we should start comparing these (mostly OSS) tools to the traditional notebook GUIs of Matlab and Mathematica, something we used in the 90s and 2000s. From my feeling, they were more robust, could handle large data better, but they lack all the tooling we get for free in the web.

nl 7y ago
Jupyter Notebook/Jupyter Lab has replaced IPython as the notebook front end. I suspect 90%+ of Python Notebook work is done in Jupyter/Jupyter Lab (or things built on it like Google Collab/Kaggle Kernels). traditional notebook GUIs of Matlab and Mathematica, something we used in the 90s and 2000s. From my feeling, they were more robust, could handle large data better I've done 10s of terabyte analysis on Jupyter (Spark backend) and I personally know people doing petabyte work on it so this seems doubtful.
- ktpsns 7y ago
  You probably were careful enough to understand the limits of the Jupyter server and client (frontend). It's easy to screw up a terminal application in data science when dumping a large array. Many REPLs cannot handle this properly (and CTRL+C won't work). It's easy to test this: What does your favourite notebook do when you call some command such as (pseudocode/python here) print(range(int(1e7))) # or 1e8 In this particular example, the python CLI seems to handle keyboard interrupts fine when the terminal (or RAM) is flooded.

rcar 7y ago

I'm someone who has been programming for a very long time and has been using notebooks for a reasonably long time (and almost always starts projects with them), my feeling is that they are a bit like C in that they make it easy to accidentally shoot yourself in the foot if you aren't careful. I always strive to end up with a notebook that can be "Run All" from a fresh clone, and I'd say that I'm successful with that maybe 60-70% of the time, and am close enough that I can fix it in the remainder. As the article (and the many others like it that have frequently cropped up as soon as IPython Notebooks first started ramping up in popularity) points out though, a lot of newer users don't have the discipline to ensure that they're not jumping around too much. It's not a problem for them in the immediate term since they know how the state ought to work, but then it becomes a mess when they try to share it with someone else (or to run it themselves again 3 months later). The challenge though is that the data analysis workflows that it allows are unbeatable by any other tools I've tried. In the end, it may just be that it's the worst form of data programming except for all of the others that have been tried.

tbenst 7y ago

I think Atom’s hydrogen and VSCode’s python are best-in-class Jupyter clients that achieve everything Jupyter Lab set out to do with more and better features. I develop scripts that function top to bottom with a notebook side-by-side that on a keyboard stroke executes code blocks from my script in the notebook.

pqs 7y ago
For those that do not know it, vscode is great for Jupyter Notebooks. https://code.visualstudio.com/docs/python/jupyter-support https://code.visualstudio.com/docs/python/jupyter-support

rb808 7y ago

I tried to encourage our team to use notebooks, however everyone prefers using PyCharm and git for sharing code. We dont have much visualization, which might be the reason, but I was surprised just how many people just hated it.

mistrial9 7y ago
Notebooks are not so much for writing programs or collections of functions.. they are better for a style of "code plus explanation" .. add flexible inline charting for data itself
ngcc_hk 7y ago
Are you using oo? Still not sure how to “explain” an oo system once sophisticated enough. Just better than go-to everywhere but not much. Of course a trigger based system (gui, system) also have the same issue. This code + explanation would not work I guess.
- pjmlp 7y ago
  > Still not sure how to “explain” an oo system once sophisticated enough. With a couple of UML diagrams, still the best option.

omarhaneef 7y ago

Good list. Their observations bring to mind the benefits of watching people program on YouTube or video where you learn a style of working you may not even have considered. However there is one other issue that is not on the list: because a notebook is meant to be read or shared, I always feel like my work is public and feel less inclined to play around and just take a look at things. When I do “transfer” my work to a notebook, it’s only surprising or interesting things that suppress the discovery process.

commandlinefan 7y ago

I don’t get the popularity of these things - I think you have to have started out with them to like them.

bloaf 7y ago
I learned to like them in Mathematica, in many ways Jupyter is just a pale imitation of the excellent system built at Wolfram.

thiagomgd 7y ago

Small note: why post an image with the pain points if I need to check the list below to understand what's written?

rossdavidh 7y ago

I think Computational Notebooks are a great idea, and yet I have the feeling that we are in the process of seeing them overapplied. They are wonderful for certain situations, and teaching or demonstrating code to others is right in its sweet spot. I get the impression that people are creeping in the direction of trying to do everything with one tool, which sounds like it would end up in the same swamp that Eclipse went into. Sometimes, you need to use different tools for different tasks, and not everything should integrate. Just my opinion.

xixixao 7y ago

This is a great list, and totally matches my experience. I also agree this is solvable with tooling. A) VS Code / IDE needs to be the primary editor B) Results are not stored with source C) Export (build) allows packaging for whatever platform. Python notebooks especially also use some crazy mutable APIs. In general notebooks align with other code written by people who aren’t usually software engineers building production systems. They’re much more about getting things done, APIs and tools are less questioned, a lot of pain is swallowed because PhDs have plenty of time to write a few lines of code. I don’t want to sound disparaging towards these people, it’s just a different set of tradeoffs from writing production grade software.

laichzeit0 7y ago
Logging, monitoring, security, versioning, etc. These are things that most often get ignored due to ignorance or inexperience, but are required for production grade software.

wwarner 7y ago

I have been heads down in jupyter for the past couple of weeks and I finally realized I just DO NOT LIKE IT AT ALL! Cracks started appearing and then suddenly there was an avalanche of disappointment. The first crack -- it's almost impossible to build a nice presentation in Jupyter, because you always have to show your code and its stderr. I imported all the TeX goodness, and it looked pretty nice, but I couldn't show the output without showing the TeX code. Importing the TeX interpreter is quite non-standard and means that my notebook doesn't play well with the public servers. I also got burned by some kind of permissions issue, so that all my charts ended up being invisible to read-only users. The second crack -- I can only look at the code from within my own jupyter server. The source is buried in a very noisy json format. The third crack -- Who wants to write code in the impoverished browser based editor provided? How many times have I deleted a closing brace that was automatically inserted incorrectly? How can I do a global search and replace? The fourth crack -- I can't test my code unless I include all the tests in the notebook! I'm complaining. I realize that I don't have anything constructive to offer, and I'm really a beginner. However, I think some of my disappointment is justified, as I think it was reasonable to assume that I could build my notebooks to be next level presentations.

mikepurvis 7y ago
This all sounds very familiar to me. I'm at a robotics company; we had some experimental infrastructure built up around processing ROS bag files via notebooks, and it just eventually became like pulling teeth. Stuff would get cut and pasted between notebooks, or moved out to helper modules which then had versioning and permissions chaos. Each bag needed its own notebook/interpreter instance because there's no way to rerun a notebook on new data, but then the server would explode because of these massive Python processes hanging around with half-processed data state still in them. In the end we dumped it all and turned the good parts into a sane CLI tool which ingests data and dumps out Bokeh plots. At some point we'll throw a Jenkins front end on it, but the current approach seems to be working fine.
bb88 7y ago
So once your code gets large enough that it doesn't fit neatly within a jupyter notebook, it's time to split the code out into another package, and then import it into your notebook. The benefit here is that now your code and be used inside the jupyter notebook, and also inside a webserver say.

tdhttt 7y ago

Previous discussion: General: https://news.ycombinator.com/item?id=18336202 https://news.ycombinator.com/item?id=18336202 Version Control: https://news.ycombinator.com/item?id=21661013 https://news.ycombinator.com/item?id=21661013

LifeIsBio 7y ago

I love jupyter notebooks. Without them, I wouldn’t have been half as productive as I was during my PhD. Here’s a post I wrote just a few weeks ago describing some of the conventions that I established for myself over the course of 5 years: https://jessimekirk.com/blog/notebook_rules/ https://jessimekirk.com/blog/notebook_rules/ I suspect that a lot of the conventions I describe help mitigate problems described here, some of which should be strictly or optionally enforced by the notebook instead of the user. (The site’s very much a work in progress, so expect to see odd and broken things if you go poking around.)

jboynyc 7y ago
Thanks for this! Didn't know about the watermark extension, that looks useful. I just started working with the Guix kernel for more easily reproducible and reusable notebooks. I suppose that's an alternative to using a conda environment. See here for an example: https://gist.github.com/jboynyc/5d0319f33e71427aa42a98c1a3a915cb https://gist.github.com/jboynyc/5d0319f33e71427aa42a98c1a3a9...

bryanhpchiang 7y ago

I interned @ Google AI last summer; used notebooks nearly everyday. Estimated productivity gain is 3-5x. Biggest tip I have is to turn auto reload on, then write the bulk of your code as modular functions and call functions within your notebooks. Keeps the notebook tidy and it’s easier to push your code this way. It’s also easier for sharing since most people viewing your notebooks (mentors, people outside your team) are interested in results/artifacts such as metrics, generated text, images, audio, which notebooks display well (not your code).

randomsearch 7y ago
Beware perceived gains that (1) benefit you at the detriment of others or (2) have hidden costs exposed at a later stage.
tastyminerals 7y ago
Yes, notebooks are good for studying.

breatheoften 7y ago

The reality is — notebooks are and need to be developed as an app platform ... In order to do notebooks properly — you need: 1. discovery (Ideally static discovery) of all the state the notebook needs, and the bulk of state the notebook will/could manipulate during its execution. Your container needs to intercept the filesystem and the networking apis that will be invoked so that a determination of the state that results from these operations can be observed by the runtime and shimmed appropriately for reproducibility and for performance optimization 2. The notebook (and the runtime inferred model of all the required inputs) needs to be repo stable — I Should be able to write a notebook app that reads from the file system on my development host, deploy it somewhere, and the runtime should take care that wherever that however that post deployment file system read is implemented matches my local development semantics 3. Pplatform level dependency graph needs to exist to model re-execution requirements automatically — incorporating code changes and external state Apple could build this And “notebook-os” would be the correct conceptual framework for it ... anything less is always going to leave us severely wanting

andrew_n 7y ago

I used Mathematica’s notebook interface quite heavily 15-20 years ago; Jupyter’s interface is a clone of that in many ways. At the time, my workflow was to use two different notebooks for everything: foo.nb and foo-scratch.nb. I’d get things working a piece at a time in foo-scratch.nb, not caring at all how it looked, not having to worry about leaving extra output or dead ends of explorations lying around; then the refined cells would be copied over to foo.nb, which would get pristine presentation, and which I could run top-to-bottom. This workflow worked pretty well for me: very clean reproducible output, with the ability to easily refer back to all the steps of how I’d derived something, along with copious detailed private notes. I never had to use it but I’m pretty sure each cell even had its modification time stored in the metadata in case I wanted to view a chronological history.

zneveu 7y ago
I make a "scratch pad" section of my notebook and work on ideas there. Then once I've pieced together a function line by line and tested it a bit I move it up to where it should be in the chronological order of the notebook. Kind of like your two notebook system but makes copying easier in Jupyter.
- EForEndeavour 7y ago
  I do the same, though it feels dangerous because both the good-copy and scratch sections share the same kernel. JupyterLab works on .ipynb files, and makes it way easier to copy (or drag and drop) cells between different notebooks. One of these days, I plan to switch to JupyterLab to get a sense of what else it offers above Jupyter Notebook.

comment_guy 7y ago

I don't get why anyone one who knows how to use an IDE would ever use a notebook, the coding experience is garbage in comparison. I understand they started as a way to get STEM kids coding quick, but now they are like a standard in data analysis and data science, with those people needing experienced devs to translate the notebook into production code. This just drives the silo walls up higher.

grp000 7y ago
Doing data science in an IDE would be terrible. With a notebook, you get the chance to load the data, view it, clean it where needed, view it again, analyze it, model it and do anything else you need to it. An IDE means that you can't use the previous output to guide your next operation in a direct fashion like you can with a notebook.
- bllguo 7y ago
  as a counterpoint, plenty of R folks are pretty happy doing all of that in Rstudio
  - kyllo 7y ago
    I'm an R folk, and I'm even happy doing all of that in Emacs!
  - meztez 7y ago
    It is interesting to see this discussion about notebooks while I'm thinking about all the RStudio users who do all their work inside the IDE and are pretty happy. Notebooks seem like such an inferior tool to me. I'm also extremely bias.
  - arminiusreturns 7y ago
    Also lots of emacs users of org-mode as an awesome notebook.
- tastroder 7y ago
  That kind of depends on your process. In many cases pdb (or the debugging interface in your IDE of choice) works just fine for that. It's certainly not "terrible". After the exploration and preprocessing stage I personally don't see much benefit of the notebook model, training/evaluation and any meaningful visualization takes forever anyway, that means I need to cache and persist intermittent results. With that it doesn't really matter all too much if I work on it in vim&pdb, an IDE, or Jupyter.

bloaf 7y ago

So literally all of these complaints are about their particular implementations of notebooks, not the concept of computational notebooks in general, or are all computational notebooks destined to have unstable kernels? In my mind, notebooks should be married to a functional style of programming, where you use the notebook's markup to thoroughly explain and document your functions. Below your "function definition" section, you keep a "trying things out" section where you actually plug the data into your functions for debugging/visualizations. You can't shoot yourself in the foot with variables because all the work is done in your function's lexical scope. You can shoot yourself in the foot with stale function definitions, but a good notebook interface gives you the ability to clear function definitions and run groups of cells, so you can make sure you always run your functions in a group that starts with a "clear function definitions" cell. When you are done, you just cut the "trying things out" section into a second notebook which references the functions in the first and viola, you've got a very well documented library of functions, and a new work notebook where you can freely polish your visualizations/whatever.

pottertheotter 7y ago

I use Jupyter Lab with Python every day. It's where I do my initial data exploration and cleaning. Jupyter Lab is not perfect, but most of these findings seem like they are more issues of inexperience with technology and programming, not computational notebooks.

eanzenberg 7y ago

Notebooks are sort of like democracy, its the worst form of government except all the others. You need to pick the best tool for the job, and often times in machine learning that tool is a notebook.

ngcc_hk 7y ago

What is wrong with life? Many but let us appreciate how to use it more instead of seemingly criticise it. The world is so much better with you alive. So is the founded tool of computational notebook. Not sure I read it covered R notebook which is really good to share info and analyst. Just wonder how to use it better. Of course they can always improve on it. But I would promote more expansion - How about a lisp notebook, a clojure notebook, a js notebook and a forth notebook. The real problem is can you have oo notebook ... it is more “serial” and graphic and data. But not for the “messy” class or trigger Based system. Hence if I may, the real problem is the scoping. It is so hard to visualise a live oo system. Unlike a live functional or even a stack based system. It is not life that is the problem. Even useless life has its use, as long as it is alive. But if it is not reaching there an alternative may have to think about. Just like we cannot be there we send in our voyagers outside solar system. Be long and prosper.

anonsivalley652 7y ago

There are these and other problems with CNs: 0. They try to be "be-all, end-all" proprietary container documents, so they lack generality, compatibility and embeddability. It would be better if live code try-out snippets were self-contained and embeddable in other documents: HTML, other software, maybe PDF, LaTex or literate programming formats. Maybe there should be standard, versioned interpreters for each kind of programming language in WebAssembly and cached for offline usage by the browser for inclusion in documentation, papers, etc.? 1. For prototyping, it is better to have try-out live code (and/or REPLs with undo) for prototyping like what is Xcode/iOS Playgrounds for Swift or ReInteract was for Python. 2. Computational notebook software, that I've seen, are terrible, complex, fragile and messy to install. The ones I've seen make TeXLive look effortless by comparison. 3. Beyond replicability what goal(s) are CN really trying to solve? 3.0. For replicability itself, why not have a GitLab/BitBucket/GitHub repo for code and a Docker/Vagrant container one-liner that grabs the latest source when built? Without a clear, consistent and simple build process, there is no replicability, only wasted time, headaches and fragile/messy results. 3.1. Are CNs "hammers" for "nails" that don't exist?

anchpop 7y ago
> Maybe there should be standard, versioned interpreters for each kind of programming language in WebAssembly and cached for offline usage by the browser for inclusion in documentation, papers, etc. This would be incredible. Even better, the output from the code (like graphs) should be able to be embedded in the paper. You have no idea how many papers have errors in the code that generated the graphs/statistics/etc. and nobody can tell because the authors rarely release the data, let alone the source
- anonsivalley652 7y ago
  For WASM, there ought to be a package-management/registry mechanism for installation (unless there is already? It might get complicated, but would seem a good idea to reuse code/plugins.)... or as below, there ought to be some caching priority mechanism. Then for HTML assets (and CSS ones too), perhaps a hint on asset-linking tags (a, script, link, img, audio, video, etc.) there ought to an offline-priority attribute to help the browser decide what to throw away when clearing cache the regular way or evicting items from the cache, while being able to leave some things deemed vital when not nuking the entire cache. Yes, websites could be goofy and game caching mechanisms, marking everything "vital" like for 0-pixel image cookies but I'm sure someone would make an "RBL" (real-time blackhole list) system of which priorities on which websites to ignore. Related aside: There's a lot of common frameworks, libraries and bits that could be cached user-side, with the trick either to a) herding web devs to de-fragment their CDNs, which could create SPoF's or b) changing the standard allowing multiple SRCs or HREFs for high-availability/less bitrot to preserve both choice and encourage de-duplication of common assets. [0] 0. https://html.spec.whatwg.org/multipage/links.html#attr-hyperlink-href https://html.spec.whatwg.org/multipage/links.html#attr-hyper...

juskrey 7y ago

Why no Mathematica?

enriquto 7y ago
It's not even free software.
- pjmlp 7y ago
  Developers need to pay bills somehow.
  - kragen 7y ago
    "Free software" is about freedom, not price. Putting the research that is your life's work inside of proprietary software that can be taken away from you, forever, at any time — that seems foolish.
    - pjmlp 7y ago
      Copyleft derived "free software" can also be taken away from me.
      - kragen 7y ago
        Only if you violate the license. By the way, I'd be interested in your thoughts on https://news.ycombinator.com/item?id=22083468 https://news.ycombinator.com/item?id=22083468.
fsh 7y ago
Mathematica is great for symbolic mathematics and terrible for anything else. The awful control flow syntax makes reading longer scripts pretty much impossible. Plotting is very clunky and by default produces output files that are essentially unreadable. Of course one can somehow work around these issues, but it is much easier (and free) to just use python.

cwyers 7y ago

It is interesting to me how this talks about "computational notebooks" but it seems to be about Jupyter and derivatives thereof -- RMarkdown notebooks run inside of the RStudio IDE, and they don't use the term 'kernels' like Jupyter does.

ageofwant 7y ago

For emacs org-mode users this https://github.com/dzop/emacs-jupyter/blob/master/README.org https://github.com/dzop/emacs-jupyter/blob/master/README.org is worth looking into.

fulafel 7y ago

What are the currently available CI options for notebooks? You'd think this would be one of the first tools people would need to make sure notebooks are reproducible, but there seems to be little sign of CI usage.

amirathi 7y ago
Checkout treon[1], open source testing framework for Jupyter notebooks. Since it runs via CLI you can hook it up to any CI platform of your choice. Disclaimer: I wrote large part of treon. [1] https://github.com/reviewNB/treon https://github.com/reviewNB/treon

enriquto 7y ago

Jupyter notebooks are great for many purposes. They have, however, two really tragic shortcomings: 1. They are stored by default in stupid json files instead of plain source code with comments. 2. The text editing interface inside the browser is horrific and very difficult to normalize (e.g., disable "smart" closing of parentheses, disable the capture of classic unix copy-pasting, etc).

sunaden 7y ago

I work at https://www.deepnote.com/ https://www.deepnote.com/, we are trying to tackle some of the pains mentioned in the article (setup, collaboration, IDE features like auto-complete or linting). We are still early access, but if you are interested in an invite just let me know. My email is filip at deepnote dot com.

serendipityisme 7y ago
Deepnote seems quite interesting, but as a cheapskate grad student, I'm compelled to ask. If this information isn't private, what sort of business model do you use? I take it you'll have a SaaS subscription model? I see it's free to use now, but how does your company plan to make money (especially taking into account the cost of the cloud hosting Deepnote requires)?
- sunaden 7y ago
  Hey, thanks for the question. Our goal right now is to build the most amazing data science notebook. We need a lot of feedback to get there, that's why we are keeping it free. But since the servers also cost us something, we haven't opened up Deepnote to the public just yet. Once in GA, we know we can support students on a free tier almost indefinitely (it doesn't really cost that much) while offering more advanced features on a subscription model for teams and enterprises.

archi42 7y ago

As a computer scientist/software engineer, please allow me the question: Why would I prefer a notebook over e.g. equivalent python script(s) in a git? I first saw jupyter notebooks when my sister (physicist, non-programmer) used it for analyzing economical data with pandas. Run-time for the full data set was half a day (and IMHO for that analysis SQL would have been better suited). I understand that as a non-programmer it looks alluring, but once the language proficiency is build up, why not use an IDE and run the code on a shell?

fsh 7y ago
The main reason is that one has to fiddle with the code a lot and re-running the whole thing is much too slow. A common example is that a huge text file containing experimental data gets parsed in the beginning. Then you have to explore the data step-by-step using all kinds of visualization and analysis such as Fourier transforms, curve fitting, etc. If you simply put everything into one giant python script, for every step you have to re-run the entire thing which takes forever. Of course you can speed things up by writing intermediate results to disk, but this adds tons of boilerplate code and is quite error prone. One alternative would be to write individual scripts for each step and read them into an interactive REPL shell. However, then you still have to somehow record the proper execution order if you ever want to repeat the analysis.
PeterisP 7y ago
The key factor is iteration speed. If step A takes 5 minutes (and 5 minutes is a very short time) and I want to experiment on step B, then I don't want to rerun step A each time while I'm writing and running code that helps me understand what step B is going to be; I'd want that to be interactive and immediate, not have each rerun take 5 minutes. Storing/loading to disk is not a good option because all the data that needs to be stored is not yet determined until the exploration is finished; If I write code to save/load A, then I need to change (and test) it after I'm done with B and now want to experiment with C, and it all becomes even more complicated when I need to add an extra step and data field to step A and rerun everything. Deciding what data should be stored in what format is something that you can do in 'productionizing' the code after you've done the exploratory analysis. REPL is not a good option because it's not convenient to save and replicate the code that got you to the current REPL state. The other aspect is that visually 'debugging' intermediary data through various plots is not conveniently possible in IDEs. I could generate some picture files in a folder or possibly an HTML 'dashboard' to see the results of my most recent run but that takes extra code and effort, and the results aren't immediately in my face like in a notebook.

fsh 7y ago

I quite like the Spyder approach: Pure python code that is segmented into cells by inserting a special comment line. The cells can then be individually executed in an ipython shell, or the entire script can be run with the regular python interpreter. This makes it easy to tweak the individual parts without having to re-run everything. In contrast to jupyter notebooks you still end up with a valid python script that can be easily version controlled. I just wish that I could use vim instead of the Spyder IDE.

kdamica 7y ago

Streamlit is imo the best alternative. I was a beta tester and I found that it encouraged good coding practice without sacrificing too much functionality. I highly recommend that other data scientists check it out. Streamlit.io

tardenoisean 7y ago

https://datalore.io https://datalore.io has (1) a reactive Datalore kernel that solves the reproducibility problem. It recalculates the code automatically when something is changed, and recalculates only the changed and the dependent code; (2) good completion; (3) online collaboration; (4) read-only sharing; (5) publishing; (6) sensitive data can be saved in .private directory that is not exposed when the notebook is shared with read-only access

LeanderK 7y ago
it seems it's cloud-based. Fun for playing around but not suitable for real work (at least for me). I can't just upload random data to some cloud service to work with it, also I can't upload data if it's too big. Often the data that's valuable is very sensitive.

tastyminerals 7y ago

Notebooks are bad and unreliable. You are repeating your code all the time, you are limited to work with smaller datasets. If you are into visual data analysis use Orange or other similar data mining tools. We allow usage of notebooks only for presentation purposes.

randomsearch 7y ago

For those asking “what’s the alternative”, RStudio and Matlab already solved the design problem (though they could be better executed).

VvR-Ox 7y ago

I would love to see some gifted people using this info to further improve tools like nteract[0]. It already eases a lot of pain you may have in comparison when setting up jupyter notebook without the knowledge of a software developer. [0]: https://nteract.io/ https://nteract.io/

Evidlo 7y ago

I wrote a plugin for ipython that some people might find useful: https://github.com/uiuc-sine/ipython-cells https://github.com/uiuc-sine/ipython-cells It lets you do linear execution of blocks like in Jupyter, but in a normal .py file. Obviously more lightweight than Jupyter and you get to use your regular editor.

Rainymood 7y ago

People love jupyter notebooks for the same reason people love Excel. This is intended to be a Zen-like Koan, so take it as you will.

kriro 7y ago

Direct link to the preprint: http://web.eecs.utk.edu/~azh/pubs/Chattopadhyay2020CHI_NotebookPainpoints.pdf http://web.eecs.utk.edu/~azh/pubs/Chattopadhyay2020CHI_Noteb... Interesting study, I like the mixed-method approach. A quick glance at the industry of the participants suggests that there might be a bias towards structural data (which I think is actually acceptable as that makes up a huge chunk of the non-academic ML-Notebook work) Edit: The authors acknowledge this in the "Limitations".

JosephRedfern 7y ago

I think it's easy to do notebooks wrong, but possible to do them right. I try and do quick prototyping in notebook cells before moving it off to a separate .py file, and avoid keeping any code that does anything other than visualisation or parameter setting inside a cell long-term. That way, if you need to run something "in production" (whatever that means in your context), you don't end up having to pick apart and re-write your code -- you just import the .py file you wrote along the way. For me, notebooks are a super handy way of visualising and sharing results during meetings, and it's difficult to imagine a more convenient alternative.

kriro 7y ago

I'm curious. How do people protocol their experiments? When I started, I used to just keep the cells but that lead to very long and impossible to parse Jupyter notebooks. I have since opted for keeping a journal.txt file in Atom where I write down hpyerparameter configurations, epochs run and results (for ML). But that feels a bit awkward as well.

procrastinatus 7y ago

I’m surprised no one has mentioned what I see as the biggest failings of notebooks: poor handling of connection loss / re-connection. The kernel will continue to run, but a connection hiccup will often make the notebook UI stop updating (and lose any kernel output).

ivan_ah 7y ago

One thing that I find to be incredibly useful is the keyboard shortcut `00` (press zero twice while focus is outside of a cell), which will restart the kernel, clear all output and re-run the whole notebook. This way I'm sure that "library code" that I'm editing in parallel in a real text editor is up to date in the notebook and also solves the limits the confusion due to run-out-of-oder problems. The overall workflow is something like this: 1. explore using thing.<TAB>, thing?, and %psource thing 2. edit draft code chunk or function 3. when chunk 80% done; move it to a module and replace it with an import statement 4. press 00 to re-run everything, then GOTO step 1 The key to preserving sanity is step 3—as soon as the exploration phase is done, move to a real text editor (and start adding tests). Don't try to do big chunks of software development in the notebook. You wouldn't write an entire program in the REPL, would you? Sometimes I keep around the notebook as a record for failed explorations or as a "test harness" for the code, but most of the time it's throwoutable since all the useful bits have moved into a normal python module/script under version control.

aldanor 7y ago
Another useful tip which doesn't require always doing '00': when editing the library code, import things like this: import mylib; importlib.reload(mylib); from mylib import foo Then in most cases except some very entangled ones, you can simply rerun this cell without having to restart the kernel (especially if it requires reloading all the data).

desmond373 7y ago

My main use for notebooks is a simple way to constantly hold a whole large dataset in memory. That way if I want to try some feature reduction or remove some bad result, I can just do that and not wait 10 minutes for my slow PC to rerun my import code. I feel like an easy way to do that in base python would draw me away from notebooks.

teekert 7y ago

I don't know what this document is meant to do but you will have to take my Jupyter-lab instance from my dead cold hands. I love notebooks, I work fast, line by line I execute commands and I immediately see the output (dataframes or graphs). For complex code I have an editor open (in jupyter-lab or vscode) for some functions and classes. But the main developing is done in the notebook, anything that ends in a module start in my notebooks. As a biologist that learned to program after 30 I just don't understand how you can develop data processing code without such a close handle on dataframes and without checking in graphs/visualizations if your code does what you expect. I don't see how I would do that in pure vscode of other IDEs. I also don't understand this sentence: "Once the data is loaded, it then has to be cleaned, which participants complained is a repetitive and time consuming task that involves copying and pasting code from their personal "library" of commonly used functions." What is the alternative? Not cleaning the code? And why copy and paste when you can perfectly fine have your own shareable module on the side? I guess most notebook users do some kind of hybrid development.

grenoire 7y ago
Good point on the last one; I think we have to 'educate' researchers on the fact that they can also write their own libraries and frameworks, and they should. Even basic data manipulation utilities can be made into Python modules and distributed at ease. If something is tedious, there definitely is a way to make it less so.

voldacar 7y ago

Performance is another pain point, at least for jupyter

ospohngellert 7y ago

I think that all the pain points of the article are a result of not using notebooks for their purpose. In my opinion, notebooks are good for: 1. POC/MVP: Showing that what you want to do will work before making a full structure. 2. Creating PDF/HTML documents with code and output. 3. Exploratory data analysis and visualization. I think many of the data scientists in the article go well beyond what a notebook is. A notebook is where you start, but should never be a production tool.

jupp0r 7y ago

Why not teach data scientists how to write software effectively? Those are smart people, it’s not like using version control, writing unit tests and extracting common code into libraries is rocket science.

zneveu 7y ago

One idea for a pain point not mentioned: better variable persistence. If I declare a variable, then delete the cell I declared it in, the variable persists. I've had this cause issues because if I use the deleted variable by accident, it will work fine right up until a kernel restart.

boomersooner 7y ago

This is akin to reviewing how well a screwdriver drives nails. Yes, it has problems. That doesn't mean it's a bad tool - you're just not using it right. Does it require discipline? Yes, but so does the screwdriver. That being said, I think jupyter specifically has some legacy issues around format, and I prefer R markdown. As much as I love pycharm, it's never going to do more than replicate the notebook experience. IMHO, the main author publishes on code UI/UX, the title seems more like click bait. Not sure why it's so upvoted.

bobbylarrybobby 7y ago

The problem of notebooks has been solved by the Python extension in Visual Studio Code (and some other editors too, although VS Code is the one I'm most familiar with). Editing an ordinary Python file, if you insert the comment "# %%", you turn everything between that comment and the next "# %%" (or the end of the file) into a code cell that can be submitted to the ipython kernel, just as in a Jupyter notebook. The editor splits into two halves, the left half your Python file and the right half the Jupyter notebook window with submitted code and formatted output (e.g., DataFrames look pretty, plots display normally, etc.). When you're done running everything, you can export the result as a Jupyter notebook. Because you're editing an ordinary Python file, standard features like version control and importing the file you're editing into other files (you cannot normally import .ipynb files IIRC) work normally. And of course since VS Code is a real editor/IDE, you can double click a file and have it open right up (no resorting to a Terminal to start your Jupyter session) and you get syntax themes, a built in Terminal, a git UI, code snippets, documentation on hover, vim mode if that's your thing, etc. The only downside I've found is that the Python extension doesn't incorporate ipython's autocomplete in its own autocompletion, but that's a small price to pay for getting to treat .py files as notebooks.

bart_spoon 7y ago

I'm not sure I understand the issue about the user repeatedly tweaking parameters for their data visualization. If anything, that is a reason notebooks are so nice. The repeated tweaks are due to the notebook format, its because that's an inherent part of the data visualization process, where the end result of a particular parameter choice is hard to predict how it will look with a given data set. So the same process would occur whether one was using a notebook or a script, but with a script it becomes much more cumbersome to actually see the result. In a notebook, the parameter tweaking for a data visualization is immediately followed by the result. I definitely agree with most of the other points though.

dkleissas 7y ago

At Gigantum, we're trying to solve some of these issues too. A Gigantum Project lets you run Jupyter or RStudio in a container that is managed for you. Everything is automatically versioned so you can sort out exactly what was run, by who, and when. https://gigantum.com https://gigantum.com