10 ms·

Most comments here express disbelief and disappointment in Jupyter from the software engineering point of view. What exactly is wrong with it? I use Jupyter da

• gaius 8y ago
  How do you find it compares to Spyder?
  • aportnoy 8y ago
    Spyder is too clunky. I work with a number of remote compute clusters, and like the ability to fire up a jupyter server through ssh, enable port forwarding, and work with the notebook in my browser. If I need a file browser or I want to quickly view two notebooks side-by-side, Jupyter Lab is still more convenient than Spyder. Although it doesn't support all the features of the notebook, and you end up installing additional extensions managed through yarn and webpack (jeez).
• taeric 8y ago
  My hunch is the same disdain you probably have towards scientists using excel is how many software engineers feel about your using jupyter. It isn't that you aren't getting good results and doing good work. It is that there are many practices that were hammered out in software development that were completely tossed out the window. 4GL comes to mind as a comparison. Makes great demos. Is not likely to lead to good engineering.
  • aportnoy 8y ago
    I use Jupyter for two things: 1. Interactive computation and plotting, run data through a pipeline. 2. Quick prototyping and testing. Once a function/class is ready, it goes into a module, together with some unit tests. Nothing beats Jupyter for that second use case, 90% of the bugs are squashed through interaction and inspection in the notebook cells, not through unit tests.
    • taeric 8y ago
      That some folks can be productive with a tool doesn't say, necessarily, much about the tool. Some could probably make the same two claims for excel. It isn't that the tool should be banned, per se. Just that many practices that have been rather proven in software are much harder to do in this environment. Sounds like what you like is the live coding aspect. Many of the lisp environments of yesteryear would have probably appealed to you. So would many earlier math packages such as matlab and mathematica. Both of those have had environments similar to the new "notebooks" for many years. For the pure coding side, imagine using an environment that actually allowed live redefinition of stuff in the code you had written. Step debugging. Breakpoints. Conditional breakpoints.
      • sheepmullet 8y ago
        > Some could probably make the same two claims for excel As a tool Excel has provided astronomical real world value. There are only a small handful of other tools that even come close.
        • taeric 8y ago
          I'm pretty sure this puts us in agreement. Right? :) I think the counter is there is probably a lot of damage excel has done, as well. I'm not convinced other tools would have resulted in no bugs/problems.
• dec0dedab0de 8y ago
  Jupyter is also really useful for figuring out how you want to do something before you copy it to your IDE. Every time I work with a new API or library I try it out in jupyter first.
  • michaelmrose 8y ago
    What are the benefits of this vs just an integrated repl?
    • cdavid 8y ago
      There is value when e.g. you need more than simply textual output (think plots, etc.).
    • dec0dedab0de 8y ago
      Being able to re-run and modify cells out of order.
    • msaharia 8y ago
      For me it is having a complete view of what I have done previously and rerunning past codeblocks by making changes.
• dnbgfher 8y ago
  Maybe I can provide a bit of perspective on this, as I have lots of conflicting feelings about Jupyter. When I'm doing some bit of data wrangling or just exploratory work with data I quite like it - at least at first. As you pointed out, it's really easy to extremely quickly iterate on things and start to get an idea of what's in the data, what techniques work, and which don't. It's great. Until it isn't. I'm probably "using it wrong" but all the suggestions I've seen for doing it "right" start cutting into the iteration time. If I'm in Jupyter it's because I want to flail around real fast and see what happens. And that causes all sorts of problems. Which cells do I need to rerun together? Which cells should I not run again. What the heck is actually in all these variables right now anyway, because I don't remember which order I've run (and rerun) all the cells in. And what was that one approach or parameter that really worked well that one time? I don't remember. These sorts of things aren't an issue with a non-Jupyter approach. And because I'm taking my time anyway I'm probably being diligent with source control, and all that. Even pulling code out of a notebook into a more stable workflow is a pain, because you have no assurances you can just copy/paste a cell and have it work. In my experience, it never will. It's one of those things that has a pretty heavy costs and benefits. And unfortunately they aren't really possible to separate. Not coincidentally, I feel the same about dynamic languages and even less usefully typed static languages. Jupyter takes an already fairly extreme position on the stable(safe)/easy continuum and really ramps it up to a point where it's hard to wrangle time and work done easy into some sort of stability. It's great, and it's awful.
  • williamstein 8y ago
    > And what was that one approach or parameter that really worked well that one time? I gave a talk at JupyterCon this week about Cocalc which partly solves this one problem by providing a Time travel slider with complete history.
    • aportnoy 8y ago
      I took abstract algebra with Prof. Kedlaya at UCSD who likes to use Cocalc for his other courses, haven't used it though.
  • 8y ago
• rajangdavis 8y ago
  I don't think there's anything inherently wrong with jupyter notebooks, but they run in the browser without behaving in the same way a webpage or web app will in terms of managing state. This is sort of tantamount to opening a website in your browser, but as soon as it loads, you need to refresh it as to "correct" the state of the site. That's my only compliant about jupyter notebooks. It's a powerful tool but acts a little funky if you're not used to it.
  • rajangdavis 8y ago
    Not sure why I got downvoted... I have been working in web development in the last 5 years and have started working with Jupyter notebooks in the last 4 weeks while taking some Data Science courses. I really feel like state management with jupyter notebooks is a bit whacky. From my experience having built and maintaining websites, it's bizarre to open a notebook with a previous state but have cells not work because it needs to be reran from top to bottom. In contrast, when opening a web page, the state is usually reflecting what you'd might expect without having to refresh the page. I am sure there is some setting to correct this (either rerun on open or don't save state on exit), but it hurts my mental model of how a browser based application should work.
• ed_balls 8y ago
  - hard to use linters - hard to use formatters like yapf - can't use the shortcuts from my editor - easy to screw up the order of execution - harder to review, generate nice diff, fix merge conflicts
• analog31 8y ago
  I also work in scientific computing, and I love Jupyter. I think out-of-order or unknown-order execution is the biggest source of my own mistakes / bugs. If the order in which your cells are executed affects your result, you've got a bug waiting to bite you or someone else. I've been there. And we tend to run cells more than once, while refining our work. This potentially affects reproducibility, which is a tenet of scientific computing. So it's important in our world. I personally think there might be a couple of ways to deal with this issue. One is to put handcuffs on Jupyter and inadvertently make it more difficult or cumbersome to use. But this might be valuable if the size of your project and importance of your result warrants it. Another way is for us to recognize that at some point we have to grow up and learn some techniques from the programmers. In my revisionist history, software engineering began when programs got too big to see the whole thing on one screen and intuitively understand. One of my little habits is that I do a "restart kernel and run all cells" when I'm ready to take a break or close out at the end of the day. If if fails, the causes of the problem tend to be superficial, and nothing to lose sleep over. Another habit is to take cells that work and are nice, and enclose their innards in a function so that their impact on the global variable space is minimized. This can be done during a sleepy time in the afternoon when you're not really in flow anyway.
  • yodon 8y ago
    > In my revisionist history, software engineering began when programs got too big to see the whole thing on one screen and intuitively understand. Laughing, politely, at that summary of "history." Software engineering started when there were no screens. Code was written on typewriters (called "terminals") and then printed to cards or long rolls of paper (and occasionally using switches to indicate the 1's and 0's you wanted set as bits in your code). Before the Mac a "screen" of code was pretty universally 24 lines of green or white text on a black background, 80 monospace characters per line. A few specialized terminals had 25x80 so they could show a status line below the 24x80 standard area, but they were rare beasts. Yes, there were a few systems like Plato that predated the Mac but if you need a tl;dr it's the observation that, like most revisionist histories, this one is unfortunately total nonsense when looked at in the context of actual history (even if lots of people who didn't experience the actual history believe it).
• geebee 8y ago
  This is a really important comment. I started my career in scientific computing, went over to software/web development for a while, and am back over to scientific computing. I've been involved in teaching efforts lately, mainly for analysts who want to do data wrangling and stats, and I think it is a good environment for some of what they want to do. I never used Jupyter before getting involved in this kind of coding, data wangling and stats. And it took me a while to get used to it. The "lack of state" is confusing, as well as the apparent non-linearity of a jupyter notebook (it can appear that certain commands are done in order from top to bottom, when if you look at the number next to them, they may have been executed in a completely different order). I actually chalked this up to my background in software development, figuring I was more wired to think of state as something that would be cleared every time a script was run. Analysts (statisticians, engineers, data scientist types) like to interact back and forth very iteratively with their data, and I think they're less likely to be tripped up by some back of mind assumption that you can determine the state of variables and so forth by looking at a bunch of commands as if they were run top to bottom in a file. It got me thinking about when I'd use jupyter notebooks for my own work. In short, if I were writing a program, even just a one page script, I wouldn't use an interactive notebook (I have opened .py files in jupyter and just used it as a text editor). But if I wanted to "do my math homework"? A one off where I need to get an answer to some complicated questions that will require a lot of data wrangling and stats? Yeah, I'd probably reach for jupyter notebook or something like it. Otherwise? I can't remember the O'Reilly book where this was written (sorry I wish I had the cite) but the author wrote that his favorite dev environment is python and a text editor. That's still the case for me as well. I hate to conclude with the disappointing "the problems are when you use it for the wrong task", mainly because I've objected when people have made this argument in the past. If a tool tends to get used for the wrong task, that's partly a problem with who is using it... but we shouldn't let the tool itself off the hook too easily. It's still worth thinking through. I'm worried I'm falling into the trap of making this argument when I like the tool, and objecting to it when I don't. I'll have to think about that a bit. As it stands, though, I haven't had a huge problem with Jupyter Notebook, and I like it a lot, probably because I don't use it when Python + vi (or a "better" but still basic text editor) would do the trick.
  • IgniteTheSun 8y ago
• a_136_chiffa 8y ago
  I am a computational biologist with a heavy emphasis on the data analysis. I did try Jupyter a couple of years ago and here are my concerns with it, compared to my usual flow (Pycharm + pure python + pickle to store results of heavy processing). 1) Extracting functions is harder 2) Your git commits become completely borked 3) Opening some data-heavy notebooks is neigh impossible once they have been shut down 4) Import of other modules you have in local is pretty non-trivial. 5) Refactoring is pretty hard 6) Sphinx for autodoc extraction is pretty much out of the picture 7) Non-deterministic re-runs - depending on the cell execution order you can get very different results. That's an issue when you are coming back to your code a couple of months later and try to figure what you did to get there. There are likely work-arounds for most of these problems, but the issue is that with my standard workflow they are non-issues to start with. In my experience, Jupyter is pretty good if you rely only on existing libraries that you are piecing together, but once you need to do more involved development work, you are screwed.