24 ms·
Why Jupyter is data scientists’ computational notebook of choice
- stephengillie 8y agoI recently got a Jupyter Notebook, and found it's a large JSON document, with some sections in markdown and some in Python. A browser could omit the Python, and an interpreter could omit the markdown. Would this work for other languages? Maybe JavaScript or Powershell instead of Python?
- deleted 8y ago[deleted]
- Quenty 8y agoYes! Juypter is actually designed to take in multiple kernel types and supports a ton of different languages. It’s not just limited to Python. https://github.com/jupyter/jupyter/wiki/Jupyter-kernels https://github.com/jupyter/jupyter/wiki/Jupyter-kernels
- closed 8y agoCan and does! Jupyter notebooks have a flexible architecture for running anything (called "kernels"). Notable examples... R: https://github.com/IRkernel/IRkernel https://github.com/IRkernel/IRkernel node: https://github.com/notablemind/jupyter-nodejs https://github.com/notablemind/jupyter-nodejs In my mind, one of this big advantages of jupyter is its extendability. I was able to quickly modify notebooks to run unit tests, so we could use them for projects at DataCamp: https://www.datacamp.com/projects https://www.datacamp.com/projects.
- notthemessiah 8y agoJupyter = Julia + Python + R
- JA7Cal 8y agoYou may be interested in jupytext https://github.com/mwouts/jupytext https://github.com/mwouts/jupytext
- glup 8y agoI've switched largely to Jupyter / Python for computational linguistics / psycholinguistics because of the pandas / numpy /numba stack, decent off-the-shelf NLP (spacy and gensim), and the ease of moving data into an R kernel for specific analyses and plots. Also nice that any reasonably sized notebook will render on GitHub (and access can be controlled through the accounts system, until something is ready to be public). One thing I haven't figure out how to do is to generate fully styled LaTeX manuscripts from notebooks (like papaja for RStudio). Is there a way to do this with pandoc?
- Quenty 8y agoYes! Juypter notebook has an export to .tex file in its export menu. If you install the right stuff you can render the output on the server. However, the output of this isn’t nearly as well formed as a hand written latex document is.
- macawfish 8y agoYeah but I believe you can customize the export template, right?
- wanderfowl 8y agoHaving NLTK and SciKitLearn in the same environment as my stats tools is... tantalizing. And if I could write Markdown-to-TeX docs straight from Jupyter, rather than the R -> TeX tables/variables read into LaTeX I'd used before, that'd be a massive win.
- lixtra 8y agoWhat was the earliest of these tools? Mathcad? Mathematica? Maple?
- enriquto 8y agotangle and weave, by knuth
- kgwgk 8y agoI don’t know, but I found this: https://patents.google.com/patent/US8407580 https://patents.google.com/patent/US8407580 Also: https://www.theatlantic.com/science/archive/2018/04/the-scientific-paper-is-obsolete/556676/ https://www.theatlantic.com/science/archive/2018/04/the-scie... “The notebook interface was the brainchild of Theodore Gray, who was inspired while working with an old Apple code editor. Where most programming environments either had you run code one line at a time, or all at once as a big blob, the Apple editor let you highlight any part of your code and run just that part. Gray brought the same basic concept to Mathematica, with help refining the design from none other than Steve Jobs.”
- tony_cannistra 8y agothere’s a relatively esoteric paradigm known as “literate programming” which has been around since Knuth (he wrote the book [0]) and that has some software tools associated, of which Jupyter is a particularly web-age example. [0]: https://en.m.wikipedia.org/wiki/Literate_programming https://en.m.wikipedia.org/wiki/Literate_programming
- tmalsburg2 8y agoLiterate programming was esoteric, true, but the concept saw a huge renaissance in academia and data science with the advent of RMarkdown¹ which for many of my colleagues is the default way of preparing technical documents. Another area in which literate programming has become hugely popular is Emacs' Org-mode ecosystem which has fantastic support in the form of Org Babel². I use literate programming for almost everything. Research papers, tech reports, notes, experiments, teaching materials, letters, student evaluations, and so on. It's completely ridiculous how useful it is once you get the hang of it and make it your default document type. [1] https://rmarkdown.rstudio.com/ https://rmarkdown.rstudio.com/ [2] https://orgmode.org/worg/org-contrib/babel/ https://orgmode.org/worg/org-contrib/babel/
- rsivapr 8y agoLink to the deck by Joel Grus' talk that is mentioned in the article: https://docs.google.com/presentation/d/1n2RlMdmv1p25Xy5thJUhkKGvjtV-dkAIsUXP-AL4ffI/edit#slide=id.g362da58057_0_1 https://docs.google.com/presentation/d/1n2RlMdmv1p25Xy5thJUh...
- amirathi 8y agoReally good talk. Here's the video: https://www.youtube.com/watch?v=7jiPeIFXb6U https://www.youtube.com/watch?v=7jiPeIFXb6U And all JupyterCon 2018 talks if anyone is interested: https://www.youtube.com/playlist?list=PL055Epbe6d5b572IRmYAHkUgcq3y6K3Ae https://www.youtube.com/playlist?list=PL055Epbe6d5b572IRmYAH...
- glup 8y agoIt's easy to grade student assignments in notebooks with https://github.com/jupyter/nbgrader https://github.com/jupyter/nbgrader, which also makes it great for teaching.
- taeric 8y agoMeh. As long as you have defined deliverables between grader and student, grading programming based assignments are relatively easy. Coursera has been around longer than Jupyter has been popular, after all. (And they aren't all just multiple choice.) Being interactive is what makes it good for teaching. But there are plenty of interactive options. And for a certain class of teaching, it is not "on rails" enough such that people will have to have a ramp up period first on Jupyter before they can really get into their topic.
- cphoover 8y agoFor people who prefer to code in JS. There is a similar application called observable notebooks that has recently come out: https://beta.observablehq.com/ https://beta.observablehq.com/ It offers some nifty things including, well, observables where cells of the scratchpad can automatically update by observing changes from other cells.
- pintxo 8y agoThx for the pointer. Seems this is related to Mike Bostock the guy behind d3.js, will definitely look into it.
- th0ma5 8y agoMore of an online service, though.
- cphoover 8y agotrue true.
- solomatov 8y agoIf you want to have interactive computations in Python, http://datalore.io/ http://datalore.io/ has support for this. It feels more or less like Excel with ability to write Python in there.
- johnminter 8y agoRStudio with using Rmarkdown is also popular. Both workflows are language agnostic.
- closed 8y agoOf note--while Rmarkdown's python support used to be pretty bad, the most recent Rstudio uses the package reticulate behind the scenes, and it works really well! It's easy to share data between the R and python session, and calling python from R, or R from python is straightforward.
- cwyers 8y agoPutting aside the R vs Python question (as as noted in this thread, you can use R in a Jupyter notebook and Python in an RMarkdown notebook), I much prefer RMarkdown notebooks. RMarkdown notebooks are plain text, so you can read them easily in any text editor (which also means they play well with git, unlike Jupyter notebooks). And it's meant to work with the RStudio IDE, so I get a much more seamless experience going between regular code and notebooks (although this is admittedly a more R-centric benefit, at least until and unless RStudio adds Python support outside of notebooks).
- IanCal 8y agoA benefit I like about rmarkdown is that it makes it very easy for me to create templated reports. They're built in a way that makes it easy for me to work either iteratively (due to caching of blocks) or rerun the whole thing and get an output.
- jonnycomputer 8y agoUsing markdown for python notebooks makes so much sense. What was the thinking behind encoding everything inside JSON?
- wanderfowl 8y agoI like R for many things, but Python just keeps getting more compelling, particularly given the excellent machine learning packages. As these sorts of toolchain elements get better and better, and as more people realize that there's a benefit to simultaneously training researchers to run code as well as stats, I suspect we'll start to see an exodus from pure R solutions. The real question is when (and whether) new social scientist stats courses will start teaching Python stats toolchains, rather than R. That seemed to be an inflection point for R (as folks moved away from SAS), and could be for stats-centric Python too.
- isatty 8y agoYou may want to try Julia
- WhompingWindows 8y agoThe migration between languages is also industry specific. They are still teaching SAS to finance and healthcare analysts, for instance, and R and Python are still rising in healthcare specifically. Keep in mind all the legacy code and all the coders who just know SAS and don't need to change. It'll take longer for the transition than you think.
- bigger_cheese 8y agoI have tried to move from SAS to R a few times at this point it's largely inertia there is so much in my org already written in SAS makes it very difficult to convince people to change. I do like the SAS dev tools (especially Enterprise Guide). I'd really love if R had some sort of GUI front end for non technical people. Eg. I know the finance analysts in our org wouldn't have a clue how to configure their own ODBC sources -which you need to do with R Studio until it's as easy for them as SAS convincing them to switch won't get any traction.
- mattkrause 8y agoIt seems to depend on what you’re doing. Python definitely has more mindshare for machine learning, and particularly deep learning. However, that’s not all of statistics. For things like mixed-effects modeling, I think R still has a clear lead. There are some python packages (e.g., statsmodels) but R’s lme4 has more features, like custom covariance structures, and virtually every textbook and tutorial currently uses R. I’m actually not sure if I’ve ever actually encountered statsmodels in the wild. PyMCMC is relatively popular, but I think bugs/jags are also more common.
- HaHa31 8y agoLink is dead?
- zimablue 8y agoI kind of find Jupyter an indictment of other coding tools really, it's 2018 and they're normally kind of weak or kind of unprogrammable. Feel like we're waiting for someone to really reinvent Emacs, preferably using web tech. Most editors can't open a terminal that you can use VIM keybindings on to search/navigate history and treate like any other buffer. VSCode -> not currently possible because they wrote it in a restrictive way with Panel as a special case very different to code window Atom -> probably possible but I don't think terminal-plus is quite it. Any IDE I've tried -> not possible. Emacs -> possible. Not that this is the be all end all feature but it is useful as hell and kind of a litmus test for whether you can program your environment. edit: LightTable seemed kind of cool but became abandonware like the author's other projects
- pavanagrawal123 8y agoHey There! I'm trying to solve this right now in VSCode's in built editor: https://github.com/pavanagrawal123/VSNotebooks https://github.com/pavanagrawal123/VSNotebooks . It's a fork from another extension somebody already built, but all activity is dead, so I'm starting up dev on an active fork. I'd love to hear any feedback y'all have! :)
- gnulinux 8y agoI use Jupyter in emacs: ein-mode. The whole concept of programming in a browser sounds bizarre to me. I have a tool that's designed for programming (emacs) and a tool that's designed for streaming cat videos (firefox) and I use the latter for programming? Thanks but no thanks. Ime emacs works pretty perfect with jupyter too, there is no need to use firefox for something it's not designed to do for my full-time job.
- zimablue 8y agoI have also tried ein-mode, and sometimes use it. I haven't used it enough to have a super educated opinion but it didn't blow me away. I don't think it supports cell folding (?) as an example missing feature Minor point but it also can't/shouldn't support widgets, which we use at work. Any extension to jupyter is going to be written in javascript so there's an element I'd be locking myself out of the ecosystem. I didn't mean this as a criticism of emacs, my post said that the best thing I know is emacs, just that it's (probably) not the future of editors imo so I'm reluctant to throw 100 hours into it.
- agibsonccc 8y agoThe only thing that stops me from being able to use notebooks full time is their intellisense compared to IDEs is horrible. I like being able to use them for demos/presentations, but I can't imagine trying to code within one primarily. Especially when it comes to tracking results. How do people cope with this? Do you supplement it with other tools? I spend a lot of my time in an IDE and then just paste some of the code in to cells. That seems easier.
- LeanderK 8y agodiving code between models/data-pipelines and experiments. Notebook are used for visualization and telling a story why you tried what for the other team-members.
- agibsonccc 8y agoYeah but the whole point is "interactive coding". It doesn't feel very interactive when I have to context switch all the time :). I'd prefer something closer to what the lisp folks get to do with the repl where you can scratch out an idea and see it working without leaving your environment.
- LeanderK 8y agowell, I don't think so. Not everything you do is interactive. Data exploration and basic model selection is, but complex models and more complicated data-pipelines/preprocessing isn't, I think. Tensorflow is the opposite of interactive, even in a notebook. putting models (in a sense of more complicated models, not just a SVM), data-pipelines, shared visualization-code in a src folders and experimenting in the notebook divides stuff that's interactive by nature from "real" coding. I don't context-switch that much to be honest. I don't really copy code into cells, because I only experiment there. Also, what happens if you need to share code between notebooks? I think notebooks should be simple and explain the experiments and the reasoning behind them to your coworkers. Otherise it's hard to coordinate and learn from each others insights into the data.
- epistasis 8y ago
- mindcrime 8y agoZeppelin[1] is another great tool of a similar nature. It leans a little bit more towards the Scala / Spark world, for people who like that stack. That said, you can use Python, R, etc. with Zeppelin as well. [1]: https://zeppelin.apache.org/ https://zeppelin.apache.org/
- projectramo 8y agoHere are the issues with Jupyter, and most other flavor, of notebook: 1. variables have to be explicitly output The most important tool for programming, for me, is that window that shows you the current state of all the variables. When I step through a program, I look at the state. 90% of my debugging solutions come from seeing that variable doesn't have the right state. 2. Intellisense For the love of god, I do not want to remember if it is len(), length(), .len(), .length(), .size(), size(1) or whatever. That's it. But those two are so big that I have to code and debug in Spyder and then paste the code into notebook. I feel sorry for people who are new who think that all the debugging is happening in the notebook.
- pavanagrawal123 8y agoHey There! I'm trying to solve the issue of IntelliSense.. I'm building/improving Jupyter Notebooks inside VSCode: https://github.com/pavanagrawal123/VSNotebooks https://github.com/pavanagrawal123/VSNotebooks . It's a fork from another extension somebody already built, but all activity is dead, so I'm starting up dev on an active fork. I'd love to hear any feedback y'all have! :) Also planning to add some nice debug features, plus hopefully integration into the inbuilt VSCode debugger!
- yodon 8y agoHow do you see the idea of a vs code notebook comparing to or being different from the goals of the hydrogen editor?
- kylebarron 8y agoTo me, my favorite part of the design of Hydrogen is that it's entirely language agnostic, and can be used with _any_ Jupyter kernel.
- pavanagrawal123 8y agoI'm planning on adding better language support in a couple of weeks! Don't want to limited to Python and R
- MrPowers 8y agoNotebooks are great for invoking existing functions and exploring data. Notebooks aren't ideal for creating functions (standard text editor features are lacking and testing is impossible). Notebooks encourage an "order dependent variable assignment" programming style without abstractions. Here's what you'll commonly see in a notebook: val df = spark.read.csv("some_data") df2 = df.withColumn("clean_name", trim("name")) df3 = df2.filter("clean_name" === "Mark") I've found that notebooks are very useful if you write all the complicated code in separate GitHub repos and attach binary executables to the cluster. If you try to write all your logic in notebooks, you'll quickly struggle with order dependent, messy code.
- edparcell 8y agoI had the same trouble with order dependence as notebooks got to a certain size, so my team and I created and open-sourced a library, Loman, to help with that. It allows you to interactively create a graph, where nodes represent inputs or functions, and then keeps track of state as you change or add inputs, intermediate functions and request recalculations. Our experience has been broadly positive with this way of working. As graphs get larger, it's easy to lift them into code files in libraries, while continuing to modify or extend them in notebooks. The graph structure and visualization make it easy to return to loman graphs with up to low hundreds of nodes, which would make for a fearsome notebook otherwise. It also makes it easy to bolt Qt or Bokeh UIs onto them for interactive dashboards - just bind UI widgets and events to the inputs and widgets to the outputs. They can be serialized, which is useful for tracking exceptions in intermediate calculations when we put them in airflow to run periodically, as you can see all the inputs to the failing calculation, and its upstreams. [1] Github: https://github.com/janushendersonassetallocation/loman https://github.com/janushendersonassetallocation/loman [2] Quickstart/Docs: https://loman.readthedocs.io/en/latest/user/quickstart.html https://loman.readthedocs.io/en/latest/user/quickstart.html
- jhokanson 8y agoThe function bit has always confused me. I tend to write code with lots of functions/modules/classes for handling various aspects of the analysis and I just don't understand how that's supposed to be integrated. Instead notebooks seem better designed to handle small code snippets that rely on well known libraries. I'd be happy to jump on the notebook bandwagon but I'm having trouble seeing how I could adapt my code to the notebook style.
- makmanalp 8y agoThe majority of the complaints I hear about notebooks I think come from a misunderstanding of what they're supposed to be. It's a mashup between a scientific paper and a repl. So it's useful for a bit of both: a) Just like with a paper, you can present scientific or mathematical ideas with accompanying visualizations or simulations. From the REPL side, as a bonus, you get interactivity, and the reader can pause and experiment with the examples you're giving to improve their understanding or test their hypotheses. If I change this variable, how will the system react? You can just try it! b) Just like with a REPL, you can type in and execute commands step by step, viewing the output of the previous command instead of running the whole thing at once. From the document side, as a bonus, you get nicer presentation (charts, interactivity, nice and wide sortable tables, etc) than you would in a shell, which comes in handy when doing things like data exploration or mathematical simulation. It's decidedly NOT there for you to type all your code in like an editor and make a huge mess. It's apples and oranges w.r.t and a poor substitute for something like PyCharm or VS Code or vim. It is there for you to a) try things out yourself, and whatever you discover hopefully eventually make it into proper python modules b) make interesting ideas presentable and explorable for others. That's all! When I see stuff like "out of order execution is confusing", I don't disagree, but it does make me wonder how long and convoluted the notebooks these people work with are - probably a ripe candidate to refactor stuff out into python modules as functions. When I see stuff around notebooks for "reproducibility", I'm a bit confused in that notebooks often don't specify any guidance on installation and dependencies, let alone things like arguments and options that a regular old script would. In that regard I think it's barely an improvement over .py files lying around. When I hear "how do I import a notebook like a python module", I'm very very scared. Granted, I've seen huge notebooks that are a mess, so I understand the frustration, but it's not like we all haven't seen the single file of code with 5000 lines and 10 nested layers of conditionals at some point in our lives.
- taeric 8y agoYou parting "Granted..." is precisely what fills me with dread when I see notebooks. Yes, I have seen poorly done source files. I made more than a few myself. However, many of the practices we have grown into as sound programming advice seem to be largely thrown out the window for these notebooks. The irony, to me, is that I actually typically argue for the mixing of presentation and content. But to me, notebooks look like an attempt by people to make a WYSIWYG out of JUnit/TestNG/whatever style reports. Only, without the repeatability. There is also the entire bend where these are taking off in a way that doesn't make sense. Do they do the things you are saying? Well, yeah. But no better than plenty of tools before them. Mathematica and Matlab both had "notebook" like features for a long long time. Complete with optimized libraries. And this is ignoring the interactivity of the old LISP machines. (You can see from my history I have a soft spot for emacs org-mode.) Jupyter is a lot of things. Bad isn't necessarily one of them, but exceptional isn't, either. Heavily marketed is.
- enriquto 8y agoI love notebooks as a way to present information, data, code and computations. However, I cannot stand typing any text into a web browser window. Is there any way to edit a jupyter notebook with a text editor and then run it in the browser? The native json is not really human-editable.
- hindsightRegret 8y agoLook into jupyter-vim-binding. It at least makes navigating within the editor more bearable
- dimatura 8y agoI have felt the same way in the past. There are some ways to do this, but none is great. Unfortunately using a text editor to put in the notebook text areas is not that straightforward because of security features in modern browsers. Since jupyter is actually a server (usually running locally) it's possible to communicate directly with it from a sufficiently advanced editor, but haven't seen any good execution of that idea. There's also the ipymd (https://github.com/rossant/ipymd https://github.com/rossant/ipymd) format, which is just markdown, and seems more or less what you want, but you lose the interactivity and display of images/plots/HTML/etc. Personally, I've found the "jupyter-vim-binding" (https://github.com/lambdalisue/jupyter-vim-binding https://github.com/lambdalisue/jupyter-vim-binding) to be a relatively acceptable emulation of vim keybindings for the code editing.
- enriquto 8y agoThis is not what I mean. Mine is a problem of file formats, not of interactivity. I want to edit a text file alone, without need for any web browser in my computer. Then I push the notebook to git, and somewhere else it is opened by the browser. This would be possible today if the notebook file was python code with comments, for example, instead of an uneditable json.
- dpwm 8y agoThis is quite a neat idea. A couple of people have mentioned Jupytext around here. I found a guide with animations which appears like it might do what you want. [0] I personally look forward to trying this out, as it means that I can use Jupyter in a way that doesn't mean adapting my workflow to the tool so much. [0] https://towardsdatascience.com/introducing-jupytext-9234fdff6c57 https://towardsdatascience.com/introducing-jupytext-9234fdff...
- amirathi 8y agoVersion control for Jupyter notebooks was one of the biggest complaint I had. Specifically, diff and merge with the JSON files (.ipynb) is ugly. I built ReviewNb[1] to solve one of those problems (diff). Note that, there is nbdime[2] which works well for local diff/merge. The idea for ReviewNb is to have much tighter integration with GitHub etc. [1] https://reviewnb.com https://reviewnb.com [2] https://nbdime.readthedocs.io/en/latest/ https://nbdime.readthedocs.io/en/latest/
- stephengillie 8y agoDiffing JSON as text must be painful. Diffing JSON as data should be somewhat simple.
- justin66 8y agoAre there any merge tools that offer features for this a lot more sophisticated than basic text comparison?
- stephengillie 8y agoPowershell has Compare-Object, which will diff .NET objects. It has the convenient alias "diff". JSON can be converted to .NET objects by ConvertFrom-JSON. So you can import 2 JSON files and diff them in Powershell.
- avip 8y agoNot merge, but jq can diff, and it can also do a consistent dump (jq -cS) for you. Edit: to clarify, jq -S does deep keys sorting. $ echo '{"z":{"b": "second", "a": "first"}, "x": 4, "y": 7}' | jq -S { "x": 4, "y": 7, "z": { "a": "first", "b": "second" } }
- SiempreViernes 8y agoTighter integration with git is very interesting, but this is sadly just integration with github. I think coupling to github makes sense if you are a building a dev-support service, but for a end user it makes little sense to wed the vcs to a specific website.
- trop 8y agoQuestion/idea: Could a notebook-model supplant bespoke photographing processing software such as the "darkroom" mode of Lightroom (or darktable). The extant programs essentially take a lot of data (camera's raw output) and apply a configurable recipe to produce intelligible output (an image). Each recipe (stored as an XMP sidecar) is essentially a list of math operations (increase brightness, wavelet decompose, change color model, etc.) and their parameters. Obviously a great part of why we use Lightroom/darktable is because of the speed with which the recipe-processing occurs. Plus a smooth UI, a catalog-viewing feature, and a well vetted choice of image operations. The appeal of moving this work to a notebook would be that an actively maintained Jupyter ecosystem could supplant lock-in to a specific software, and open up the underlying math magic. At the very least, this could be an interesting platform for experimenting with image processing methods. And the reordering of cells could become a virtue, to run an image processing pipeline out of the standard order. I'm curious if anyone has already worked along these lines. I find through a quick web search that people are doing some image processing, but more in the face detection or ML for medical imaging aspects. I see as a basic toolkit that http://scikit-image.org/docs/dev/auto_examples/ http://scikit-image.org/docs/dev/auto_examples/ is something, though this isn't the whole range of operations needed for, say, fine art image tuning.
- dimatura 8y agoYes and no. I do computer vision and like photography as well. I use Jupyter notebooks extensively for computer vision, and they work pretty well for (semi) interactive manipulation of image data with code. But as a general purpose tool, it's too clunky for anything more than prototyping. I don't see them replacing darktable/lightroom anytime soon.
- thomasfedb 8y agoJupyter is lovely (and JupyterLab looks delicious), but the set up required to achieve a reproducible local server with the R kernel and versioned R packages is 100% not. Installing R packages through anaconda is like pulling teeth and the docker images for my Jupyter notebooks push past 6GB and take multiple cups of tea to build. Is there a good solution I'm missing? A good hosted solution perhaps?
- enriquto 8y agoyou have google colab notebooks: https://colab.research.google.com https://colab.research.google.com
- ianhowson 8y agoI run https://rnotebook.io https://rnotebook.io -- it's free for the foreseeable future. The Docker image is 6GB.
- smortaz 8y agohttps://notebooks.azure.com https://notebooks.azure.com (python, r, f#)
- kvlr 8y agoI run https://nextjournal.com https://nextjournal.com We archive full-stack reproducibility by allowing you to install arbitrary software and version these environments using docker. You can reuse these in other articles or pull and use them locally. `xoxo` is a signup code you can you if you want to give it a try.
- thomasfedb 8y agoLooks cool, do you have an article that explains the stack from a technical perspective? Is this based on Jupyter?
- kvlr 8y agoNot yet. It isn't based on Jupyter but it's all written in Clojure. Been meaning to do a writeup on our stack for some time… In addition to our own runtime protocol, we support Jupyter kernels and you can import Jupyter and (R)markdown documents.
- ralmidani 8y agoOut of curiosity, what other options are there?
- zamazingo 8y agoR Notebooks (https://bookdown.org/yihui/rmarkdown/notebook.html https://bookdown.org/yihui/rmarkdown/notebook.html)
- jonnycomputer 8y agofor python, org-mode in emacs will do it. but that ties you down to emacs.
- rrmm 8y agoThe dreaded emacs lock-in: before you know it, you're browsing the web and editing your photos in emacs.
- tmalsburg2 8y agoOrg-mode was a total game changer for my work life. I use it for almost everything. Yes, it ties me to Emacs. I don't see this as a problem.
- kvlr 8y agoI run https://nextjournal.com https://nextjournal.com Use `xoxo` as a signup code if you want to try it, thanks!
- blululu 8y agoIMO, Jupyter is nice for presenting the final results of research (like LaTeX), but it is often not the right tool to get there. It's good for professors who teach and publish but it's bad for students to learn and research. Frankly I find that all programming environments for scientific computing are deficient in some way or another. If you look at the set of features in Visual Studio, R Studio and Jupyter notebooks, you will see that the Union of useful features is large, and the intersection is almost empty.
- reilly3000 8y agoDoes anybody know of a good hosted solution of JupyterHub? I made a neat notebook that I needed to share with my non-technical team, it was using iPyWidgets to do some interactive modeling, but they each needed to be able to use it independently. It has private data so I couldn't use Binder. I've been following Zepl.com for a long time, but couldn't use them here because Zepplin doesn't support iPyWidgets. Pretty soon I found myself installing helm and trying to follow along a tutorial on how to deploy JupyterHub on a kubernetes cluster. That started to add an unmanageable level of complexity to own, especially to share a simple notebook. And while spinning up a GKE node per user is the whole point of Kubernetes, it got expensive quickly in my test. We cannot spend $75K a year on Domino. Any other options?
- reilly3000 8y agoAfter laying that all out there I now realize I could have just given my handful of users their own files. Doh. Still, the fact remains that JuptyerHub is powerful, but difficult to install and manage if you're not a university IT dept. Any SMB solutions?
- SiempreViernes 8y agoDo you have some server to host it on? If so, I’d run the notebook on the remote server and just teach them whatever command they need to make a ssh tunnel there. Something like what is described here: https://techtalktone.wordpress.com/2017/03/28/running-jupyter-notebooks-on-a-remote-server-via-ssh/ https://techtalktone.wordpress.com/2017/03/28/running-jupyte... So they would utter the unknowable incantation and then point their browser at localhost:8000 or whatever and then use their version of the notebook.
- reilly3000 8y agoI think that would be within reason for most of our users, but we have a couple of Windows users. I'm not particularly keen on telling them to install putty or WSL.
- SiempreViernes 8y ago
- RandomInteger4 8y agoI hate Jupyter notebooks; I seriously do. Maybe that's an irrational sentiment, but I find them nauseating to me in the same vein that I find country music nauseating.
- mike_ivanov 8y agoThere is no "why" because it is not. An awkward JSON format plus the inability to survive network interruptions. A WiFi router goes down during a long computation, and voila -- the results are lost, which is unimaginable with RStudio.
- dangirsh 8y agoI've found ob-ipython [1] within Org Mode to be one of the best options for interfacing with Jupyter. If you're sick of the limitations of working in a browser, it's worth checking out. Scimax version: https://github.com/jkitchin/scimax/blob/master/scimax-ipython.org https://github.com/jkitchin/scimax/blob/master/scimax-ipytho... Video of Scimax version: https://www.youtube.com/watch?v=dMira3QsUdg https://www.youtube.com/watch?v=dMira3QsUdg Previous HN discussion highlighting key features: https://news.ycombinator.com/item?id=17839926 https://news.ycombinator.com/item?id=17839926 Some relevant blog posts: - https://vxlabs.com/2017/11/24/getting-ob-ipython-to-show-documentation-during-company-completion/ https://vxlabs.com/2017/11/24/getting-ob-ipython-to-show-doc... - https://vxlabs.com/2017/11/30/run-code-on-remote-ipython-kernels-with-emacs-and-orgmode/ https://vxlabs.com/2017/11/30/run-code-on-remote-ipython-ker... - https://kozikow.com/2016/05/21/very-powerful-data-analysis-environment-org-mode-with-ob-ipython/ https://kozikow.com/2016/05/21/very-powerful-data-analysis-e... [1] https://github.com/gregsexton/ob-ipython https://github.com/gregsexton/ob-ipython
- GChevalier 8y agoI recently wrote on how to build/grow clean software out of Jupyter notebooks and on pitfalls to avoid when coding like that: https://github.com/guillaume-chevalier/How-to-Grow-Neat-Software-Architecture-out-of-Jupyter-Notebooks https://github.com/guillaume-chevalier/How-to-Grow-Neat-Soft...
- GChevalier 8y agoI submitted it to HN too, why not! Here: https://news.ycombinator.com/item?id=18339703 https://news.ycombinator.com/item?id=18339703
- betolink 8y agoIMO Jupyter notebooks are popular because it's open source, they are convenient and they help a lot illustrating an idea. The results of a notebook can be shared more easily than a plain repository(via nbviwer or binder) and more importantly the science there it's reproducible.
- bravura 8y agoHave people been finding that AWS SageMaker disconnects you and you have to restart after a couple hours? I am curious people's thoughts on using Jupyter for long-running code. Having a totally self-contained experiment in one notebook, even if it long-running, is very useful for reproducibility. It works fine on my local laptop and a remote server, but not with SageMaker.
- azag0 8y agoI love Jupyter Notebook for experimenting and rapid creation of reports, but dislike it for not being able to use my editor and for intermingling inputs and outputs in a single file. So I'm working on an alternative frontend to Jupyter kernels, which is heavily inspired by KnitR: https://github.com/azag0/knitj https://github.com/azag0/knitj It is still being developed, but it's functional and I use it every day.
- huac 8y agoyour project also seems to play much more nicely with git!
- azag0 8y agoTrue. Actually that's what I meant by "intermingling inputs and outputs". KnitJ still shows both code and its output in the rendered HTML, but unlike in Jupyter Notebook, the code is stored and edited separately in a single source file.
- mikepurvis 8y agoThat's my biggest frustration by far with trying to use Jupyter for anything Serious Business. Will definitely check out your project; thanks for sharing.
- y4mi 8y ago> for not being able to use my editor At least Atom has good integration with hydrogen
- bodkan 8y agoThis looks incredible! Does your project already support other language kernels than the Python kernel? I use R for 90% of my work, but most of it has been happening in Jupyter notebooks (which I'm not a huge fan of, despite practically living in them for the past 4 years of my life). Thanks for sharing!
- 8y ago
- ChuckMcM 8y agoI see Jupyter notebooks as the next step in spreadsheets with more code foundation and different media support. For me the interface doesn't work as well as I would like but I can see the potential. My brother in law wants something like this for structural analysis reports where the code, data and report are all one thing that can be pulled out and examined. Watching the new iPad announcement today I think this is something that would make an excellent iPad app as well.
- solomatov 8y agoIf you are interested in spreadsheets but with Python, try http://datalore.io/ http://datalore.io/ It tracks dependencies automatically and recalculates them when needed (this behavior can also be made less automatic).
- azag0 8y agoAlready now you can run a Jupyter notebook on an http server and serve it to Safari in an iPad.
- selimthegrim 8y agoYour brother in law’s wish is granted. Blockpad is specifically targeted towards such reports https://blockpad.net https://blockpad.net
- rb808 8y agoI used to use Excel extensively. I've started using Jupyter as a replacement. Some things are great like python modules that can do anything, and visualizations. But if there are less than 100k rows, its still much easier to just use Excel. I'm kinda disappointed. IF you have more than 100k rows then Excel starts to be cumbersome. That is the sweet spot for me.
- huac 8y agoyeah, i have a use case now where we pull data from a database, manipulate it, and then have a final table/csv/dataframe/whatever. the problem is then how to share this with non-technical users. in an ideal world, this would get inserted into a google sheet, and that sheet would just update daily after new data is loaded into the database. i'm pretty sure this is a usecase which others have and curious what people use to solve it. i've heard, variously, that some options are to use tableau/similar or email a csv and ask the end user to import into google sheets/excel
- tcpekin 8y agoHaving spent a decent amount of time learning to be a programmer while doing scientific image analysis in Matlab (shudders from the real programmers), and with a decent amount of time spent in Mathematica as well, I just can't seem to buy into the Jupyter/notebook based programming enthusiasm. The talk linked in the article explains it better than I ever could, but for me, when I am leaving data in memory, it is much more convenient for me to have a completely linear history, ordered by command execution time. In python I have found the best way to do this is writing standard python functions and scripts, and running them in an IPython environment with the %run magic. You have the linear history, git works well on standard .py files, and you can interactively work with the data in the IPython prompt without worrying that something is proceeding nonlinearly. What I find works best is to explore with the data in the live prompt, which gives you interactivity, and then build up slowly a master collection of functions and commands that when run, with a single command, can reproduce the results you got while exploring. Then to come back to the data at a later point in time, you have to run one script file on the raw data. Of course, this is kind of the point of the Jupyter notebook, but I find that often when I want to change parameters, the ability to jump around and redefine things means I do. By moving from IPython to a script/functions I run, I ensure that everything progresses linearly. Idk, just my two cents.
- jackbravo 8y agoWhich talk? I didn't see any link to a video.
- tcpekin 8y agoHere they are - they're definitely irreverent, but I find myself strongly agreeing with everything he says. https://docs.google.com/presentation/d/1n2RlMdmv1p25Xy5thJUhkKGvjtV-dkAIsUXP-AL4ffI/edit#slide=id.g3d168d2fd3_0_130 https://docs.google.com/presentation/d/1n2RlMdmv1p25Xy5thJUh... https://www.youtube.com/watch?v=7jiPeIFXb6U&feature=youtu.be https://www.youtube.com/watch?v=7jiPeIFXb6U&feature=youtu.be
- zmmmmm 8y agoAs with so many things python related (including python itself), I am perplexed by how willing people seem to be to fall in love with solutions that have so many limitations and problems. I find Jupyter just barely usable. I constantly have issues with editing in the cells, diagrams not sizing correctly, cells accidentally displaying huge amounts of data and freezing my browser, complete failure of autocompletion in many languages, a very awkward security model involving manual cutting and pasting of auth tokens around, nearly impossible to get a reasonable rendering of the notebook into something reasonable like PDF (yes there attempts at solutions, they are full of problems). Many limitations derive directly from the architecture where the kernels are limited in what they can do because specific parts have to be interpreted in the browser that are language specific. From my perspective, it's a dumpster fire - in 2018 there should be something so much better than this. RStudio is a thousand times better but only does R. I used to like Beaker Notebook but it gave up due to Jupyter's popularity and converted itself into a bunch of Jupyter extensions which now have all of Jupyter's limitations. Yet despite all this I can see that there's this enormous community that loves this and keeps developing and contributing to it.
- stult 8y agoI love everything about RStudio, except for all the R stuff.
- gaius 8y agoit's a dumpster fire - in 2018 there should be something so much better than this In 1998 I was using a tool called MathCAD that provided a notebook interface running as a plugin to MS Word. In 2018, Jupyter is still not as good as that. Some things are just not meant to be webpages.
- deleted 8y ago[deleted]
- ontouchstart 8y agoMany comments are about implementation details such as JSON format of Jupyter and comparing user experience with IDE and shell and fail to see the fundamental difference between Emacs and Jupyter, which is in this quote in the article: “In many cases, it’s much easier to move the computer to the data than the data to the computer,” says Pérez of Jupyter’s cloud-based capabilities. “What this architecture helps to do is to say, you tell me where your data is, and I’ll give you a computer right there.”
- ianamartin 8y agoThe problem with Jupyter isn't with what it does. It's the people who use it. My experience as a data engineer/architect/application developer attached to data science teams for a while now is that most really good data scientists are very good at what they do, write somewhat competent code, and do not--in any way--care about writing good software or good application code. Jupyter is a bane of my existence because people who use it want to use it for everything. Oh, it can have a web interface? Okay. The app is done. DEPLOY TO WEB USERS! NOW!! It's a great tool. A lot of the people who use it are not software engineers, and they don't want to be. For a lot of people it's the straight line from point a to point b. But in my experience, legit data scientists are pretty smart and are willing to learn a little if you're willing to give a little. This is a good exercise because they are typically skeptical about everything. So you have to be really secure about why you want certain things done certain ways, and why you definitely don't want things done other ways. It's a good exercise for everyone involved if you have the right team dynamic and mutual, healthy respect for each other. If you don't . . . well, then Jupyter notebooks completely suck.
- throwaway487548 8y agoJupyter could be viewed as a modern reminiscence of Lisp Machine UI, without the elegance of homoiconicity, of course. Python is a good "glue" for optimized C++ or fortran libraries which the core of things like tensorflow or numpy. Everything fits together nicely.
- lispm 8y ago> Jupyter could be viewed as a modern reminiscence of Lisp Machine UI maybe some part of it, but the Lisp Machine UI has a full window system, many different applications based on it with different UIs (font editor, file system browser, process overview, chat program, terminal, Zmacs editor, debugger, documentation browser, documentation editor, drawing program, ...)
- dual_basis 8y agoThat is what Jupyter Lab is becoming.
- lispm 8y agogreat, but how is it Lisp Machine UI like?
- jxramos 8y ago"Two additional tools have enhanced Jupyter’s usability. One is JupyterHub, a service that allows institutions to provide Jupyter notebooks to large pools of users. The IT team at the University of California, Berkeley, where Pérez is a faculty member, has deployed one such hub, which Pérez uses to ensure that all students on his data-science course have identical computing environments. “We cannot possibly manage IT support for 800 students, helping them debug why the installation on their laptop is not working; that’s simply infeasible,” he says." I think this result is a real winner, I recall the problems of setup in university student labs. Good win for reducing teaching friction.