21 ms·

Introduction to Pluto.jl

161 points by joshday 5y ago

spinningslate 5y ago
I'm always impressed by the quality of the Julia ecosystem. It seems to be in that sweet spot with sufficient use & contribution to be viable, but not so popular that quality suffers.
- machineko 5y ago
  https://madnight.github.io/githut/#/pull_requests/2021/1 https://madnight.github.io/githut/#/pull_requests/2021/1 Just saying but julia open source is kinda dead outside few very high quality packages
- kruxigt 5y ago
  Yes! Hope it stays that way for a long time. The continued popularity of Python is probably good for this. Mostly only the right people move on from Python to Julia.
- teruakohatu 5y ago
  I love Julia and part of its charm is that everything is relatively new and so quite consistent, also helped by the community ethos and technical features that aid composition. Python and R (especially R) have plenty of libraries that are high-quality, or even industry standard, but which are decades old and feel it. Python's NLTK is 20 years old for example and it can feel grating switching between NLTK and spaCy. R has three different object systems (four according to some), so you might be using some ancient battle tested library with Hadley Wickham's latest cutting edge libraries.
  - f6v 5y ago
    R has a terrible naming problem where you don’t know which convention is used in a particular library.
xal 5y ago
It's funny because this is probably a really non-standard sentiment but I really wish that they would make an electron app out of this. Installing it is reasonably easy but definitely beyond a lot of people who could get value from it.
- nerdponx 5y ago
  Normally I dislike Electron apps (with some very well-built exceptions) but in this case it makes perfect sense. It already renders HTML, CSS, and JS!
teruakohatu 5y ago
I have played around with Pluto.jl, and colleagues of mine use it for research, but I keep going back to Jupyter. I tend to have long running cells that are pulling information from external sources or training models, and triggering one of those cells accidentally will waste a lot of time running something that may not be reliably interrupted. There is talk about putting in execution barriers that would help with this, at the risk of making Pluto more complicated for users: https://github.com/fonsp/Pluto.jl/discussions/298 https://github.com/fonsp/Pluto.jl/discussions/298
- nerdponx 5y ago
  FWIW I've significantly improved my experience by breaking up my notebooks into smaller pieces such that each notebook only does "one thing", while using DVC to run them and keep track of intermediate results. Or in a case where the intermedaite result was itself somewhat "exploratory", having the notebook itself check for the existence of an intermediate result and load it from disk instead of recomputing it. Execution barriers are a nice idea though. There is/was a Jupyter notebook extension for "initialization cells", but the whole notebook extension ecosystem seems kind of dead and it's unclear if Jupyter Lab will ever have equivalents.
- dandanua 5y ago
  This can be easily solved. You can bind a variable to a checkbox like this: @bind allow_run html"Run cell below <input type=checkbox>" and wrap your long running cell in the if block: if allow_run your_code end
- oivey 5y ago
  The fact that Pluto only runs dependent cells on changes mostly solves this for me. For example, a cell can load things into the variable data, and then another cell can apply a function f(data). If I alter f, data is not reloaded and f(data) automatically runs.
  - teruakohatu 5y ago
    That is fine if you are working sequentially, but often tasks involve going back to the original data and doing some wrangling. data -> model(data) -> output(model) So if you go back to mess around with the data, your model and output could be or would be recomputed, which you would need to do eventually but not while making iterative tweaks. Another commenter suggested adding checkboxes which is a good idea, although then you are managing a bunch of checkbox states.
nerdponx 5y ago
I really wish the Julia ecosystem would stop assuming that you always interact with your computer through the Julia REPL and started supporting proper command line interfaces. This is one of the big annoyances and mistakes of the R ecosystem, and I think it's unwise to carry that mistake over to Julia. Also, big "ugh" to browser-based tooling. I want to browse webpages in my browser, I don't want to do my data science work there. We don't even have a good native client for Jupyter notebooks yet, let alone for this new Jupyter alternative that doesn't support the existing Jupyter kernel protocol. Not only that, but Pluto also apparently has some obnoxious UX limitations that remind me of other less-than-usable wannabe-Jupyter-notebooks (e.g. Apache Zeppelin, Databricks): https://towardsdatascience.com/could-pluto-be-a-real-jupyter-replacement-6574bfb40cc6 https://towardsdatascience.com/could-pluto-be-a-real-jupyter... In short: nice idea, but I'd rather see continued unification around Jupyter and a proper IDE that can at least emit and interact with Jupyter-compatible data. On the other hand, the Jupyter notebook JSON format is bad for a variety of reasons (e.g. you need special tools for readable Git diffs) and I really wish we had all settled on R Markdown instead. But R has its own NIH tooling problem and nobody was ever going to adopt it because the R community itself (driven by RStudio) has little interest in sharing or interoperability with other languages. </cynical-angry-rant>
- clarkevans 5y ago
  Pluto notebooks are Julia scripts, usable at the command line. Edit: Pluto uses Julia's package manager; moreover, Manifest.toml can be used to pin all of your project's dependencies so the notebook is repeatable, from a code perspective.
  - nerdponx 5y ago
    That's good to know. But I was talking about the package manager and starting the Pluto server.
    - newswasboring 5y ago
      You can start pluto server from command line > julia -e "using Pluto; Pluto.run()" Also, package manager can be used from inside Pluto. To install somethin, you can just write in a cell > using Pkg > Pkg.add("Package Name")
borodi 5y ago
For those that are put off byt the "weird" cell execution behavior there is also https://github.com/compleathorseplayer/Neptune.jl https://github.com/compleathorseplayer/Neptune.jl A non reactive fork of Pluto that has basically all the benefits of pluto and multi-line cell execution without begin without the reactive behaviour. Also running code blocks with inline results in vscode also has some notebook feel to me.
- krastanov 5y ago
  Why would someone use Neptune instead of just using Jupyter? I see how Pluto has a new value proposition that Jupyter lacks (reactivity), but it looks to me like Neptune simply removes that value.
  - ChrisRackauckas 5y ago
    It's also an unmaintained fork. Forked in February and hasn't a commit since to the source. None of the patches are getting downstreamed. It just keeps updating its README and posting more advertising. If someone wants to do this project they should do it correctly, but this is just not how you do that. You'd need to keep floating your patches over a changing master, not just force an old version, force all packages to be on older versions without patches (HTTP.jl), etc.
legerdemain 5y ago
LOL, how often do you want your entire notebook to recompute just because you change something somewhere? Have you never tried pursuing a little side experiment in an existing notebook, or have ten abandoned false starts leading to one good result? I have many extremely long notebooks that would almost certainly crash if you tried to recompute the whole thing, and many of the cells won't work at all because the inputs are long gone. Some of these notebooks are years old. The datasets they have in memory aren't saved anywhere else. What possible motivation do I have to lose all of this precious state? If I wanted a software-grade, rock-solid data pipeline, I would just copy-paste some code from an existing notebook and run it on Papermill.
- lacker 5y ago
  Some of these notebooks are years old. The datasets they have in memory aren't saved anywhere else. That sounds dangerous to me. If your computer crashes or you introduce a bug to your notebook, you could lose all that data. Personally, I prefer my notebooks to be reproducible at any point.
  - yunohn 5y ago
    Exactly, or at the very least, pickle/serialise/export/whatever the models so that the computer can survive a reboot.
    - legerdemain 5y ago
      These are usually small aggregates and summaries, so I just display them in notebook output. It does make it take a bit longer to scroll through the notebook to find something, but that's what being disciplined with organization is for.
      - KMag 5y ago
        Sorry, I'm not sure I'm following your argument. Are you saying your notebooks hold state that's easily reconstitutable, and so it's not actually such a big deal to regenerate your "precious state"?
maximilianroos 5y ago
I used Pluto for last year's Advent of Code. It's extremely good for these sorts of problems — rapid iteration with modest computational requirements. Think of something you might use a spreadsheet for — Pluto has a similar feeling of instant feedback. --- Some features that are missing: – Some things are difficult to do with the keyboard; I used my mouse more than with other tools. The author doesn't like modal editing, but ideally they could be implemented with modifier keys (https://github.com/fonsp/Pluto.jl/issues/65 https://github.com/fonsp/Pluto.jl/issues/65) - It's hard to understand what happens _within_ a cell — logging goes to the terminal rather than the notebook — and there aren't many introspection tools. This is an environment where transparency / introspection would be particularly helpful. --- Pluto doesn't solve every problem, or completely replace notebooks; to respond to a couple of comments: > I have many extremely long notebooks that would almost certainly crash if you tried to recompute the whole thing Right, don't use Pluto for that! It's not one environment to rule them all > Many of the cells won't work at all because the inputs are long gone That seems bad! Pluto will help you ensure that doesn't happen.
LetThereBeLight 5y ago
I know that this site mentions MIT's Introduction to Computational Thinking course, but the hyperlink doesn't send me there. For those interested in seeing Pluto in action I highly recommend checking out the course notebooks here: https://computationalthinking.mit.edu/Spring21/ https://computationalthinking.mit.edu/Spring21/
- joshday 5y ago
  My apologies! I’ve fixed the link.
dandanua 5y ago
I don't get why people dislike reactivity. This feature alone makes Pluto superior to Jupyter. If you don't want recomputation of some dependent cells there are easy ways to avoid that. But there are no easy ways to add reactivity to Jupyter. Besides that, Pluto can bind UI elements to your code. You can make simple interactive games that run in Pluto! How it's not awesome?
_ZeD_ 5y ago
I don't want to be that guy, but it seems to me those tools are converging to ... excel spreadsheets
- dagw 5y ago
  An 'Excel' that is less opaque, easier to test and debug and backed by a more sane and powerful language is what a lot of the world is clamoring for. So yea, that would be great.
shusson 5y ago
> When you change a variable, that change gets propagated through all cells which reference that variable. I've always thought this was the most annoying quirk of notebooks in general, so it's nice to see a different take.
truth_ 5y ago
I found this a while ago in HN- https://towardsdatascience.com/my-first-encounter-with-julia-15777c6189f9 https://towardsdatascience.com/my-first-encounter-with-julia... It was much better.
enriquto 5y ago
I like the idea of Pluto, because I cannot stand the non-deterministic cells of Jupyter notebooks anymore. Reading this page is like having sex with someone you love. Where has Pluto been all this time? I have finally found all what was missing for a complete life! There's even things that I didn't know I needed because I didn't even have the language to express them! This is my favorite page on the internet and Pluto is my favorite thing ever. I can see no downside to this, no defects, even with a conscious effort to do so. Yet, trying Pluto, it seems to be outrageously slow and clunky. Is it expected? Sometimes it takes a few seconds to do something. I'm not talking about the initialization (which is still a shame, but that's a different issue). I'm talking about running individual cells with simple code. This is unusable as of today, at least on my 3-year old laptop.
- joppy 5y ago
  Pluto is quite fast for me - could you perhaps be hitting the first-run JIT startup time in Julia? Do the cells re-evaluate quickly, after whatever code they depend on has been JITted?
  - enriquto 5y ago
    I'm talking about my second run of the notebook. On the first run it took one minute and a half just to open the notebook (it seemed it was downloading stuff, and then compiling).
- newswasboring 5y ago
  hey, did you use Julia 1.5 or 1.6? There is a massive improvement in latency between those two versions.
  - enriquto 5y ago
    I'm using 1.5.3. Gonna update to 1.6 and see what happens. EDIT: just running it on julia's "master" branch (v1.7.0-DEV), the initialization seems to be slower, but then the cells run maybe marginally faster. Looks good, but I could not push this to my students yet...
mark_l_watson 5y ago
Have cells reactive immediately to variable changes in other cells is great. I wish Jupiter did that. I also wish I had an excuse to get more into Julia. I really like Flux.
- dunefox 5y ago
  Sometimes the only excuse you need is interest.
RocketSyntax 5y ago
so every time i change a variable i train my neural network? yikes. non-linear <3 front end programmers coding for data science use case <x3

161 points by joshday 5y ago

I'm always impressed by the quality of the Julia ecosystem. It seems to be in that sweet spot with sufficient use & contribution to be viable, but not so popular that quality suffers.

machineko 5y ago
https://madnight.github.io/githut/#/pull_requests/2021/1 https://madnight.github.io/githut/#/pull_requests/2021/1 Just saying but julia open source is kinda dead outside few very high quality packages
kruxigt 5y ago
Yes! Hope it stays that way for a long time. The continued popularity of Python is probably good for this. Mostly only the right people move on from Python to Julia.
teruakohatu 5y ago
I love Julia and part of its charm is that everything is relatively new and so quite consistent, also helped by the community ethos and technical features that aid composition. Python and R (especially R) have plenty of libraries that are high-quality, or even industry standard, but which are decades old and feel it. Python's NLTK is 20 years old for example and it can feel grating switching between NLTK and spaCy. R has three different object systems (four according to some), so you might be using some ancient battle tested library with Hadley Wickham's latest cutting edge libraries.
- f6v 5y ago
  R has a terrible naming problem where you don’t know which convention is used in a particular library.

xal 5y ago

It's funny because this is probably a really non-standard sentiment but I really wish that they would make an electron app out of this. Installing it is reasonably easy but definitely beyond a lot of people who could get value from it.

nerdponx 5y ago
Normally I dislike Electron apps (with some very well-built exceptions) but in this case it makes perfect sense. It already renders HTML, CSS, and JS!

teruakohatu 5y ago

I have played around with Pluto.jl, and colleagues of mine use it for research, but I keep going back to Jupyter. I tend to have long running cells that are pulling information from external sources or training models, and triggering one of those cells accidentally will waste a lot of time running something that may not be reliably interrupted. There is talk about putting in execution barriers that would help with this, at the risk of making Pluto more complicated for users: https://github.com/fonsp/Pluto.jl/discussions/298 https://github.com/fonsp/Pluto.jl/discussions/298

nerdponx 5y ago
FWIW I've significantly improved my experience by breaking up my notebooks into smaller pieces such that each notebook only does "one thing", while using DVC to run them and keep track of intermediate results. Or in a case where the intermedaite result was itself somewhat "exploratory", having the notebook itself check for the existence of an intermediate result and load it from disk instead of recomputing it. Execution barriers are a nice idea though. There is/was a Jupyter notebook extension for "initialization cells", but the whole notebook extension ecosystem seems kind of dead and it's unclear if Jupyter Lab will ever have equivalents.
dandanua 5y ago
This can be easily solved. You can bind a variable to a checkbox like this: @bind allow_run html"Run cell below <input type=checkbox>" and wrap your long running cell in the if block: if allow_run your_code end
oivey 5y ago
The fact that Pluto only runs dependent cells on changes mostly solves this for me. For example, a cell can load things into the variable data, and then another cell can apply a function f(data). If I alter f, data is not reloaded and f(data) automatically runs.
- teruakohatu 5y ago
  That is fine if you are working sequentially, but often tasks involve going back to the original data and doing some wrangling. data -> model(data) -> output(model) So if you go back to mess around with the data, your model and output could be or would be recomputed, which you would need to do eventually but not while making iterative tweaks. Another commenter suggested adding checkboxes which is a good idea, although then you are managing a bunch of checkbox states.

nerdponx 5y ago

I really wish the Julia ecosystem would stop assuming that you always interact with your computer through the Julia REPL and started supporting proper command line interfaces. This is one of the big annoyances and mistakes of the R ecosystem, and I think it's unwise to carry that mistake over to Julia. Also, big "ugh" to browser-based tooling. I want to browse webpages in my browser, I don't want to do my data science work there. We don't even have a good native client for Jupyter notebooks yet, let alone for this new Jupyter alternative that doesn't support the existing Jupyter kernel protocol. Not only that, but Pluto also apparently has some obnoxious UX limitations that remind me of other less-than-usable wannabe-Jupyter-notebooks (e.g. Apache Zeppelin, Databricks): https://towardsdatascience.com/could-pluto-be-a-real-jupyter-replacement-6574bfb40cc6 https://towardsdatascience.com/could-pluto-be-a-real-jupyter... In short: nice idea, but I'd rather see continued unification around Jupyter and a proper IDE that can at least emit and interact with Jupyter-compatible data. On the other hand, the Jupyter notebook JSON format is bad for a variety of reasons (e.g. you need special tools for readable Git diffs) and I really wish we had all settled on R Markdown instead. But R has its own NIH tooling problem and nobody was ever going to adopt it because the R community itself (driven by RStudio) has little interest in sharing or interoperability with other languages. </cynical-angry-rant>

clarkevans 5y ago
Pluto notebooks are Julia scripts, usable at the command line. Edit: Pluto uses Julia's package manager; moreover, Manifest.toml can be used to pin all of your project's dependencies so the notebook is repeatable, from a code perspective.
- nerdponx 5y ago
  That's good to know. But I was talking about the package manager and starting the Pluto server.
  - newswasboring 5y ago
    You can start pluto server from command line > julia -e "using Pluto; Pluto.run()" Also, package manager can be used from inside Pluto. To install somethin, you can just write in a cell > using Pkg > Pkg.add("Package Name")

borodi 5y ago

For those that are put off byt the "weird" cell execution behavior there is also https://github.com/compleathorseplayer/Neptune.jl https://github.com/compleathorseplayer/Neptune.jl A non reactive fork of Pluto that has basically all the benefits of pluto and multi-line cell execution without begin without the reactive behaviour. Also running code blocks with inline results in vscode also has some notebook feel to me.

krastanov 5y ago
Why would someone use Neptune instead of just using Jupyter? I see how Pluto has a new value proposition that Jupyter lacks (reactivity), but it looks to me like Neptune simply removes that value.
- ChrisRackauckas 5y ago
  It's also an unmaintained fork. Forked in February and hasn't a commit since to the source. None of the patches are getting downstreamed. It just keeps updating its README and posting more advertising. If someone wants to do this project they should do it correctly, but this is just not how you do that. You'd need to keep floating your patches over a changing master, not just force an old version, force all packages to be on older versions without patches (HTTP.jl), etc.

legerdemain 5y ago

LOL, how often do you want your entire notebook to recompute just because you change something somewhere? Have you never tried pursuing a little side experiment in an existing notebook, or have ten abandoned false starts leading to one good result? I have many extremely long notebooks that would almost certainly crash if you tried to recompute the whole thing, and many of the cells won't work at all because the inputs are long gone. Some of these notebooks are years old. The datasets they have in memory aren't saved anywhere else. What possible motivation do I have to lose all of this precious state? If I wanted a software-grade, rock-solid data pipeline, I would just copy-paste some code from an existing notebook and run it on Papermill.

lacker 5y ago
Some of these notebooks are years old. The datasets they have in memory aren't saved anywhere else. That sounds dangerous to me. If your computer crashes or you introduce a bug to your notebook, you could lose all that data. Personally, I prefer my notebooks to be reproducible at any point.
- yunohn 5y ago
  Exactly, or at the very least, pickle/serialise/export/whatever the models so that the computer can survive a reboot.
  - legerdemain 5y ago
    These are usually small aggregates and summaries, so I just display them in notebook output. It does make it take a bit longer to scroll through the notebook to find something, but that's what being disciplined with organization is for.
    - KMag 5y ago
      Sorry, I'm not sure I'm following your argument. Are you saying your notebooks hold state that's easily reconstitutable, and so it's not actually such a big deal to regenerate your "precious state"?

maximilianroos 5y ago

I used Pluto for last year's Advent of Code. It's extremely good for these sorts of problems — rapid iteration with modest computational requirements. Think of something you might use a spreadsheet for — Pluto has a similar feeling of instant feedback. --- Some features that are missing: – Some things are difficult to do with the keyboard; I used my mouse more than with other tools. The author doesn't like modal editing, but ideally they could be implemented with modifier keys (https://github.com/fonsp/Pluto.jl/issues/65 https://github.com/fonsp/Pluto.jl/issues/65) - It's hard to understand what happens _within_ a cell — logging goes to the terminal rather than the notebook — and there aren't many introspection tools. This is an environment where transparency / introspection would be particularly helpful. --- Pluto doesn't solve every problem, or completely replace notebooks; to respond to a couple of comments: > I have many extremely long notebooks that would almost certainly crash if you tried to recompute the whole thing Right, don't use Pluto for that! It's not one environment to rule them all > Many of the cells won't work at all because the inputs are long gone That seems bad! Pluto will help you ensure that doesn't happen.

LetThereBeLight 5y ago

I know that this site mentions MIT's Introduction to Computational Thinking course, but the hyperlink doesn't send me there. For those interested in seeing Pluto in action I highly recommend checking out the course notebooks here: https://computationalthinking.mit.edu/Spring21/ https://computationalthinking.mit.edu/Spring21/

joshday 5y ago
My apologies! I’ve fixed the link.

dandanua 5y ago

I don't get why people dislike reactivity. This feature alone makes Pluto superior to Jupyter. If you don't want recomputation of some dependent cells there are easy ways to avoid that. But there are no easy ways to add reactivity to Jupyter. Besides that, Pluto can bind UI elements to your code. You can make simple interactive games that run in Pluto! How it's not awesome?

_ZeD_ 5y ago

I don't want to be that guy, but it seems to me those tools are converging to ... excel spreadsheets

dagw 5y ago
An 'Excel' that is less opaque, easier to test and debug and backed by a more sane and powerful language is what a lot of the world is clamoring for. So yea, that would be great.

shusson 5y ago

> When you change a variable, that change gets propagated through all cells which reference that variable. I've always thought this was the most annoying quirk of notebooks in general, so it's nice to see a different take.

truth_ 5y ago

I found this a while ago in HN- https://towardsdatascience.com/my-first-encounter-with-julia-15777c6189f9 https://towardsdatascience.com/my-first-encounter-with-julia... It was much better.

enriquto 5y ago

I like the idea of Pluto, because I cannot stand the non-deterministic cells of Jupyter notebooks anymore. Reading this page is like having sex with someone you love. Where has Pluto been all this time? I have finally found all what was missing for a complete life! There's even things that I didn't know I needed because I didn't even have the language to express them! This is my favorite page on the internet and Pluto is my favorite thing ever. I can see no downside to this, no defects, even with a conscious effort to do so. Yet, trying Pluto, it seems to be outrageously slow and clunky. Is it expected? Sometimes it takes a few seconds to do something. I'm not talking about the initialization (which is still a shame, but that's a different issue). I'm talking about running individual cells with simple code. This is unusable as of today, at least on my 3-year old laptop.

joppy 5y ago
Pluto is quite fast for me - could you perhaps be hitting the first-run JIT startup time in Julia? Do the cells re-evaluate quickly, after whatever code they depend on has been JITted?
- enriquto 5y ago
  I'm talking about my second run of the notebook. On the first run it took one minute and a half just to open the notebook (it seemed it was downloading stuff, and then compiling).
newswasboring 5y ago
hey, did you use Julia 1.5 or 1.6? There is a massive improvement in latency between those two versions.
- enriquto 5y ago
  I'm using 1.5.3. Gonna update to 1.6 and see what happens. EDIT: just running it on julia's "master" branch (v1.7.0-DEV), the initialization seems to be slower, but then the cells run maybe marginally faster. Looks good, but I could not push this to my students yet...

mark_l_watson 5y ago

Have cells reactive immediately to variable changes in other cells is great. I wish Jupiter did that. I also wish I had an excuse to get more into Julia. I really like Flux.

dunefox 5y ago
Sometimes the only excuse you need is interest.

RocketSyntax 5y ago

so every time i change a variable i train my neural network? yikes. non-linear <3 front end programmers coding for data science use case <x3