11 ms·

Livebook: A collaborative and interactive code notebook for Elixir

585 points by bcardarella 5y ago

seanhandley 5y ago
So this is like Jupyter notebooks for Elixir?
- rajandatta 5y ago
  Yes - it would seem so. The support for custom run times is interesting and seems like an improvement. Not sure Jupyter notebooks do that - certainly not for all supported kernels.
  - nerdponx 5y ago
    Jupyter is great but it's somewhat of a "lowest common denominator". So there is always the temptation to make a more task-specific version; see also the Pluto notebook project for Julia and R Markdown (although the latter evolved concurrently with IPython/Jupyter, not after it). LSP is in a similar situation right now, where it's great for languages that lacked tooling to begin with (like Python), but it's not yet possible to migrate existing powerful language tooling (Lisp SLIME, OCaml Merlin) to use LSP.
    - c-cube 5y ago
      For the record, ocamllsp uses merlin under the hood, and other tools, so it can be a more integrated experience and be easier to setup.
- lawik 5y ago
  With google docs style collaborative editing, can connect to an existing cluster to run code in it. You can run one instance of the Livebook application and have different livebook docs run for different Elixir projects you happen to be working on. Not sure what parts are in Jupyter but I've heard collab editing isn't. Pretty hype about trying this for some living docs and teaching. Will see if I can do anything neat with it. Already tried it some.
  - bostonvaulter2 5y ago
    Yeah I think this will be really neat when paired with an example project for libraries.
  - aleDsz 5y ago
    I used to teach my viewers of the basics of Elixir. It's in Brazilian Portuguese, but it's kinda interesting how easy is to use Livebook and show some code for people. https://www.twitch.tv/videos/991797943 https://www.twitch.tv/videos/991797943
- matreyes 5y ago
  If you compare the work done in both projects (github insights), you will feel the power of the Beam, Elixir, Phoenix and LiveView.
dnautics 5y ago
"connect to an existing node" is going to be amazing. I imagine two interesting uses for this. 1. producing reports. Connect a livebook session to your system in prod, run a livebook (which pulls data from DB eg), then get back a report. User engagement, cloud resources consumed, p90s, or an incident report... 2. manual database intervention. Suppose you need to manually make a batch of changes to the DB. It would be very cool if you could run a livebook inside of a sandboxed checkout (a temporary transaction) and write your db changes, then write your code that verifies that the db is in a state that you want it to be in. If you mess up your code, then you just terminate the transaction without committing. Then when you're happy with your notebook, you run it once with a commit at the end, and then save your notebook immutably in your records so that you can keep track of what you've done. I think for intervening in prod, it would be very cool if someone could figure out how to extend erlang distribution over websockets so you don't have to figure out complicated mechanisms (open up ssh port, forward encrypted epmd, e.g.) to get into your system in prod
- qbasic_forever 5y ago
  Be careful, websockets are tricky to secure and get right. It's easy to do something like forget to check the origin and now you've opened up a whole class of cross-site websocket scripting attacks. I love the idea of easy access and nice UI to prod machines, but put it behind a secure proxy or VPN layer and not facing the public internet.
  - dnautics 5y ago
    That's very reasonable.
  - bcardarella 5y ago
    Phoenix Channels accounts for many of these attack vectors. Livebook uses Phoenix LiveView which is built on Phoenix Channels.
  - wrren 5y ago
    'Connect to an existing node' is likely implemented using ERTS (http://erlang.org/doc/reference_manual/distributed.html http://erlang.org/doc/reference_manual/distributed.html). So you're basically connecting to the runtime of the target node. The security model for ERTS isn't great, though, so your point stands. All you need is a matching secure cookie and you can connect and run any commands you want.
talknewswale 5y ago
डोनाल्ड ट्रम्प फेसबुक, इंस्टाग्राम को फिर से कब कर पाएंगे इस्तेमाल? https://talknewswale.com https://talknewswale.com
mstipetic 5y ago
These guys are amazingly productive, while also running a commercial company
- fastball 5y ago
  This is software that directly helps their business, so it's not really productivity outside of their business endeavors.
  - xcambar 5y ago
    Try not to sound like you're diminishing the efforts and accomplishments of others. This is tremendous work, they're running their business and come up with generic/generalisable solutions instead of narrowing the outcome to their own benefits only.
    - fastball 5y ago
      When Facebook spends engineering effort to work on React, it's not out of the kindness of their heart, and that's ok. It's still productive and an accomplishment, it's just productive for a reason that pertains to their business. It is the same here. This tool helps the adoption of Elixir, running Elixir is their business, so yay! But the comment I was replying to implied this is somehow entirely separate from their business, which it's not. Not really sure how it sounded like I was diminishing the accomplishments of others. How would you have phrased the point?
      - mstipetic 5y ago
        FB has thousands of engineers. It's not that often that a small group of people can both keep the lights running financially and also develop fully featured massive libraries. I'd say that counts as a very productive company.
        fastball 5y ago
        Right, but again my point is that developing this fully-featured library helps keep the lights on. The two things are not separate as you keep implying. Not disagreeing that they are productive and accomplishing a lot.
Pandabob 5y ago
Is it just me or is Elixir being talked/written about more in the past couple of months?
- cpursley 5y ago
  Years.
  - cpursley 5y ago
    Interesting that I'm getting down voted, I'm long on Elixir and betting my startup on it.
    - Ankhers 5y ago
      I didn't down vote you, but my guess is that it was because you commented a single word. Also, because of the position of your comment, I had to scroll back up in order to find the context.
- SatvikBeri 5y ago
  There were 300 elixir stories in 2020, an average of roughly .8 per day. In 2021 there have been 89 elixir stories for 108 days, so about the same. (Source: hnsearch)
  - bostonvaulter2 5y ago
    Is it possible to tell how many of those hit the front page? I think the majority of those 300 stories in 2020 did not hit the front page. I'm curious how many there were in years prior as well.
- lawik 5y ago
  It is growing and having good adoption. Not a huge language but it also has a fair share of novel features (being built on Erlang) that aren't commonly seen with other langs. So I think it might have a disproportionate visibility compared to size. Lots of work happening in the community, lots of ambitious ideas and releases.
out_of_protocol 5y ago
Readable `.livemd` format looks lovely (regular markdown without extra elements) http://lon-screenshots.s3.amazonaws.com/2021-04-18_Screenshot_20210418.jpg http://lon-screenshots.s3.amazonaws.com/2021-04-18_Screensho...
AlchemistCamp 5y ago
I'll admit that I was initially skeptical about Elixir for anything ML related when I first picked up the pragprog book on Elixir and Genetic Algorithms. Since that time, things have moved quite a bit. Between the Erlang VM JIT, then Nx, and now this, it's starting to look hopeful! I'm looking forward to digging into this.
CTmystery 5y ago
Awesome work! Towards the end, José hits on a couple points that Jupyter does not do well: - Minute 25: Saved notebook is readable (markdown) - Minute 23: Live collaborate on a single notebook - Minute 22: Visual indication that cell is stale after an upstream cell changes
- srush 5y ago
  I've been playing around with some of these ideas for Python notebooks. https://github.com/srush/streambook https://github.com/srush/streambook This is a proof of concept that combines Jupytext for markdown readable notebooks with Streamlit for in-order execution+caching. More difficult to get some of the more advanced features of Pluto / Livebook due to Python state management. Curious to hear any thoughts.
  - O_H_E 5y ago
    Would you be able to expand or drop a few keywords on how (and what is) "python state management" prohibit few advance features. By in-order-execution you mean pluto's reactivity, right.
transfire 5y ago
All this cool Elixir news. I'm really having a hard time deciding between Elixir and Julia. (For programs that I must have compiled to a distributable executable, I've pretty much settled on Crystal. I find it more productive and pleasent than Go or Rust.)
- thruflo22 5y ago
  It’s not quite the same story but https://github.com/spawnfest/bakeware https://github.com/spawnfest/bakeware builds elixir releases into distributable executables.
  - lobo_tuerto 5y ago
    That one is out of date, the one it forked of is not: https://github.com/bake-bake-bake/bakeware https://github.com/bake-bake-bake/bakeware
- cultofmetatron 5y ago
  if you want to build a saas, go with elixir. Its got a great batteries included web framework thats built to scale and an emerging suit of ml and linear algebra software coming out. If you want to experiment with ML and scientific programming where you do a lot of matrix processing, go with julia. thats literally what its made for.
  - neolog 5y ago
    Julia has a web framework too of course. https://genieframework.com/ https://genieframework.com/
- neolog 5y ago
  https://github.com/JuliaLang/PackageCompiler.jl https://github.com/JuliaLang/PackageCompiler.jl
- dnautics 5y ago
  As someone who started on julia first and then moved to elixir: If you're doing computational/numerical stuff, stick with julia, for now. You really can't do that elixir, not today, and maybe not for at least another year or two. It will likely never be as good as julia since computation is not first class in the vm it runs on. If you're doing or planning or doing anything with web, or orchestration, go with elixir.
namelosw 5y ago
Wow, this is great. The fact is can connect to existing nodes also makes it can also be great for operation and administration - instead of building an awkward admin panel, you can just having a collection of scripts. The Elixir community is insanely productive. Also, the Elixir system is impressively pretty balanced - productivity-wise like Rails, concurrent-wise like Erlang, and continuously brings novel yet practical ideas to life in a robust manner!
cancan 5y ago
Love how prolific the elixir community and especially José is. Also, if you are into liveview/elixir, definitely dive into the codebase. There is a small operational-transform lib built in there too.
losvedir 5y ago
I'm pretty excited about this. I'm admit I was skeptical when I saw a similar LiveView based notebook a couple months ago, and wondered why not just make a kernel for Jupyter. But with the idea of real-time collaboration I get it. And with the momentum of Jose and team behind it, I think there's real promise here, especially once they figure out widgets and charts, which they have an open issue for. I program in Elixir at work, so I usually have an iex shell open, but lately I've been using a LiveBook and it's great. And I love the new Mix.install in Elixir 1.12 demonstrated here. Using a hex package in an adhoc way was a real shortcoming in Elixir before. Shows the power of having this thing developed by the language team.
- filmor 5y ago
  There are already Jupyter kernels, IElixir and ierl (which I wrote). The latter supports Erlang, Elixir and LFE right now and supports connecting to another node as well. https://github.com/filmor/ierl https://github.com/filmor/ierl
hades32 5y ago
Super nice! It's only missing auto-complete and it'll be light-years ahead of other tools on terms of usability
- chrismccord 5y ago
  There’s already a PR for it with a video of autocomplete in action :) https://github.com/elixir-nx/livebook/pull/208 https://github.com/elixir-nx/livebook/pull/208
  - losvedir 5y ago
    I'm just gobsmacked at how fast they're moving here. This is incredible.
spicyramen 5y ago
If I'm a data scientists with a bunch of existing Python notebooks can I use this product?
- astrowilson 5y ago
  Not yet. It only supports Elixir for now (there is an issue for supporting other languages) and the notebook formats are different, so someone would also need to write a nbconvert to Markdown (.livemd).
ch4s3 5y ago
I'm not sure how I'm going to use this yet but I think there are going to be some great opportunities for prototyping here.
aleDsz 5y ago
I'm using this tool to teach the basics of Elixir on my livestream. I think there's a lot of good usecases for Livebook, not only machine learning and stuff like that.
rkss 5y ago
Would like to know how Livebook achieve collaborate-editing. Does it use algorithms like CRDT or OT ?
- lawik 5y ago
  OT. Using quill js on the frontend and a subset of that delta format handled on the backend. From conversation with Jose on Twitter.
cocco 5y ago
Using it everyday. Great tool.
cocco 5y ago
Using it everyfay. Greate tool.
arthurcolle 5y ago
How to use dependencies within a livebook? I just tried to use the enum_type (version 1.1.3) by just installing it inside the cloned livebook repo, but I'm getting the error that EnumType isn't loaded. Why is this?

585 points by bcardarella 5y ago

seanhandley 5y ago

So this is like Jupyter notebooks for Elixir?

rajandatta 5y ago
Yes - it would seem so. The support for custom run times is interesting and seems like an improvement. Not sure Jupyter notebooks do that - certainly not for all supported kernels.
- nerdponx 5y ago
  Jupyter is great but it's somewhat of a "lowest common denominator". So there is always the temptation to make a more task-specific version; see also the Pluto notebook project for Julia and R Markdown (although the latter evolved concurrently with IPython/Jupyter, not after it). LSP is in a similar situation right now, where it's great for languages that lacked tooling to begin with (like Python), but it's not yet possible to migrate existing powerful language tooling (Lisp SLIME, OCaml Merlin) to use LSP.
  - c-cube 5y ago
    For the record, ocamllsp uses merlin under the hood, and other tools, so it can be a more integrated experience and be easier to setup.
lawik 5y ago
With google docs style collaborative editing, can connect to an existing cluster to run code in it. You can run one instance of the Livebook application and have different livebook docs run for different Elixir projects you happen to be working on. Not sure what parts are in Jupyter but I've heard collab editing isn't. Pretty hype about trying this for some living docs and teaching. Will see if I can do anything neat with it. Already tried it some.
- bostonvaulter2 5y ago
  Yeah I think this will be really neat when paired with an example project for libraries.
- aleDsz 5y ago
  I used to teach my viewers of the basics of Elixir. It's in Brazilian Portuguese, but it's kinda interesting how easy is to use Livebook and show some code for people. https://www.twitch.tv/videos/991797943 https://www.twitch.tv/videos/991797943
matreyes 5y ago
If you compare the work done in both projects (github insights), you will feel the power of the Beam, Elixir, Phoenix and LiveView.

dnautics 5y ago

"connect to an existing node" is going to be amazing. I imagine two interesting uses for this. 1. producing reports. Connect a livebook session to your system in prod, run a livebook (which pulls data from DB eg), then get back a report. User engagement, cloud resources consumed, p90s, or an incident report... 2. manual database intervention. Suppose you need to manually make a batch of changes to the DB. It would be very cool if you could run a livebook inside of a sandboxed checkout (a temporary transaction) and write your db changes, then write your code that verifies that the db is in a state that you want it to be in. If you mess up your code, then you just terminate the transaction without committing. Then when you're happy with your notebook, you run it once with a commit at the end, and then save your notebook immutably in your records so that you can keep track of what you've done. I think for intervening in prod, it would be very cool if someone could figure out how to extend erlang distribution over websockets so you don't have to figure out complicated mechanisms (open up ssh port, forward encrypted epmd, e.g.) to get into your system in prod

qbasic_forever 5y ago
Be careful, websockets are tricky to secure and get right. It's easy to do something like forget to check the origin and now you've opened up a whole class of cross-site websocket scripting attacks. I love the idea of easy access and nice UI to prod machines, but put it behind a secure proxy or VPN layer and not facing the public internet.
- dnautics 5y ago
  That's very reasonable.
- bcardarella 5y ago
  Phoenix Channels accounts for many of these attack vectors. Livebook uses Phoenix LiveView which is built on Phoenix Channels.
- wrren 5y ago
  'Connect to an existing node' is likely implemented using ERTS (http://erlang.org/doc/reference_manual/distributed.html http://erlang.org/doc/reference_manual/distributed.html). So you're basically connecting to the runtime of the target node. The security model for ERTS isn't great, though, so your point stands. All you need is a matching secure cookie and you can connect and run any commands you want.

talknewswale 5y ago

डोनाल्ड ट्रम्प फेसबुक, इंस्टाग्राम को फिर से कब कर पाएंगे इस्तेमाल? https://talknewswale.com https://talknewswale.com

mstipetic 5y ago

These guys are amazingly productive, while also running a commercial company

fastball 5y ago
This is software that directly helps their business, so it's not really productivity outside of their business endeavors.
- xcambar 5y ago
  Try not to sound like you're diminishing the efforts and accomplishments of others. This is tremendous work, they're running their business and come up with generic/generalisable solutions instead of narrowing the outcome to their own benefits only.
  - fastball 5y ago
    When Facebook spends engineering effort to work on React, it's not out of the kindness of their heart, and that's ok. It's still productive and an accomplishment, it's just productive for a reason that pertains to their business. It is the same here. This tool helps the adoption of Elixir, running Elixir is their business, so yay! But the comment I was replying to implied this is somehow entirely separate from their business, which it's not. Not really sure how it sounded like I was diminishing the accomplishments of others. How would you have phrased the point?
    - mstipetic 5y ago
      FB has thousands of engineers. It's not that often that a small group of people can both keep the lights running financially and also develop fully featured massive libraries. I'd say that counts as a very productive company.
      - fastball 5y ago
        Right, but again my point is that developing this fully-featured library helps keep the lights on. The two things are not separate as you keep implying. Not disagreeing that they are productive and accomplishing a lot.

Pandabob 5y ago

Is it just me or is Elixir being talked/written about more in the past couple of months?

cpursley 5y ago
Years.
- cpursley 5y ago
  Interesting that I'm getting down voted, I'm long on Elixir and betting my startup on it.
  - Ankhers 5y ago
    I didn't down vote you, but my guess is that it was because you commented a single word. Also, because of the position of your comment, I had to scroll back up in order to find the context.
SatvikBeri 5y ago
There were 300 elixir stories in 2020, an average of roughly .8 per day. In 2021 there have been 89 elixir stories for 108 days, so about the same. (Source: hnsearch)
- bostonvaulter2 5y ago
  Is it possible to tell how many of those hit the front page? I think the majority of those 300 stories in 2020 did not hit the front page. I'm curious how many there were in years prior as well.
lawik 5y ago
It is growing and having good adoption. Not a huge language but it also has a fair share of novel features (being built on Erlang) that aren't commonly seen with other langs. So I think it might have a disproportionate visibility compared to size. Lots of work happening in the community, lots of ambitious ideas and releases.

out_of_protocol 5y ago

Readable `.livemd` format looks lovely (regular markdown without extra elements) http://lon-screenshots.s3.amazonaws.com/2021-04-18_Screenshot_20210418.jpg http://lon-screenshots.s3.amazonaws.com/2021-04-18_Screensho...

AlchemistCamp 5y ago

I'll admit that I was initially skeptical about Elixir for anything ML related when I first picked up the pragprog book on Elixir and Genetic Algorithms. Since that time, things have moved quite a bit. Between the Erlang VM JIT, then Nx, and now this, it's starting to look hopeful! I'm looking forward to digging into this.

CTmystery 5y ago

Awesome work! Towards the end, José hits on a couple points that Jupyter does not do well: - Minute 25: Saved notebook is readable (markdown) - Minute 23: Live collaborate on a single notebook - Minute 22: Visual indication that cell is stale after an upstream cell changes

srush 5y ago
I've been playing around with some of these ideas for Python notebooks. https://github.com/srush/streambook https://github.com/srush/streambook This is a proof of concept that combines Jupytext for markdown readable notebooks with Streamlit for in-order execution+caching. More difficult to get some of the more advanced features of Pluto / Livebook due to Python state management. Curious to hear any thoughts.
- O_H_E 5y ago
  Would you be able to expand or drop a few keywords on how (and what is) "python state management" prohibit few advance features. By in-order-execution you mean pluto's reactivity, right.

transfire 5y ago

All this cool Elixir news. I'm really having a hard time deciding between Elixir and Julia. (For programs that I must have compiled to a distributable executable, I've pretty much settled on Crystal. I find it more productive and pleasent than Go or Rust.)

thruflo22 5y ago
It’s not quite the same story but https://github.com/spawnfest/bakeware https://github.com/spawnfest/bakeware builds elixir releases into distributable executables.
- lobo_tuerto 5y ago
  That one is out of date, the one it forked of is not: https://github.com/bake-bake-bake/bakeware https://github.com/bake-bake-bake/bakeware
cultofmetatron 5y ago
if you want to build a saas, go with elixir. Its got a great batteries included web framework thats built to scale and an emerging suit of ml and linear algebra software coming out. If you want to experiment with ML and scientific programming where you do a lot of matrix processing, go with julia. thats literally what its made for.
- neolog 5y ago
  Julia has a web framework too of course. https://genieframework.com/ https://genieframework.com/
neolog 5y ago
https://github.com/JuliaLang/PackageCompiler.jl https://github.com/JuliaLang/PackageCompiler.jl
dnautics 5y ago
As someone who started on julia first and then moved to elixir: If you're doing computational/numerical stuff, stick with julia, for now. You really can't do that elixir, not today, and maybe not for at least another year or two. It will likely never be as good as julia since computation is not first class in the vm it runs on. If you're doing or planning or doing anything with web, or orchestration, go with elixir.

namelosw 5y ago

Wow, this is great. The fact is can connect to existing nodes also makes it can also be great for operation and administration - instead of building an awkward admin panel, you can just having a collection of scripts. The Elixir community is insanely productive. Also, the Elixir system is impressively pretty balanced - productivity-wise like Rails, concurrent-wise like Erlang, and continuously brings novel yet practical ideas to life in a robust manner!

cancan 5y ago

Love how prolific the elixir community and especially José is. Also, if you are into liveview/elixir, definitely dive into the codebase. There is a small operational-transform lib built in there too.

losvedir 5y ago

I'm pretty excited about this. I'm admit I was skeptical when I saw a similar LiveView based notebook a couple months ago, and wondered why not just make a kernel for Jupyter. But with the idea of real-time collaboration I get it. And with the momentum of Jose and team behind it, I think there's real promise here, especially once they figure out widgets and charts, which they have an open issue for. I program in Elixir at work, so I usually have an iex shell open, but lately I've been using a LiveBook and it's great. And I love the new Mix.install in Elixir 1.12 demonstrated here. Using a hex package in an adhoc way was a real shortcoming in Elixir before. Shows the power of having this thing developed by the language team.

filmor 5y ago
There are already Jupyter kernels, IElixir and ierl (which I wrote). The latter supports Erlang, Elixir and LFE right now and supports connecting to another node as well. https://github.com/filmor/ierl https://github.com/filmor/ierl

hades32 5y ago

Super nice! It's only missing auto-complete and it'll be light-years ahead of other tools on terms of usability

chrismccord 5y ago
There’s already a PR for it with a video of autocomplete in action :) https://github.com/elixir-nx/livebook/pull/208 https://github.com/elixir-nx/livebook/pull/208
- losvedir 5y ago
  I'm just gobsmacked at how fast they're moving here. This is incredible.

spicyramen 5y ago

If I'm a data scientists with a bunch of existing Python notebooks can I use this product?

astrowilson 5y ago
Not yet. It only supports Elixir for now (there is an issue for supporting other languages) and the notebook formats are different, so someone would also need to write a nbconvert to Markdown (.livemd).

ch4s3 5y ago

I'm not sure how I'm going to use this yet but I think there are going to be some great opportunities for prototyping here.

aleDsz 5y ago

I'm using this tool to teach the basics of Elixir on my livestream. I think there's a lot of good usecases for Livebook, not only machine learning and stuff like that.

rkss 5y ago

Would like to know how Livebook achieve collaborate-editing. Does it use algorithms like CRDT or OT ?

lawik 5y ago
OT. Using quill js on the frontend and a subset of that delta format handled on the backend. From conversation with Jose on Twitter.

cocco 5y ago

Using it everyday. Great tool.

cocco 5y ago

Using it everyfay. Greate tool.

arthurcolle 5y ago

How to use dependencies within a livebook? I just tried to use the enum_type (version 1.1.3) by just installing it inside the cloned livebook repo, but I'm getting the error that EnumType isn't loaded. Why is this?