7 ms·
Elixir Livebook is a secret weapon for documentation
- azophy_2 3y agothis is really interesting. I've never thought on using this or any jupyter-like tools to document working software. Maybe I'll try it later
- throwawaymaths 3y agoJupyter outputs to JSON. Livebook outputs to markdown, so it's more naturally renderable outside of the tool itself. There are still some rough edges -- it would be nice if livebook could cache prerendered content. for example, kino graphs only chuck a big fat JSON into a html comment, it would be nice if it also popped out an svg or encoded png with data uri.
- bongobingo1 3y agoI believe you can hover the corner of a graph and hit the (...) disclosure for a "save svg" button.
- throwawaymaths 3y agoWell it should be automatic and it should inline it.
- mfwgenerics 3y agoThe ability to execute sample code during documentation generation seems invaluable. Instead of being subject to rot, documentation turns into an executable test suite. I've been working on something like this for Kotlin using a compiler plugin that allows code to access the source text of lambdas, functions and classes being executed. You write code that spits out markdown and captures its own source into code blocks. https://github.com/mfwgenerics/kapshot https://github.com/mfwgenerics/kapshot
- martypitt 3y agoThat's fantastic. We use Kotlin to build Taxi (taxilang.org) and orbitalhq.com. this would be an awesome addition to our docs. However, do I understand it correctly that this allows me to output verified, working code in my test suite, but its not a tool that I can embed in my docs to create runnable code samples? (which is what the OP is showing). super valuable either way, just trying to validate understanding.
- mfwgenerics 3y agoThat sounds about right. The approach I'm working on won't allow you to embed runnable code in your docs. Instead, it requires you to write your documentation in Kotlin e.g. using a Markdown DSL where source code from the codebase can be included in the output. I'm working on a separate project to programmatically generate file trees and Docusaurus doc sites in code. I'm not aware of anything exactly like Livebook in the Kotlin ecosystem but you might be interested in kotlinx-knit which takers a more incremental approach to executable documentation: https://github.com/Kotlin/kotlinx-knit https://github.com/Kotlin/kotlinx-knit Neither of these approaches will result in interactive documentation.
- a6chris 3y agoI haven't used Livebook, but I think Kotlin Notebook is very similar - https://www.youtube.com/watch?v=2PLYlDJPelQ https://www.youtube.com/watch?v=2PLYlDJPelQ
- mcintyre1994 3y agoA related Elixir feature you might also find interesting is Doctests: https://elixir-lang.org/getting-started/mix-otp/docs-tests-and-with.html https://elixir-lang.org/getting-started/mix-otp/docs-tests-a... You can write inline examples in your docstrings and they’re automatically turned into unit tests. So again you get that documentation as an executable test suite not subject to rot, but at a per-function level. Same kind of idea but different levels of the documentation stack.
- impulser_ 3y agoI would suggest to anyone to download Livebook and test it out, even if you don't know Elixir or Erlang. You can do so much with it. I even use it for things outside of programming due to it being pretty much a markdown notebook that you can run code inside of. It's one of my favorite tools.
- troupo 3y agoTo do so much with it you need to know Elixir or Erlang ;)
- impulser_ 3y agoYou actually don't. You can use smart cells which lets you do a lot without actually coding. You can transform data, build charts, run AI model and more all without writing code. You might have to Google how to read a file in Elixir, but you don't actually need to know Elixir. In the future, you won't even need to know that to use it.
- troupo 3y agoAh. I never actually looked into smart cells. Mind blown.gif :)
- cpursley 3y agoFwiw, I’ve been using Elixir professionally for 5 or so years and even created a few open source libraries. Never once had to touch Erlang.
- throwawaymaths 3y agoOr, not and. Livebook just added Erlang cells.
- nesarkvechnep 3y agoErlang is very nice. I prefer it for writing libraries nowadays. It’s much easier to use Erlang from Elixir than the other way around.
- zogrodea 3y agoThis concept sounds quite a bit like the usage of .mdx files in OCaml. Those are markdown files which can load and execute OCaml code in them. They are useful for unit testing code like libraries, providing usage examples for end-users (as well as the text format markdown usually supports) and testing it at the same time. Definitely a great idea for documentation, and one which I should use more of. https://github.com/realworldocaml/mdx https://github.com/realworldocaml/mdx
- aayushmau5 3y agoApart from running code inside a "markdown" file, livebook can do much more. You have Smart cells to show charts, run sql queries against a db, run Neural Network tasks such as Image-To-Text generation using Bumblebee[1], etc. It is collaborative as well. [1] https://github.com/elixir-nx/bumblebee https://github.com/elixir-nx/bumblebee
- rahimnathwani 3y agoAfter reading your comment, I watched the first half of this video: https://youtu.be/4hVIxyHxwK8 https://youtu.be/4hVIxyHxwK8 Wow!
- rapnie 3y agoYes, more uptake of mdx in general. E.g. I use Astro static site generator and they offer it too. https://docs.astro.build/en/guides/markdown-content/ https://docs.astro.build/en/guides/markdown-content/
- theCodeStig 3y agoOrg-babel can do a lot of this, no?
- disgruntledphd2 3y agoYeah but to be fair, it's a bit gnarly to get set up and it can't do interactive graphs. Don't get me wrong, I love org babel but there are some advantages of the browser based approach.
- theCodeStig 3y agoYou’re right. I love emacs, though I must admit it’s quite a barrier to entry
- hitchstory 3y agoWhile I'm glad the idea of tying documentation to tests is catching on, I think markdown documentation -> create tests is the wrong way around. I built this tool to let people generate evergreen markdown documentation from annotated type-safe YAML integration tests: https://github.com/hitchdev/hitchstory https://github.com/hitchdev/hitchstory
- Terretta 3y agoThis is incredible work. To anyone curious, I highly recommend: - https://hitchdev.com/hitchstory/approach/ https://hitchdev.com/hitchstory/approach/ - https://hitchdev.com/hitchstory/why-not/ https://hitchdev.com/hitchstory/why-not/ From the overall RDD/BDD type home page: - https://hitchdev.com/hitchstory/ https://hitchdev.com/hitchstory/ The entire product site is a thing of richly informative beauty. --- My only question was whether the generated 'docs' snippets would add value over just reading the story in your DASL. Any markdown site generator (such as the chosen Material for MKDocs) can just embed the ```yaml anyway. But then I realized what was generating e.g. … - https://hitchdev.com/hitchstory/using/engine/rewrite-story/ https://hitchdev.com/hitchstory/using/engine/rewrite-story/ … and how superior that is to typical docs, especially typical docstring or swagger factories.
- hitchstory 3y agoThank you!
- YousefED 3y agoIf you're interested in this hybrid of docs and live coding, I'm building something similar for the Typescript / Javascript / React ecosystem @ https://www.typecell.org https://www.typecell.org (open source). See https://twitter.com/YousefED/status/1677342430545301504 https://twitter.com/YousefED/status/1677342430545301504 for a preview of the next version which will be more "wysiwyg" than the current notebook environment
- kartoshechka 3y agohopefully this will get us closer to a orgmode-ish confluence that can run any language, with CI docs test checks, and something more human than jiraql or whatever
- anonyfox 3y agoOne simple thing I need thats stopping me from jumping into Livebook fully: Salesforce API integration. Right now, I can't manage to make the auth work there (since oAuth needs some http redirect dance). Connecting to some proper SQL database is a breeze, but if I cannot access some silo'ed data that is absilute critical, its not that useful after all :-/
- josevalim 3y agoPlease open up a discussion on Livebook repo? We would love to understand more what is missing here. :)
- anonyfox 3y agoI slightly complain a single time for the last few years, and jose directly chimes in with kind words immediately. You rock, honestly! But on the topic: I think its a simple problem ultimately, the SF Api (https://developer.salesforce.com/docs/atlas.en-us.api_rest.meta/api_rest/intro_rest.htm https://developer.salesforce.com/docs/atlas.en-us.api_rest.m...) needs OAuth, which needs a redirect at some point, where I have no idea how to capture it from a livebook instance (easy in a phx app). This is a reference lib in JS ( https://jsforce.github.io/ https://jsforce.github.io/ ) and there seems to be no maintained alternative on hex.pm unfortunately. So, from scratch it seems, somehow. Additional insight: many big companies do use salesforce, and its internal tooling to produce reports/insights is rather limited, but there is a SQL-like language to do complicated stuff, but there is no way to execute a handwritten query in SF itself, except via API calls from outside. This... would be a perfect fit fro livebook! query SF + other databases to generate a report ad-hoc thats not easily possible right now without adding some heavyweight data warehouse connection!
- josevalim 3y agoI wonder if we could proxy OAuth flows to notebooks. For example, a notebook could initiate an OAuth2 flow, Livebook handles it, and once it gets back, we notify the notebook which may approve or deny the request. The biggest concern though is that we need to have a static callback URL, so you would need to run Livebook internally at a fixed address? If this is something you would like to work on, please reach out, I think it could open up interesting possibilities!
- thibaut_barrere 3y agoTo ensure you do not miss this: LiveBook comes with a Vega Lite integration (https://livebook.dev/integrations https://livebook.dev/integrations -> https://livebook.dev/integrations/vega-lite/ https://livebook.dev/integrations/vega-lite/), which means you get access to a lot of visualisations out of the box, should you need that (https://vega.github.io/vega-lite/ https://vega.github.io/vega-lite/). In the same "standing on giant's shoulders" stance, you can use Explorer (see example LiveBook at https://github.com/elixir-explorer/explorer/blob/main/notebooks/exploring_explorer.livemd https://github.com/elixir-explorer/explorer/blob/main/notebo...), which leverages Polars (https://www.pola.rs https://www.pola.rs), a very fast DataFrame library and now a company (https://www.pola.rs/posts/company-announcement/ https://www.pola.rs/posts/company-announcement/) with 4M$ seed.
- chx 3y agoOh I needed vega-lite badly in my life, thanks so much!
- bongobingo1 3y agoThe vega-lite wrapper is very slim too, basically just converts your slithery_snake keys into humpyCamels and corrals between different data sources, so if the docs on the elixir page seem sparse its because most of the "how do I build a graph like x" is just on the vega docs. You can easily dump it to json to share too, etc.
- thibaut_barrere 3y agoGood note :-) The wrapper can by the way be used outside of LiveBook (e.g. https://twitter.com/thibaut_barrere/status/1688533590965903360 https://twitter.com/thibaut_barrere/status/16885335909659033...) to provide charting in an Elixir app.
- throwawaymaths 3y agoIt's a shame because elixir documentation is gorgeous
- goeiedaggoeie 3y agoSorry for asking a dumb question, but how is this different from using Jupyter to write your interactive documentation/runbooks and interactive tests?
- Terretta 3y ago> how is this different from using Jupyter It requires you to run a far less common tech stack and limits your hiring to a vanishingly small subset of developers. Paradoxically, that may be a good thing.
- goeiedaggoeie 3y agoLove it! yes, it might be better. We have found paying above market average and being tighter on interviews has a similar outcome.
- throwawaymaths 3y agoElixir market is already more high paying than python.
- nesarkvechnep 3y agoAccording to SO’s survey, Erlang is the highest paying technology or something like that.
- voicedYoda 3y agoThis is just, wrong. There's actually a huge number of Elixirists, but after this long in the game, we're used to cynical elitists...
- throwawaymaths 3y agoOutputs markdown, easy to share. Jupyter outputs JSON.
- krstf13 3y ago
- bsaul 3y agoxcode swift playground are supposed to be the same kind of ideas. However, it's really one of the worst maintained piece of software in the whole toolsuite. Really a shame..
- jimmychoozyx 3y agoFor NodeJS enthusiasts, Check out Docusaurus (by Meta). For Markdown Docs, Markdown Blogs, and JSX-static pages - https://docusaurus.io https://docusaurus.io - https://docusaurus.io/docs/next/api/themes/@docusaurus/theme-mermaid https://docusaurus.io/docs/next/api/themes/@docusaurus/theme...
- Dowwie 3y agoThis is about live "documentation". Docusaurus doesn't have programmable cells linked to live processes or systems. Docusaurus is fantastic, but it's static. Livebooks are like next-generation REPLs with slick UI, security, reproducibility, etc.
- slorber 3y agoHey, Docusaurus maintainer here. I'm not sure to understand what you mean by "Docusaurus is static". Docusaurus builds static pages and allows you to plug JS/React code anywhere in your docs, so it's quite interactive and can run anything that can run in a browser, including REPLs. I read the Fly article but still don't really understand what Livebook is about. They say they use Livebook themselves, but the examples linked to only display a regular non-interactive doc to me. Do you have any production url showing me an experience that is possible in Livebook, and impossible in Docusaurus?
- Dowwie 3y agoSure, here's a simple demonstration of what livebook can do that docusaurus cannot, at least without maintainer expertise ;) Imagine the following. <here's a code block containing project metadata> we click a play button to start an elixir runtime based on that metadata code block <here's a "component" that securely connects to an elixir system running in a kubernetes pod in production, exposing an elixir REPL to interact with it> with this component, we then issue commands to inspect state, manage elixir "processes" in the runtime, etc
- slorber 3y ago
- Dowwie 3y agoToday, in a very round-about way, one could use a livebook as if it were a blog post: download-install Livebook on your machive, download the blog post livemd file, run it locally. Here's an example. Note in the first paragraph a link to the post as a livemd file: https://genericjam.com/blog/image-processing https://genericjam.com/blog/image-processing If you go through the steps, you'll notice that the livemd isn't stylized as a blog post. This is where the next opportunity lies: creating beautiful blog posts as livebooks, and without the need to install and run a livebook server locally.
- StevenNunez 3y agoIs there a way to render the livebook as read only until they try a change? I tried to open the link and it failed. I could download it and run it but I just want to see it as you intended.
- Dowwie 3y agoThere's a link on that render page to download the livebook file and run locally on your own livebook server
- slorber 3y agoAfter reading this article, I'm still not sure to understand what Livebook is. Can someone show me a real production url of what is possible to achieve in Livebook and impossible/difficult to achieve with other tools? I'm the Docusaurus maintainer, and making your docs interactive, and giving the ability to run the documented project inside its doc does not feel like something new.
- clessg 3y agoOne small note: It looks like the 'Mix standalone' runtime option was removed late last year. Instead, you can use `Mix.install/2` pointing to your project directory: Mix.install( [ {:my_app, path: Path.join(__DIR__, ".."), env: :dev} ], config_path: :my_app, lockfile: :my_app ) (The above snippet assumes a notebook located within a `notebooks/` directory, as per the article.) Alternatively, you can connect to a running instance, e.g. a Phoenix project: $ elixir --sname my-app --cookie cookie -S mix phx.server $ LIVEBOOK_DEFAULT_RUNTIME=attached:my-app:cookie livebook server