6 ms·

Evcxr: A Rust REPL and Jupyter Kernel

170 points by batterylow 6y ago

virtualritz 6y ago
I added jupyter notebook support to my nsi[1] crate via evcxr. It allows you to do a 3D (offline) render with 3Delight (on AWS, if you want) – directly into a notebook. Here is an example notebook[2]. I haven't managed to update the image dynamically while it's rendering (there's no support for that in evcxr). If anyone knows how to do this, I'd be keen to add that too. [1] https://crates.io/crates/nsi https://crates.io/crates/nsi [2] https://github.com/virtualritz/nsi/blob/master/examples/jupyter/jupyter.ipynb https://github.com/virtualritz/nsi/blob/master/examples/jupy...
- sitkack 6y ago
  Are you talking about streaming data from a thread on the backend to say a canvas or image element in the browser? If you streaming in blocks on the server side and compositing them into a final render, in this case there are a couple hacks. 1) just use a timer and reload the image every x seconds 2) do a hanging read on and endpoint, return updated image. This has the nice quality of the server being able to decide the framerate.
  - virtualritz 6y ago
    Yes, I am streaming buckets/blocks. That would be both on the evcxr side, right? I was wondering if sending it as an MNG with loop=1 would work.
    - sitkack 6y ago
      Rich display handles sending html/svg/etc back to the UI layer. https://ipython.readthedocs.io/en/stable/config/integrating.html#rich-display https://ipython.readthedocs.io/en/stable/config/integrating.... You might be able to do everything in there with a little JS.
rsstack 6y ago
I have been using evcxr for the last 2-3 weeks and I'm loving it. There are two things that bother me: (a) fancy UI components seem to only work with Python and C++ kernel (not theoretically, but with available packages), (b) while you can redefine values and functions when you're iterating, you cannot do so with structs (yes, big structs go to a local package that I depend on, but sometimes I have ad-hoc utility structs).
- westurner 6y ago
  Here's the xeus-cling (Jupyter C++ Kernel) source: https://github.com/jupyter-xeus/xeus-cling/tree/master/src https://github.com/jupyter-xeus/xeus-cling/tree/master/src Do any of the other non-Python Jupyter kernels have examples of working fancy UI components? https://github.com/jupyter/jupyter/wiki/Jupyter-kernels https://github.com/jupyter/jupyter/wiki/Jupyter-kernels Jupyter kernels implement the Jupyter kernel message spec. Introspection, Completion: https://jupyter-client.readthedocs.io/en/latest/messaging.html#introspection https://jupyter-client.readthedocs.io/en/latest/messaging.ht... Debugging (w/ DAP: Debug Adapter Protocol) https://jupyter-client.readthedocs.io/en/latest/messaging.html#debug-request https://jupyter-client.readthedocs.io/en/latest/messaging.ht... A `display_data` Jupyter kernel message includes a `data` key with a dict value: "The data dict contains key/value pairs, where the keys are MIME types and the values are the raw data of the representation in that format." https://jupyter-client.readthedocs.io/en/latest/messaging.html#display-data https://jupyter-client.readthedocs.io/en/latest/messaging.ht... This looks like it does something with MIME bundles: https://github.com/jupyter-xeus/xeus-cling/blob/00b1fa69d17b390b7d3b470a9859281350863d30/src/xmime_internal.hpp#L173 https://github.com/jupyter-xeus/xeus-cling/blob/00b1fa69d17b... ipython.display: https://github.com/ipython/ipython/blob/master/IPython/display.py https://github.com/ipython/ipython/blob/master/IPython/displ... ipython.core.display: https://github.com/ipython/ipython/blob/master/IPython/core/display.py https://github.com/ipython/ipython/blob/master/IPython/core/... ipython.lib.display: https://github.com/ipython/ipython/blob/master/IPython/lib/display.py https://github.com/ipython/ipython/blob/master/IPython/lib/d... You can also run Jupyter kernels in a shell with jupyter/jupyter_console: pip install jupyter-console jupyter-client jupyter kernelspec list jupyter console --kernel python3
minimaxir 6y ago
An example Rust Jupyter Notebook for data visualization using this tool: https://plotters-rs.github.io/plotters-doc-data/evcxr-jupyter-integration.html https://plotters-rs.github.io/plotters-doc-data/evcxr-jupyte... Rust hold interesting promise for data analysis and visualization at scale, especially as someone who hits use cases where Python/R start to bottleneck. The ecosystem is not there yet, though.
- ezluckyfree 6y ago
  not to sound like a broken record, but Julia has the performance while also having dynamic memory management and an ecosystem close to Python/R (depending on the area). Rust is nice when you want something lightweight and fast, since Julia has a pretty heavyweight runtime.
  - 127 6y ago
    Julia is cool, but can I have a version where arrays start at zero?
    - sc11 6y ago
      No need for another version, Julia supports custom indices by default. Check out https://docs.julialang.org/en/v1/devdocs/offset-arrays/ https://docs.julialang.org/en/v1/devdocs/offset-arrays/ and https://github.com/JuliaArrays/OffsetArrays.jl https://github.com/JuliaArrays/OffsetArrays.jl
  - DougBTX 6y ago
    Yes, Julia looks very impressive, a language I'd like to try at some point.
  - deleted 6y ago
    [deleted]
  - minimaxir 6y ago
    "Lightweight and fast" are the key features I'm looking for.
qchris 6y ago
If anyone's interested in a collection of scientific programming/machine learning libraries written in Rust, we maintain a collection of them across different domains at https://www.arewelearningyet.com/ https://www.arewelearningyet.com/.
batterylow 6y ago
First and third party materials to get up and running with Evcxr here https://github.com/google/evcxr/blob/master/evcxr_jupyter/README.md https://github.com/google/evcxr/blob/master/evcxr_jupyter/RE...
shahinrostami 6y ago
I enjoy literate programming so Evcxr has made learning Rust a familiar experience. The data science ecosystem isn’t quite there yet, but you can still do some things easily if you want to. If you’re interested, you may want to check out my book on the topic https://datacrayon.com/shop/product/data-analysis-with-rust-notebooks/ https://datacrayon.com/shop/product/data-analysis-with-rust-...
tveita 6y ago
There's an explanation of how it works: https://github.com/google/evcxr/blob/master/evcxr/HOW_IT_WORKS.md https://github.com/google/evcxr/blob/master/evcxr/HOW_IT_WOR... That's an impressive amount of hacks under the hood! I thought maybe there would be an interpreter involved, but it really does invoke rustc and then loads the code with dynamic linking.
metreo 6y ago
Despite my aversion to these 'live-coding' environments for compiled languages, I'm actually interested to see what can be managed with Rust. The dynamic linking they are using sort of reminds me of rust-analyzer which quickly re-parses it's tree at every save. I don't code in a way that normally requires interpretative feedback but quality tools like evcxr might contribute to changing my habits.
perrohunter 6y ago
wow, this looks so cool, nice job!
xixixao 6y ago
This is amazing! But I still posit there are better UI paradigms than Jupyter (and REPL). Working in Jupyter is like having one hand tied behind my back.
- sieste 6y ago
  I'm interested in what better/ other UI paradigms there are. Could you elaborate?
  - ploika 6y ago
    Not OP, but I like to avoid Jupyter where I can too. I like using a REPL though. For very interactive, experimental work I like having code in one pane and a REPL in another separate one - so that you can highlight and send snippets of a file to the REPL with a keystroke or two, and the code and results aren't mixed together. RStudio or Vim Slime[1] would be my favourite examples of this kind of approach. For more "static" literate programming, I prefer the way RMarkdown and Sweave work to Jupyter Notebooks (though I don't even use R anymore). Both have their strengths and weaknesses of course. [1] https://github.com/jpalardy/vim-slime https://github.com/jpalardy/vim-slime
cies 6y ago
I really loved the Rails console for developing web apps. Write a few lines on the REPL, then past them in an app/model file. Knowing Rust (and other strong typed langs) I understand that it is not so easy to implement this workflow for these languages. But looking at this project I kind of have new hopes. With the strong languages it should even be possible to have intellisense-like completion/suggestion available on the REPL! Would it be, eventually, possible use Evcxr for this purpose?
_hl_ 6y ago
I used this about a year ago for a small astrophysics simulation+plotting etc, together with futhark for some GPU code. It wasn't perfect (some crashes and weird behaviour), but worked remarkably well. I imagine it's even better today. It was really useful to have easy acces to the simulation code and datastructures from the notebook where I made the plots, and use the same language for both. Glad to see the project still going strong.

170 points by batterylow 6y ago

I added jupyter notebook support to my nsi[1] crate via evcxr. It allows you to do a 3D (offline) render with 3Delight (on AWS, if you want) – directly into a notebook. Here is an example notebook[2]. I haven't managed to update the image dynamically while it's rendering (there's no support for that in evcxr). If anyone knows how to do this, I'd be keen to add that too. [1] https://crates.io/crates/nsi https://crates.io/crates/nsi [2] https://github.com/virtualritz/nsi/blob/master/examples/jupyter/jupyter.ipynb https://github.com/virtualritz/nsi/blob/master/examples/jupy...

sitkack 6y ago
Are you talking about streaming data from a thread on the backend to say a canvas or image element in the browser? If you streaming in blocks on the server side and compositing them into a final render, in this case there are a couple hacks. 1) just use a timer and reload the image every x seconds 2) do a hanging read on and endpoint, return updated image. This has the nice quality of the server being able to decide the framerate.
- virtualritz 6y ago
  Yes, I am streaming buckets/blocks. That would be both on the evcxr side, right? I was wondering if sending it as an MNG with loop=1 would work.
  - sitkack 6y ago
    Rich display handles sending html/svg/etc back to the UI layer. https://ipython.readthedocs.io/en/stable/config/integrating.html#rich-display https://ipython.readthedocs.io/en/stable/config/integrating.... You might be able to do everything in there with a little JS.

rsstack 6y ago

I have been using evcxr for the last 2-3 weeks and I'm loving it. There are two things that bother me: (a) fancy UI components seem to only work with Python and C++ kernel (not theoretically, but with available packages), (b) while you can redefine values and functions when you're iterating, you cannot do so with structs (yes, big structs go to a local package that I depend on, but sometimes I have ad-hoc utility structs).

westurner 6y ago
Here's the xeus-cling (Jupyter C++ Kernel) source: https://github.com/jupyter-xeus/xeus-cling/tree/master/src https://github.com/jupyter-xeus/xeus-cling/tree/master/src Do any of the other non-Python Jupyter kernels have examples of working fancy UI components? https://github.com/jupyter/jupyter/wiki/Jupyter-kernels https://github.com/jupyter/jupyter/wiki/Jupyter-kernels Jupyter kernels implement the Jupyter kernel message spec. Introspection, Completion: https://jupyter-client.readthedocs.io/en/latest/messaging.html#introspection https://jupyter-client.readthedocs.io/en/latest/messaging.ht... Debugging (w/ DAP: Debug Adapter Protocol) https://jupyter-client.readthedocs.io/en/latest/messaging.html#debug-request https://jupyter-client.readthedocs.io/en/latest/messaging.ht... A `display_data` Jupyter kernel message includes a `data` key with a dict value: "The data dict contains key/value pairs, where the keys are MIME types and the values are the raw data of the representation in that format." https://jupyter-client.readthedocs.io/en/latest/messaging.html#display-data https://jupyter-client.readthedocs.io/en/latest/messaging.ht... This looks like it does something with MIME bundles: https://github.com/jupyter-xeus/xeus-cling/blob/00b1fa69d17b390b7d3b470a9859281350863d30/src/xmime_internal.hpp#L173 https://github.com/jupyter-xeus/xeus-cling/blob/00b1fa69d17b... ipython.display: https://github.com/ipython/ipython/blob/master/IPython/display.py https://github.com/ipython/ipython/blob/master/IPython/displ... ipython.core.display: https://github.com/ipython/ipython/blob/master/IPython/core/display.py https://github.com/ipython/ipython/blob/master/IPython/core/... ipython.lib.display: https://github.com/ipython/ipython/blob/master/IPython/lib/display.py https://github.com/ipython/ipython/blob/master/IPython/lib/d... You can also run Jupyter kernels in a shell with jupyter/jupyter_console: pip install jupyter-console jupyter-client jupyter kernelspec list jupyter console --kernel python3

minimaxir 6y ago

An example Rust Jupyter Notebook for data visualization using this tool: https://plotters-rs.github.io/plotters-doc-data/evcxr-jupyter-integration.html https://plotters-rs.github.io/plotters-doc-data/evcxr-jupyte... Rust hold interesting promise for data analysis and visualization at scale, especially as someone who hits use cases where Python/R start to bottleneck. The ecosystem is not there yet, though.

ezluckyfree 6y ago
not to sound like a broken record, but Julia has the performance while also having dynamic memory management and an ecosystem close to Python/R (depending on the area). Rust is nice when you want something lightweight and fast, since Julia has a pretty heavyweight runtime.
- 127 6y ago
  Julia is cool, but can I have a version where arrays start at zero?
  - sc11 6y ago
    No need for another version, Julia supports custom indices by default. Check out https://docs.julialang.org/en/v1/devdocs/offset-arrays/ https://docs.julialang.org/en/v1/devdocs/offset-arrays/ and https://github.com/JuliaArrays/OffsetArrays.jl https://github.com/JuliaArrays/OffsetArrays.jl
- DougBTX 6y ago
  Yes, Julia looks very impressive, a language I'd like to try at some point.
- deleted 6y ago
  [deleted]
- minimaxir 6y ago
  "Lightweight and fast" are the key features I'm looking for.

qchris 6y ago

If anyone's interested in a collection of scientific programming/machine learning libraries written in Rust, we maintain a collection of them across different domains at https://www.arewelearningyet.com/ https://www.arewelearningyet.com/.

batterylow 6y ago

First and third party materials to get up and running with Evcxr here https://github.com/google/evcxr/blob/master/evcxr_jupyter/README.md https://github.com/google/evcxr/blob/master/evcxr_jupyter/RE...

shahinrostami 6y ago

I enjoy literate programming so Evcxr has made learning Rust a familiar experience. The data science ecosystem isn’t quite there yet, but you can still do some things easily if you want to. If you’re interested, you may want to check out my book on the topic https://datacrayon.com/shop/product/data-analysis-with-rust-notebooks/ https://datacrayon.com/shop/product/data-analysis-with-rust-...

tveita 6y ago

There's an explanation of how it works: https://github.com/google/evcxr/blob/master/evcxr/HOW_IT_WORKS.md https://github.com/google/evcxr/blob/master/evcxr/HOW_IT_WOR... That's an impressive amount of hacks under the hood! I thought maybe there would be an interpreter involved, but it really does invoke rustc and then loads the code with dynamic linking.

metreo 6y ago

Despite my aversion to these 'live-coding' environments for compiled languages, I'm actually interested to see what can be managed with Rust. The dynamic linking they are using sort of reminds me of rust-analyzer which quickly re-parses it's tree at every save. I don't code in a way that normally requires interpretative feedback but quality tools like evcxr might contribute to changing my habits.

perrohunter 6y ago

wow, this looks so cool, nice job!

xixixao 6y ago

This is amazing! But I still posit there are better UI paradigms than Jupyter (and REPL). Working in Jupyter is like having one hand tied behind my back.

sieste 6y ago
I'm interested in what better/ other UI paradigms there are. Could you elaborate?
- ploika 6y ago
  Not OP, but I like to avoid Jupyter where I can too. I like using a REPL though. For very interactive, experimental work I like having code in one pane and a REPL in another separate one - so that you can highlight and send snippets of a file to the REPL with a keystroke or two, and the code and results aren't mixed together. RStudio or Vim Slime[1] would be my favourite examples of this kind of approach. For more "static" literate programming, I prefer the way RMarkdown and Sweave work to Jupyter Notebooks (though I don't even use R anymore). Both have their strengths and weaknesses of course. [1] https://github.com/jpalardy/vim-slime https://github.com/jpalardy/vim-slime

cies 6y ago

I really loved the Rails console for developing web apps. Write a few lines on the REPL, then past them in an app/model file. Knowing Rust (and other strong typed langs) I understand that it is not so easy to implement this workflow for these languages. But looking at this project I kind of have new hopes. With the strong languages it should even be possible to have intellisense-like completion/suggestion available on the REPL! Would it be, eventually, possible use Evcxr for this purpose?

_hl_ 6y ago

I used this about a year ago for a small astrophysics simulation+plotting etc, together with futhark for some GPU code. It wasn't perfect (some crashes and weird behaviour), but worked remarkably well. I imagine it's even better today. It was really useful to have easy acces to the simulation code and datastructures from the notebook where I made the plots, and use the same language for both. Glad to see the project still going strong.