7 ms·

Rustler – Safe Elixir and Erlang NIFs in Rust

201 points by hansihe 10y ago

moosingin3space 10y ago
This, IMO, is quite possibly the best instance of "best tool for the job" I've ever seen. Rust is great for writing high-performance code, and Erlang is fantastic for distributed systems.
- jbhatab 10y ago
  code like this gives me polygot chills. Rust + Go = love :DDDD.
kibwen 10y ago
I've been very excited at Rust's prospects as an alternative to C for extending all sorts of higher-level languages, but I'm most excited at the prospect of combining Rust with Erlang given the latter's focus on reliability. I'd love to figure out where Erlang programmers hang out to ask them what they think about using Rust for NIFs.
- dozzie 10y ago
  #erlang on Freenode.
- im_down_w_otp 10y ago
  Mostly the Erlang mailing list. http://erlang.org/mailman/listinfo http://erlang.org/mailman/listinfo As an Erlanger, and maker of a NIF -> Rust library I wasn't allowed to open source (sadly), I can tell you I think this is excellent. My current science project is trying to rebuild some small components of BEAM and the Erlang Runtime System in Rust that are currently in C. Mostly the above is an exercise in understanding the internals of BEAM better, because my ideal end state is to figure out what it would take to make an Erlang/OTP-like library for Rust (with task/thread/process/whatever preemption)... which is actually what originally drew me to Rust 3-4 years ago back when it was shaped more like a systems-Erlang (task supervision hierarchies, etc) and less like a renovated-safe-C/C++.
  - kibwen 10y ago
    > As an Erlanger, and maker of a NIF -> Rust library I wasn't allowed to open source (sadly) Presuming that this happened on company time, I'm curious to know if there's a company using Erlang+Rust in production or if this was just a fun side project. :)
    - im_down_w_otp 10y ago
      You are correct on the first count. On the latter... I've always been a fan of Twitter's mutual opt-in requirement for DM chatting. You're being followed. Which feels really weird and funny to write. But it'll be reasonably obvious.
      - deleted 10y ago
        [deleted]
  - tormeh 10y ago
    >Rust 3-4 years ago back when it was shaped more like a systems-Erlang (task supervision hierarchies, etc) and less like a renovated-safe-C/C++ Wow, any chance Erlang-like supervision and actor model in general might make a comeback in Rust? I'm sure the Erlang guys has thought about this, but no static type checking makes me a bit nervous. Erlang can be surprisingly strict (a good thing) in some ways, but as far as I can tell, you only discover any failures at runtime.
    - yellowapple 10y ago
      One could probably discover failures at compile-time with Erlang/Elixir as well, especially if one is diligent with typespecs and/or pattern matching / destructuring. For example, if I have the following Elixir function definitions: def foo({:bar, bell}), do: IO.puts("got a barbell: #{bell}") def foo({:baz, bell}), do: IO.puts("got a bazbell: #{bell}") Subsequently calling `foo({:bat, "some value for bell"})` should be detectable by the compiler and thus generate at least a warning. We can luckily catch it relatively easily and painlessly at runtime with Erlang's normal insistence on process supervision trees, but it's still a crash that could be easily avoided without having to jump into the realm of full-blown "defensive" programming.
- yellowapple 10y ago
  I'm using Rust in exactly that capacity (along with ports). In fact, if it weren't for Rust, I'd probably steer clear from writing my own NIFs for the sake of my own sanity; coming from the Perl 5 world, where XS is king, I'm not exactly keen on splaying open Erlang's C API unless I have a language with which to do so without slicing off my own leg. (Ports, on the other hand, are inherently safer due to being isolated from the Erlang VM, such that even if they crash they don't bring down the whole VM. They also communicate over STDIO, meaning that I'm free to use them to interface with components written in languages with less-than-pleasant methods of achieving C ABI compatibility, like Perl 5 and Ruby when not augmented with extra packages.) On that note, Rust also has some wonderful implications for the Perl 5 world in terms of providing some semblance of safety around native Perl extensions. Same deal for Perl 6 (though its FFI mechanisms are infinitely nicer than Perl 5's).
fabian2k 10y ago
NIFs are the fastest method to call external code from Erlang/Elixir, as far as I understand. But I wonder how high the actual overhead is. My understanding is that if I still want to ensure that the Beam VM can continue to schedule all processes efficiently, these NIFs shouldn't calculate forever, but return quickly to avoid blocking all other processes. So the straightforward way to ensure this for longer calculations would be to split the calculation in smaller, parallelizable jobs, if that is possible. But then the overhead of calling NIFs might actually matter, if you split them into chunks that are too small. I've no idea how big the overhead actually is, I'd be interested in a rough estimate. Is it small enough that I can just ignore it entirely?
- im_down_w_otp 10y ago
  There's very little overhead in calling C (or C ABI shaped things) from Erlang. The reason why NIFs can trivially bring down the whole Erlang VM is because those libraries are linked straight into the same process space as the runtime itself, so in effect calling those NIF functions is roughly the same as calling internal VM functions that are written in C too. There's some overhead in type-casting/data-type-interpretation, but not much. The internal representations of the core data-types in Erlang are mostly already represented in C in the runtime. There's some overhead in memory copies across that boundary too unless you make your data opaque to Erlang and always manipulate it via the NIF in which case you can get away with using NIF "resources" and more or less just pass a pointer and environment back and forth.
  - kibwen 10y ago
    Appropriate username. :)
- jerf 10y ago
  If you look at what NIFs exist up to this point and where Erlang itself tends to use that mechanism, they are all for very tiny uses, like type checks and such. But that's probably precisely because they're too scary to use for anything very large. The existence of a provably-safe and practical mechanism for writing them may cause and/or require some changes in the ecosystem to account for the fact it's now possible to write NIFs that may take a long time to run. I wouldn't be surprised that at least one of this library or the Erlang VM itself develop an official way to easily use a NIF to run a longer-running process safely. There hasn't been a need up to this point because there hasn't been a such thing as a long-running NIF. At the moment, if you're looking at something that may take several seconds you're probably still better off coordinating something over a port to an external process or something. Or working with this project to make it feasible to run a long-running native process. Trying to write a NIF that could somehow yield back and then be "continued" later would be a royal pain. Cooperative scheduling was enough of a pain when we were in the hundreds of processes mostly doing nothing on OSes; trying to use it on an Erlang server is probably just infeasible.
Anderkent 10y ago
> This means you can write and run totally safe code in Rust, no worrying about segfaults. This is not quite accurate; you can still segfault rust writing 100% safe code, for example if you have a large stack overflow ever since __morestack was killed. Though these cases are fortunately rare
- hansihe 10y ago
  That's very true. Do you think I should have written it differently? I might want to add a clause in the caveats section at the very least.
  - Anderkent 10y ago
    Nah, just nitpicking. Though I guess the point is that AFAIK there's no documentation for all the ways that safe rust code is allowed to segfault; the distinction between a in-rust panic is that a segfault would probably crash beam, whereas an in-rust panic might be recoverable?
    - kibwen 10y ago
      > there's no documentation for all the ways that safe rust code is allowed to segfault I wouldn't say "allowed" to segfault. :P The behavior you're referring to is currently due to a deficiency in LLVM for non-Windows platforms. Here's the bug on the Rust repo tracking it: https://github.com/rust-lang/rust/issues/16012 https://github.com/rust-lang/rust/issues/16012 Its resolution is long-awaited, to say the least, but it requires someone proficient with LLVM to do the legwork... > an in-rust panic might be recoverable? Anytime you're writing an interface where code is going to be calling into Rust via C FFI, you ought to be using https://doc.rust-lang.org/std/panic/fn.catch_unwind.html https://doc.rust-lang.org/std/panic/fn.catch_unwind.html , which is specifically intended to prevent panics from crossing FFI boundaries.
      - iopq 10y ago
        > This function only catches unwinding panics, not those that abort the process. How do you expect a stack overflow to unwind? I have difficulty imagining the implementation
andy_ppp 10y ago
Really nice to see this linked here, I was chatting with hansihe about getting the html5ever parser from servo into a nif and he magically created it here: https://github.com/hansihe/html5ever_elixir https://github.com/hansihe/html5ever_elixir It shows how create a threadpool and avoid the 1ms maximum nif execution time too.

201 points by hansihe 10y ago

This, IMO, is quite possibly the best instance of "best tool for the job" I've ever seen. Rust is great for writing high-performance code, and Erlang is fantastic for distributed systems.

jbhatab 10y ago
code like this gives me polygot chills. Rust + Go = love :DDDD.

kibwen 10y ago

I've been very excited at Rust's prospects as an alternative to C for extending all sorts of higher-level languages, but I'm most excited at the prospect of combining Rust with Erlang given the latter's focus on reliability. I'd love to figure out where Erlang programmers hang out to ask them what they think about using Rust for NIFs.

dozzie 10y ago
#erlang on Freenode.
im_down_w_otp 10y ago
Mostly the Erlang mailing list. http://erlang.org/mailman/listinfo http://erlang.org/mailman/listinfo As an Erlanger, and maker of a NIF -> Rust library I wasn't allowed to open source (sadly), I can tell you I think this is excellent. My current science project is trying to rebuild some small components of BEAM and the Erlang Runtime System in Rust that are currently in C. Mostly the above is an exercise in understanding the internals of BEAM better, because my ideal end state is to figure out what it would take to make an Erlang/OTP-like library for Rust (with task/thread/process/whatever preemption)... which is actually what originally drew me to Rust 3-4 years ago back when it was shaped more like a systems-Erlang (task supervision hierarchies, etc) and less like a renovated-safe-C/C++.
- kibwen 10y ago
  > As an Erlanger, and maker of a NIF -> Rust library I wasn't allowed to open source (sadly) Presuming that this happened on company time, I'm curious to know if there's a company using Erlang+Rust in production or if this was just a fun side project. :)
  - im_down_w_otp 10y ago
    You are correct on the first count. On the latter... I've always been a fan of Twitter's mutual opt-in requirement for DM chatting. You're being followed. Which feels really weird and funny to write. But it'll be reasonably obvious.
    - deleted 10y ago
      [deleted]
- tormeh 10y ago
  >Rust 3-4 years ago back when it was shaped more like a systems-Erlang (task supervision hierarchies, etc) and less like a renovated-safe-C/C++ Wow, any chance Erlang-like supervision and actor model in general might make a comeback in Rust? I'm sure the Erlang guys has thought about this, but no static type checking makes me a bit nervous. Erlang can be surprisingly strict (a good thing) in some ways, but as far as I can tell, you only discover any failures at runtime.
  - yellowapple 10y ago
    One could probably discover failures at compile-time with Erlang/Elixir as well, especially if one is diligent with typespecs and/or pattern matching / destructuring. For example, if I have the following Elixir function definitions: def foo({:bar, bell}), do: IO.puts("got a barbell: #{bell}") def foo({:baz, bell}), do: IO.puts("got a bazbell: #{bell}") Subsequently calling `foo({:bat, "some value for bell"})` should be detectable by the compiler and thus generate at least a warning. We can luckily catch it relatively easily and painlessly at runtime with Erlang's normal insistence on process supervision trees, but it's still a crash that could be easily avoided without having to jump into the realm of full-blown "defensive" programming.
yellowapple 10y ago
I'm using Rust in exactly that capacity (along with ports). In fact, if it weren't for Rust, I'd probably steer clear from writing my own NIFs for the sake of my own sanity; coming from the Perl 5 world, where XS is king, I'm not exactly keen on splaying open Erlang's C API unless I have a language with which to do so without slicing off my own leg. (Ports, on the other hand, are inherently safer due to being isolated from the Erlang VM, such that even if they crash they don't bring down the whole VM. They also communicate over STDIO, meaning that I'm free to use them to interface with components written in languages with less-than-pleasant methods of achieving C ABI compatibility, like Perl 5 and Ruby when not augmented with extra packages.) On that note, Rust also has some wonderful implications for the Perl 5 world in terms of providing some semblance of safety around native Perl extensions. Same deal for Perl 6 (though its FFI mechanisms are infinitely nicer than Perl 5's).

fabian2k 10y ago

NIFs are the fastest method to call external code from Erlang/Elixir, as far as I understand. But I wonder how high the actual overhead is. My understanding is that if I still want to ensure that the Beam VM can continue to schedule all processes efficiently, these NIFs shouldn't calculate forever, but return quickly to avoid blocking all other processes. So the straightforward way to ensure this for longer calculations would be to split the calculation in smaller, parallelizable jobs, if that is possible. But then the overhead of calling NIFs might actually matter, if you split them into chunks that are too small. I've no idea how big the overhead actually is, I'd be interested in a rough estimate. Is it small enough that I can just ignore it entirely?

im_down_w_otp 10y ago
There's very little overhead in calling C (or C ABI shaped things) from Erlang. The reason why NIFs can trivially bring down the whole Erlang VM is because those libraries are linked straight into the same process space as the runtime itself, so in effect calling those NIF functions is roughly the same as calling internal VM functions that are written in C too. There's some overhead in type-casting/data-type-interpretation, but not much. The internal representations of the core data-types in Erlang are mostly already represented in C in the runtime. There's some overhead in memory copies across that boundary too unless you make your data opaque to Erlang and always manipulate it via the NIF in which case you can get away with using NIF "resources" and more or less just pass a pointer and environment back and forth.
- kibwen 10y ago
  Appropriate username. :)
jerf 10y ago
If you look at what NIFs exist up to this point and where Erlang itself tends to use that mechanism, they are all for very tiny uses, like type checks and such. But that's probably precisely because they're too scary to use for anything very large. The existence of a provably-safe and practical mechanism for writing them may cause and/or require some changes in the ecosystem to account for the fact it's now possible to write NIFs that may take a long time to run. I wouldn't be surprised that at least one of this library or the Erlang VM itself develop an official way to easily use a NIF to run a longer-running process safely. There hasn't been a need up to this point because there hasn't been a such thing as a long-running NIF. At the moment, if you're looking at something that may take several seconds you're probably still better off coordinating something over a port to an external process or something. Or working with this project to make it feasible to run a long-running native process. Trying to write a NIF that could somehow yield back and then be "continued" later would be a royal pain. Cooperative scheduling was enough of a pain when we were in the hundreds of processes mostly doing nothing on OSes; trying to use it on an Erlang server is probably just infeasible.

Anderkent 10y ago

> This means you can write and run totally safe code in Rust, no worrying about segfaults. This is not quite accurate; you can still segfault rust writing 100% safe code, for example if you have a large stack overflow ever since __morestack was killed. Though these cases are fortunately rare

hansihe 10y ago
That's very true. Do you think I should have written it differently? I might want to add a clause in the caveats section at the very least.
- Anderkent 10y ago
  Nah, just nitpicking. Though I guess the point is that AFAIK there's no documentation for all the ways that safe rust code is allowed to segfault; the distinction between a in-rust panic is that a segfault would probably crash beam, whereas an in-rust panic might be recoverable?
  - kibwen 10y ago
    > there's no documentation for all the ways that safe rust code is allowed to segfault I wouldn't say "allowed" to segfault. :P The behavior you're referring to is currently due to a deficiency in LLVM for non-Windows platforms. Here's the bug on the Rust repo tracking it: https://github.com/rust-lang/rust/issues/16012 https://github.com/rust-lang/rust/issues/16012 Its resolution is long-awaited, to say the least, but it requires someone proficient with LLVM to do the legwork... > an in-rust panic might be recoverable? Anytime you're writing an interface where code is going to be calling into Rust via C FFI, you ought to be using https://doc.rust-lang.org/std/panic/fn.catch_unwind.html https://doc.rust-lang.org/std/panic/fn.catch_unwind.html , which is specifically intended to prevent panics from crossing FFI boundaries.
    - iopq 10y ago
      > This function only catches unwinding panics, not those that abort the process. How do you expect a stack overflow to unwind? I have difficulty imagining the implementation

andy_ppp 10y ago

Really nice to see this linked here, I was chatting with hansihe about getting the html5ever parser from servo into a nif and he magically created it here: https://github.com/hansihe/html5ever_elixir https://github.com/hansihe/html5ever_elixir It shows how create a threadpool and avoid the 1ms maximum nif execution time too.