6 ms·
Rustler is a library for writing Erlang NIFs in safe Rust code
- davidw 7y agofn add<'a>(env: Env<'a>, args: &[Term<'a>]) -> Result<Term<'a>, Error> { As someone not familiar with Rust, I'm curious what all the symbols there mean. What it's doing is of course obvious, just wondering about the details.
- roblabla 7y agofn is analogous to function in javascript. It takes the form `fn fnname<generics, generics>(argname: argtype) -> ReturnType` The "weird" 'a here is a generic lifetime. It basically says that the returned value (Term) must not outlive the input values (env and args). A &[Term<'a>] is a slice of Term<'a>. You can think of a slice as if it was an array whose size is known at runtime. A Result<Term<'a>, Error> is a type in the standard library used for error handling. It can either be an `Ok(Term)` or an `Err(Error)` depending on whether the function succeeded or not. (There is no special magic here, you can define your own type that can be "either" one or the other. They're called enums in Rust).
- thenewwazoo 7y agoJust a quick re-contextualization. I wouldn't say > "It basically says that the returned value (Term) must not outlive the input values (env and args)." It doesn't say that it must not, it says that it doesn't. Because lifetime information can only be inferred about callees and not callers, the lifetime annotation tells the compiler what it needs to know about the caller's environment. It tells the compiler "the lifetimes of these two things are tied together in my caller". The compiler can then use that information when the borrow checker is doing its magic. I've got an article written in my head about how to think about lifetimes in terms of visibility up and down the call stack.
- saghm 7y ago`'a` is a "lifetime parameter"; syntactically, it's like a generic parameter, only it's use to denote how long a given reference stays alive. This is one of the important mechanisms that Rust uses to ensure memory safety without a GC. Every reference in Rust has a lifetime, but many of them can be elided. `fn add<'a>` just means `define a function with a lifetime parameter called `'a`. This is analogous to if you defined a function as `fn add<T>` with a generic parameter T. `Env<'a>` means that the type `Env` is parametrized by a lifetime. As before, this is similar to a type being parametrized by a generic parameter. This is usually done when a type contains a reference; Rust requires that references stored in a type must have explicit lifetimes specified. In this case, we're specifying that the lifetime that Env is parametrized over is the lifetime parameter that the function defines. This doesn't mean anything in particular on it's own, but when we use `'a` elsewhere in the signature, it means that the two places use the same lifetime, which is necessary to specify sometimes to clarify to the compiler that something is safe. `&[Term<'a>]` is a slice of `Term<'a>` values; any type can go between the `[]`, e.g. `&[i32]` for a slice of 32-bit integers. A slice is a view into a sequence of values in memory. One of the things Rust does to aid the programmer in low-level optimizations is define separate types for sequences, namely slices, arrays, and vectors. A slice is a reference to another sequence type (which could be another slice); they're very cheap to make, but cannot be resized. An array is a value type (i.e. not a reference) and must be constant-sized. Finally, a `Vec` is resizable and heap-allocated, so it's expensive to create and copy (relative to slices). Finally, `Result<Term<'a>, Error>` means that the function returns a `Result` (i.e. either a success or an error); if no error occurs, it will return a `Term<'a>; otherwise, it will return the type `Error` (which is generally custom-defined in a given package, but could be an instance of a standard library error like `std::io::Error`. This is roughly analogous to returning a `Term<'a>` and marking that an exception of type `Error` may be thrown. One thing to note is that Rust's return values with lifetimes tend be tied to one of the parameters. In this case, the compiler's rules aren't sufficient to infer which of `Env` and `Term` to tie it to, so the programmer needs to disambiguate. The programmer chose to tie them all together, which presumably indicates that the output will reference both parameters in some way (although it's impossible to figure out more precisely how it will from the signature alone). Lifetimes are definitely the hardest part of Rust in my opinion; the important thing to realize is that changing the lifetime parameters in a function will never change the user-visible semantics of the code. If the lifetimes are invalid, the compiler will error, and if they are valid but not optimal, then the worst case is that that memory might be kept around longer than needed. Unless you're programming something that is super performance critical or will be long running, you generally won't have huge issues if you fix the lifetime compiler errors with trial and error when they come up (which is much less often than you might expect due to the compiler being able to infer them most of the time).
- monocasa 7y agoThe 'a stuff is marking the lifetimes. In this case saying that the return value in the happy case needs the inputs to last at least as long as the return value does before being freed because there might be a dependency between them.
- pornel 7y agoThe same `<'a>` name on all of the inputs and outputs tell the borrow checker that these things are related, i.e. they all borrow from the same thing. The borrow checker tracks these annotations through all the scopes and function calls back to the "thing" they borrow from, and checks that nothing is borrowed for too long. I think this particular case is trivial enough that the annotation could be omitted: fn add(env: Env, args: &[Term]) -> Result<Term, Error> { but in more complex cases where data from some arguments is returned, and from some it is not, it's useful to annotate which inputs/outputs are related and which aren't, so that the unrelated arguments can be temporary values without tainting everything else as temporary.
- hinkley 7y agoI've been trying to decide between learning Elixir and learning Rust and Elixir had a narrow lead because of employment potential and my growing dissatisfaction with Docker (container orchestration contains an ad-hoc, informally-specified, bug-ridden, slow implementation of half of Erlang). Then I learned that the Achilles' heel of the BEAM VM is that if native code crashes it takes the whole thing with it, so people have been using Rust for native code (NIFs). So now the plan is Elixir then Rust which is easing my FOMO quite a bit.
- davidw 7y ago> Then I learned that the Achilles' heel of the BEAM VM is that if native code crashes it takes the whole thing with it, That's not really an 'Achilles heel' any more than most other languages that allow you to load C code that could segfault or - maybe even worse - get stuck in an infinite loop. Indeed, the fact that BEAM focuses on and acknowledges this is an advantage for people trying to create robust systems. They've had workarounds in place for years, like running stuff in a separate system process.
- andy_ppp 7y agoThis is actually the point of Rustler, rust can make guarantees about crashes and as such you can write safe low level code that never crashes the VM.
- davidw 7y agoThey ought to point out the notion of dirty NIFs a bit more prominently: https://bgmarx.com/2018/08/15/using-dirty-schedulers-with-rustler/ https://bgmarx.com/2018/08/15/using-dirty-schedulers-with-ru... Not crashing is good, but not sufficient to play nicely with the BEAM scheduler. You need to also ensure you're not using up a lot of time in some long-running operation, unless it's a "dirty NIF".
- fortran77 7y agoYes. I write NIFs all the time, and it isn't crashing that's the problem for me (I'm a good programmer!) -- it's taking too much time. We have a NIF that does CUDA computations and we have to make sure we break them up right. Rust won't help me on this. I need C++ for CUDA.
- steveklabnik 7y agoFor an example of this library used in production: https://blog.discordapp.com/using-rust-to-scale-elixir-for-11-million-concurrent-users-c6f19fc029d3 https://blog.discordapp.com/using-rust-to-scale-elixir-for-1...
- skosch 7y agoRelated: I've found that to run time-consuming number-crunching code from Elixir, the most convenient approach is to call a Python script via ports. It's super easy and can't crash your VM. To convert from/to Erlang terms in your Python script, you can use a library like https://github.com/skosch/python-erlastic https://github.com/skosch/python-erlastic (a fork I maintain, the original project was abandoned).
- algaeontoast 7y agoDoes anyone know of an example of a rust library or function that’s been turned into a NIF? Ideally available on GitHub both before and after the conversion? Also, the “nerves” framework is basically 80% NIFS of embedded C code.
- elcritch 7y agoNerves overall has a much smaller portion of C nifs than 80%, probably < 10%. Mainly for properly trapping and cleaning up subprocesses. There’s some nifs too setup Linux specific sockets/ioctls. Still surprises me that almost every Linux kernel interface essentially overloads `ioctl`. Not sure of an example. Really just wrap any rust function with appropriate code to convert Erlang terms to rust and back... There’s a few elixir rustler howtos: https://medium.com/@jacob.lerche/writing-rust-nifs-for-your-elixir-code-with-the-rustler-package-d884a7c0dbe3 https://medium.com/@jacob.lerche/writing-rust-nifs-for-your-...
- lostjohnny 7y agoYou could check the list of packages depending on Rustler https://hex.pm/packages?search=depends%3Arustler https://hex.pm/packages?search=depends%3Arustler
- h91wka 7y agoAs a professional Erlang developer I don't trust libraries that are heavy on NIFs, and Rust isn't going to change this. Key feature of BEAM VM is process preemption. It makes NIF development harder than it looks, and typically performance of NIF-based libraries looks best in synthetic benchmarks.