5 ms·
If you look at what NIFs exist up to this point and where Erlang itself tends to use that mechanism, they are all for very tiny uses, like type checks and such.
by jerf 10y ago
If you look at what NIFs exist up to this point and where Erlang itself tends to use that mechanism, they are all for very tiny uses, like type checks and such. But that's probably precisely because they're too scary to use for anything very large. The existence of a provably-safe and practical mechanism for writing them may cause and/or require some changes in the ecosystem to account for the fact it's now possible to write NIFs that may take a long time to run.
I wouldn't be surprised that at least one of this library or the Erlang VM itself develop an official way to easily use a NIF to run a longer-running process safely. There hasn't been a need up to this point because there hasn't been a such thing as a long-running NIF.
At the moment, if you're looking at something that may take several seconds you're probably still better off coordinating something over a port to an external process or something. Or working with this project to make it feasible to run a long-running native process.
Trying to write a NIF that could somehow yield back and then be "continued" later would be a royal pain. Cooperative scheduling was enough of a pain when we were in the hundreds of processes mostly doing nothing on OSes; trying to use it on an Erlang server is probably just infeasible.
- dilatedmind 10y agoit is almost trivial to write nifs which yield and continue, enif_schedule_nif exists for this purpose. nifs are ideal for operations which mutate binaries, eg hashing or unmasking a websocket frame
- im_down_w_otp 10y agoYeah... but that hasn't existed for very long. :-)
- jerf 10y agoI didn't just mean the mere ability to yield, which isn't that complicated; I meant code supporting the common use cases around having a long-running NIF, such as the person I replied to's comment about having some sort of build in support for being able to answer back to a PID or something. I could easily imagine a NIF library that provided an easy ability to set up an independent thread pool and work sharing mechanism specific to that NIF, which sends answers back to given PIDs, has timeout support or the ability to query it for progress, etc. All fairly straightforward stuff to expect to develop over time, but in a world where long running NIFs are too dangerous to hardly even contemplate, they haven't developed yet. There's some some parameters in there that are going to be tricky to set up correctly (number of workers in the pool for a given NIF has a lot of implications at scale), but in the end it's not significantly different than communicating to a separate process on the some machine; it's all the same resources being used.
- quodlibetor 10y ago> to write a NIF that could somehow yield back and then be "continued" later would be a royal pain. I know... nothing... about the NIF interface, but the way that I would want this to work wouldn't involve NIF resumption at all, but rather for NIFs to be able to put responses back on a channel. Unfortunately neither quick internet searches nor searching the rustler docs revealed anything about a NIF/channel interface. I'm not sure how feasible it is to pass a channel into a nif and allow it to pass that into some thread and return, nor how good of an idea that would be.
- im_down_w_otp 10y agoThis wouldn't make sense because NIFs are supposed to be just straight function calls that provide immediate returns. It's a blocking interface. If you want the behavior you're talking about you would communicate via ports, or you'd write a NIF that maintained its own off scheduler thread(s), returned immediately, but kept a reference to the calling pid, and then sent a message to that pid at some later date. Several IO related NIFs do that today. That's actually an excellent use for Rust in this case because it can help you make that multi-threaded implementation safer and more reliable.
- quodlibetor 10y agoThat all makes sense, and the immediate return + threads solution you describe sounds like what I was imagining, I just didn't have the context to use the correct parts of the Erlang FFI. I'm glad that there is a pattern for that kind of thing!