4 ms·
To add to the point of one of the "better solutions": if you've got a native library that does heavy lifting (for example, a game physics engine), and you want
by derefr 10y ago
To add to the point of one of the "better solutions": if you've got a native library that does heavy lifting (for example, a game physics engine), and you want to "integrate" it with Erlang, please don't use NIFs. Use ports. Write a small C-process wrapper around your native library and talk to it over streams or sockets or shared memory. Use the OS's pre-emptive scheduler for what it's for.
Externalizing your bulky native code into its own port-programs heavily increases your system's robustness. Very few native libraries implement a Rust-like set of safe abstractions and then rely solely on them; most just do stupid native things with pointers et al. So most native libraries likely will abort() at some point or another. When the one you're relying on does, it will of course take down "its own OS process." You don't want that OS process to be your Erlang node. You want that crash to be isolated such that the only state that's destroyed is the corrupted state of the subsystem that caused the crash. And then you want a supervisor running in your Erlang node to see the dead port and restart it, so the system can keep chugging along.
Of course, if you want that to be a low-overhead solution, then you should try as much as possible to cache anything you'd have to repeatedly send to the C process, within the C process. Treat your externalized native-library "processor" server like an SQL database server: insert complex state into it, get handles back, then manipulate the state without retrieving it by using high-level queries and commands on those handles. (I say cache, but I don't mean persist. The state your Erlang node hands a port-program should all be derived state that the Erlang node holds the canonical copies of, that you can feed into the port-program again when it crashes. Treat port-program state like state in memcached.)
---
ETA: I've never seen this design myself in the wild, but I've heard it suggested a couple of times, and it's kind of a cool idea:
Instead of a native C port-program, you can get the same set of advantages as the above from making your library a set of "dirty" NIFs running in its own isolated Erlang node. The Erlang runtime itself doesn't have much overhead, so it's pretty cheap to use Erlang as the way to "network-enable" each library you have into its own server process. Then your "business logic" Erlang node can communicate with your native-library-wrapper Erlang node, over the Erlang distribution protocol (which is very cheap if they're on the same machine.) Saves you a bunch of hassle in trying to beat C into a form that does IPC protocol-handling well.
- yuubi 10y ago> trying to beat C into a form that does IPC protocol-handling well. The opposite worked at least once: have Erlang code deal with C-friendly structures. Last time I got to work on an Erlang project, we needed an external C program to do some network stuff we couldn't do in Erlang. We just defined a simplistic protocol easy to parse from C, with {packet, 4} framing (each message was preceded by a 4-byte length) wrapped around the messages. Pretty much any format easy to parse or generate in C is trivial to deal with in Erlang.
- new_hackers 10y agoThat last bit sounded very cool. When you said "communicate with streams or sockets or shared memory", I immediately thought "oh no, I don't want my simple little C code to have to do all that!" But by wrapping it with Erlang, then my C code can do what its good at, and Erlang can do what it is good at.
- di4na 10y agoThat last part feel really like dirty schedulers. Which have an experimental implementation in the BEAM recently.