4 ms·
If you follow OTP design principles, you end up with a supervision tree, and a lot of code like... ok = do_something_that_might_fail() If it returns ok: gr
by toast0 3y ago
If you follow OTP design principles, you end up with a supervision tree, and a lot of code like...
ok = do_something_that_might_fail()
If it returns ok: great, it worked and you move on. If it doesn't return ok, the process crashes, you get a crash report, and the supervisor restarts it, if that's how the supervisor is configured. Presumably it starts properly and deals with future requests.
There's two issues you might rapidly encounter.
1) if a supervised process restarts too many times in an interval, the supervisor will stop (and presumably restart), and that cascades up to potentially your node stopping. This is by design, and has good reasons, but might not be expected and might not be a good fit for larger nodes running many things.
2) if your process crashes, its message queue (mailbox) is discarded, and if you were sending to a process registered by name or process group (pg), the name is now unregistered. This means a service process crashing will discard several requests; the one in progress which is probably fine (it crashed after all), but also others that could have been serviced. In my experience, you end up wanting to catch errors in service processes, log them, and move on to the next request, so you don't lose unrelated requests. Depending on your application, a restart might be better, or you might run each request in a fresh process for isolation... Lots of ways to manage this.
- roeles 3y agoThanks for your insights!
- pmontra 3y agoWhat we did in a project was storing message data in a JSONB field of a database table, with an id of the recipient process in another field. We used the module name for that. Then we sent a normal message with the id of the database record. If the process succeeded it marked the record as done. If it crashed, any of those proceesses would read all of its unprocessed records upon restart and start working on them. We took care never to restart a failed process too fast. Sometimes they kept failing for hours or days until we had a fix, then they processed their backlog. If some jobs were not important anymore or must not be performed because something else took care of them (even people manually) the fix had to skip them. Manual UPDATEs, code, etc.
- jimsimmons 3y agoWhy wouldn't you be able to implement this in Rust or Python? Seems like typical exception handling. Erlang isn't even type checked
- conradfr 3y agoI think the let it fails slogan is taken too literally, especially by people who don't use OTP. You can handle errors and exceptions if you want, the supervisors etc are more here for the unexpected failures that happens for all sort of reasons in any software. Sometimes it also hides bugs because your solution appears to work correctly;)
- sangnoir 3y ago> I think the let it fails slogan is taken too literally, especially by people who don't use OTP I like the philosophy - when taken literally, it saves writing error-handling code at unit-level for an entire class of errors (equivalent to JVMs unchecked exceptions). A well-architected supervision tree will handle that for you, when paired with the excellent out-of-the-box instrumentation for observability. > You can handle errors and exceptions if you want Yes, which is why it's standard to pattern-match on a function's results to check if returned a known err or not - after all, (checked) errors are just a return value in Erlang/BEAM.
- throwawaymaths 3y agoThat's the basics. The next level is, what if you want one process dying to interrupt and trigger killing another process that is associated with it -- perhaps it's the parent in an rpc -- that's the easiest case (and trigger the correct exception message in the logs of both nodes) Oh. And it's on the other side of a cluster. Sure, you could do it in Python or rust. It won't be zero lines of code. You're probably gonna get it wrong.
- felixgallo 3y ago1. You can implement supervisor trees in rust or python, but neither of them have a runtime that supports that out of the box and an exception handling system that works harmoniously with that runtime, so you'd have to implement that all yourself, and it turns out to be a non-trivial task. 2. erlang has a variety of type checks at a number of conceptual levels, so strongly recommend you go plow through something like https://learnyousomeerlang.com/ https://learnyousomeerlang.com/ in order to improve your understanding here.
- alberth 3y ago@toast0, would you still use Erlang/OTP today? If so, for what kind of apps. If not anymore, what would you use instead.
- toast0 3y agoYes. Certainly. In my mind, there's two really good fits (and some cross over between them). a) binary matching syntax is really nice for dealing with bit-packed things; although it's not pretty if dealing with little endian values where there's a couple bits in one byte and a couple more in the neighboring byte --- you've got to get the pieces and put them together if it's not just whole bytes. Big endian bit packed structures are easy, but little endian is dominant these days. I don't know if performance is good, but it's easy to read and write for developers. b) anything with a large number of connections and significant state per connection. A chat server, video conference, etc. This is why you see 'everyone' build chat from ejabberd. Erlang makes code for this kind of service fairly simple, and hotloading means you can fix bugs without kicking everyone off to restart. Observability features make it reasonable to see what's going on in your system and where bottlenecks are.
- alberth 3y agoThanks for your insights as always. Related to (b), any thoughts on Rustler? My understanding it’s used extensively at Discord to eliminate Erlang/OTP performance bottlenecks. https://github.com/rusterlium/rustler https://github.com/rusterlium/rustler
- toast0 3y agoI haven't used it. I've written and used NIFs, but mostly they're simple enough that C is fine, and Rust isn't really needed. I think discord ends up with complex data structures in their native layer; Rust seems reasonable for that, but I never experienced that need. That said, I did have a concurrency issue with a NIF I used, but I don't know if Rust would have been helpful or just made it really hard to express.
- asa400 3y ago