4 ms·

For me the question often hasn't been "is it distributed" but "how often do you need to spawn some async task (that can be run supervised, or unsupervised, in a

For me the question often hasn't been "is it distributed" but "how often do you need to spawn some async task (that can be run supervised, or unsupervised, in a fault tolerant manner)". I find that is often a lot more relatable to average-joe-tasks than bigger picture distributed ideas, even if they're actually the same thing at their heart. I just don't want one request to blow out the system, or some sporadically turned remote API to hose everything, Elixir gives me that (with the simplicity other languages often need other stuff stuck onto to handle).

davidw 3y ago
It's kind of ironic that stock Erlang/OTP has fairly crude tools for "spawn some async task" and deal with it failing. Your options are * Keep trying to restart it forever if it fails (by setting the timeout so low that it'll keep trying) * Try a set number of restarts, then pass the failure higher up, potentially taking down the whole system if there are enough failures. * Just let it fail and walk away. https://www.erlang.org/doc/design_principles/sup_princ.html#restart-strategy https://www.erlang.org/doc/design_principles/sup_princ.html#... If you have something like a web site with a database, where 1) the database is critical to the normal functioning of the system, but 2) you still want the site up to let users know that "it's broken and we're fixing it", and maybe also send some alerts out, you might want something more like an exponential backoff or some other more involved strategy, which isn't shipped with Erlang.
- sph 3y ago
  There are infinite ways of dealing with failure. Erlang gives you the building blocks, if you need something more complex, you're supposed to have to write some code. How is this surprising? In fact, I am writing this week a dynamic process pool in Elixir which is so peculiar in its requirements I don't think I've ever seen anywhere, and it's in total 500 lines of code spread over 2 GenServers (+ Registry and DynamicSupervisor). I reckon it would easily be 25x as much in C++ or Rust. If you want exponential backoff, I suppose it should be pretty easy to implement with a custom Supervisor (though the best practice would be to have a sidecar GenServer to deal with the backoff/alerting business rules, and using a regular Supervisor.)
  - davidw 3y ago
    This is the kind of thing I'm talking about though... we go from "Erlang is a piece of cake for distributed systems" to some pretty advanced concepts and telling people to "roll your own" for an extremely common use case: not wanting your entire application to be taken down by an unreachable external service. Back in the day, I found this to be a pretty good 'circuit breaker' type of thing: https://github.com/jlouis/fuse https://github.com/jlouis/fuse I wrote something similar for the (OTP) application level: https://github.com/davidw/hardcore https://github.com/davidw/hardcore
    - sph 3y ago
      > not wanting your entire application to be taken down by an unreachable external service If your BEAM app goes down because your GenServer crashes, something is terribly wrong with your code.

davidw 3y ago

It's kind of ironic that stock Erlang/OTP has fairly crude tools for "spawn some async task" and deal with it failing. Your options are * Keep trying to restart it forever if it fails (by setting the timeout so low that it'll keep trying) * Try a set number of restarts, then pass the failure higher up, potentially taking down the whole system if there are enough failures. * Just let it fail and walk away. https://www.erlang.org/doc/design_principles/sup_princ.html#restart-strategy https://www.erlang.org/doc/design_principles/sup_princ.html#... If you have something like a web site with a database, where 1) the database is critical to the normal functioning of the system, but 2) you still want the site up to let users know that "it's broken and we're fixing it", and maybe also send some alerts out, you might want something more like an exponential backoff or some other more involved strategy, which isn't shipped with Erlang.

sph 3y ago
There are infinite ways of dealing with failure. Erlang gives you the building blocks, if you need something more complex, you're supposed to have to write some code. How is this surprising? In fact, I am writing this week a dynamic process pool in Elixir which is so peculiar in its requirements I don't think I've ever seen anywhere, and it's in total 500 lines of code spread over 2 GenServers (+ Registry and DynamicSupervisor). I reckon it would easily be 25x as much in C++ or Rust. If you want exponential backoff, I suppose it should be pretty easy to implement with a custom Supervisor (though the best practice would be to have a sidecar GenServer to deal with the backoff/alerting business rules, and using a regular Supervisor.)
- davidw 3y ago
  This is the kind of thing I'm talking about though... we go from "Erlang is a piece of cake for distributed systems" to some pretty advanced concepts and telling people to "roll your own" for an extremely common use case: not wanting your entire application to be taken down by an unreachable external service. Back in the day, I found this to be a pretty good 'circuit breaker' type of thing: https://github.com/jlouis/fuse https://github.com/jlouis/fuse I wrote something similar for the (OTP) application level: https://github.com/davidw/hardcore https://github.com/davidw/hardcore
  - sph 3y ago
    > not wanting your entire application to be taken down by an unreachable external service If your BEAM app goes down because your GenServer crashes, something is terribly wrong with your code.