3 ms·

I'd rather use an external job scheduler and message queue, eg in kubernetes, rather than build it into the language runtime. With a message queue and horizonta

I'd rather use an external job scheduler and message queue, eg in kubernetes, rather than build it into the language runtime. With a message queue and horizontal pod autoscaling you can very quickly build an easy distributed workload with (at least to me) little effort in any language

throwawaymaths 2y ago
Or you could do with almost zero effort in a BEAM language. And sometimes setting up a kuberbetes cluster is not easy. Suppose you want to have one part that is hosted on a cloud service and another that is on-prem. It's certainly possible with kuberbetes, but you'll be fighting the kuberbetes paradigm. Plus you're checking in to this kuberbetes middleman for part A to understand the availability of part B, for example. Kuberbetes is highly devex optimized for stateless services that are kinda cattle. Not all use cases fit so neatly into that rubric.
- conradludgate 2y ago
  I would not say it's zero effort in BEAM. I wasn't able to figure out how to get it to work at all (I spent a weekend trying to build a distributed auth system, failed). But I would be able to spin up kubernetes, rabbitmq and a non BEAM language in not very long. > Suppose you want to have one part that is hosted on a cloud service and another that is on-prem. Is this hard with kubernetes? Seems pretty simple to add node taints and set up the appropriate VPN to me. BEAM is not magic. It will be doing a lot of the same stuff that any other scheduler or message queue will do, so I would personally rather choose the more composable solution rather than have to manage BEAM and be forced to use one family of languages only. I'm thinking if your company is already running kubernetes, adding BEAM on top is probably more effort than adding a message queue (given my limited understanding of BEAM)
  - throwawaymaths 2y ago
    Never said it was zero effort. It's almost zero effort. But you have to have deep knowledge of BEAM primitives if you intend on doing something crazy (still almost zero effort). If you don't know what you're doing or don't have experience in the BEAM you will probably get it wrong. Sometimes libraries haven't gotten it right so composability might not be there and you might have to write a sidecar genserver and supervisor (~40 LOC) Good news is in elixir, though it is not easy to write (still low effort) most of it is pretty straightforward to read so a junior could probably successfully work through and understand a CR. The other very nice thing about doing it all in the BEAM is that you can make distribution and availability guarantees part of your testing suite very easy, because its even part of you localdev flow. That also means your devs will write that test, instead of of assuming they understand how that kuberbetes operator works (they don't).
    - petejodo 2y ago
      To add onto this, you actually write code when on the BEAM rather than yaml and feels like it takes less lines to accomplish the same things

throwawaymaths 2y ago

Or you could do with almost zero effort in a BEAM language. And sometimes setting up a kuberbetes cluster is not easy. Suppose you want to have one part that is hosted on a cloud service and another that is on-prem. It's certainly possible with kuberbetes, but you'll be fighting the kuberbetes paradigm. Plus you're checking in to this kuberbetes middleman for part A to understand the availability of part B, for example. Kuberbetes is highly devex optimized for stateless services that are kinda cattle. Not all use cases fit so neatly into that rubric.

conradludgate 2y ago
I would not say it's zero effort in BEAM. I wasn't able to figure out how to get it to work at all (I spent a weekend trying to build a distributed auth system, failed). But I would be able to spin up kubernetes, rabbitmq and a non BEAM language in not very long. > Suppose you want to have one part that is hosted on a cloud service and another that is on-prem. Is this hard with kubernetes? Seems pretty simple to add node taints and set up the appropriate VPN to me. BEAM is not magic. It will be doing a lot of the same stuff that any other scheduler or message queue will do, so I would personally rather choose the more composable solution rather than have to manage BEAM and be forced to use one family of languages only. I'm thinking if your company is already running kubernetes, adding BEAM on top is probably more effort than adding a message queue (given my limited understanding of BEAM)
- throwawaymaths 2y ago
  Never said it was zero effort. It's almost zero effort. But you have to have deep knowledge of BEAM primitives if you intend on doing something crazy (still almost zero effort). If you don't know what you're doing or don't have experience in the BEAM you will probably get it wrong. Sometimes libraries haven't gotten it right so composability might not be there and you might have to write a sidecar genserver and supervisor (~40 LOC) Good news is in elixir, though it is not easy to write (still low effort) most of it is pretty straightforward to read so a junior could probably successfully work through and understand a CR. The other very nice thing about doing it all in the BEAM is that you can make distribution and availability guarantees part of your testing suite very easy, because its even part of you localdev flow. That also means your devs will write that test, instead of of assuming they understand how that kuberbetes operator works (they don't).
  - petejodo 2y ago
    To add onto this, you actually write code when on the BEAM rather than yaml and feels like it takes less lines to accomplish the same things