3 ms·

As someone who worked with Erlang for a few of years I still wonder if using parallel processing in program top level is worth it or not. Because there are many

As someone who worked with Erlang for a few of years I still wonder if using parallel processing in program top level is worth it or not. Because there are many ways to process data efficiently using queues, pipelines, etc and being clear on when it happens rather than a "wild west" of Erlang where you need to manage processes, links, restarts, and it's harder to focus on the business logic.

stevan 4y ago
Do I understand you correctly in that you'd like more structure? E.g. that you can only deploy an `application` (= supervisor tree)?
- mapcars 4y ago
  I'm talking about development, deployment should be automated anyways. As a developer I think it's easier to think in terms of how to send data to hadoop, sqs queue etc for processing and read results later than keep in mind the supervision tree, messages, mailbox size, linking and so on. And the "processing" side can be as well implemented in Erlang just I don't feel Erlang's features are needed in the "top level development" and create more problems and barriers.
Kototama 4y ago
What do you mean with "process data efficiently"? You speak about performance? But the main goal of the design of Erlang is fault tolerance, not performance. You can still use queues or whatever in Erlang if you want, but having a fault-tolerant system with process tree and process supervision in other languages is a different story.
- mapcars 4y ago
  Right, I was mainly describing arguments for parallel processing which Erlang is also famous for. In terms of being fault-tolerant I think the modern approach with (micro)services is quite similar, one can have multiple services running and communicating using something like protobuf, having restart strategies, fallbacks and so on. From my experience Erlang doesn't offer any killer features in this case, does it?
  - ElectricalUnion 4y ago
    The killer feature is that the fault-tolerant behaviour is something either implicit, builtin or bundled [Erlang/OTP] with the language, instead of something that you need to either reinvent or bolt yourself to both your own code and your dependencies code as a afterthought when things start to fail.
    - mapcars 4y ago
      In Erlang the fault-tolerant behavior is not builtin either, only tools to make it. You still have to make the right supervision tree, dependencies between processes, links, making sure you handle the process termination messages correctly and many other details. In my experience in non-Erlang setups while doing requests to other services you have to check the response status and add some code handling it, so it's not really a complete afterthought. The only difference I see here is that Erlang handles failure in a real-time way, but it also can be done using some periodic task to query important services. And implementing in outside of Erlang gives more flexibility (think of Erlang cluster size and network limitations)
      - throwawaymaths 4y ago
        That's wrong. Fault tolerance is basically the default. Yes you have to build a supervision tree, but unless you're writing a one-off script, you have to build it to anyways to do anything.
        mapcars 4y ago
        Well you confirming what I said - you have to build it, it's not that Erlang programs automatically never fail and always handle problem correctly as required.
      - hosh 4y ago
        Our team had worked on an Elixir app for a couple years, before splitting off game logic into Dotnet. Scaling the dotnet server was a much different beast: - It wasn't designed to crash on failure. It uses thread pools with no supervision trees. We had to add in liveliness probes to check if it is alive. I've only had to use readiness checks for Elixir - No REPL. With a REPL in production, we can debug things live, even try patches to see if those work. Can't do that with Dotnet. That's also something that contributes to reliability Now, cluster size do matter. The way Erlang and BEAM was designed were for vertical scaling. You can minimize cluster size by biasing towards vertical scaling. That's what we do on our systems. There's a way to do that with Kubernetes so that we scale vertically during our daily traffic cycle. At some point though, you start looking at partial clustering topology for BEAM, or use one of the many process registeries that are better suited for dynamic membership. (The one bitwalker wrote comes to mind).
rozap 4y ago
But isn't the alternative wrangling infrastructure with yaml files?
- mapcars 4y ago
  It is yes. When you just starting a project Erlang's approach might be favorable but once the whole infrastructure is there the difference is negligible.

stevan 4y ago

Do I understand you correctly in that you'd like more structure? E.g. that you can only deploy an `application` (= supervisor tree)?

mapcars 4y ago
I'm talking about development, deployment should be automated anyways. As a developer I think it's easier to think in terms of how to send data to hadoop, sqs queue etc for processing and read results later than keep in mind the supervision tree, messages, mailbox size, linking and so on. And the "processing" side can be as well implemented in Erlang just I don't feel Erlang's features are needed in the "top level development" and create more problems and barriers.

Kototama 4y ago

What do you mean with "process data efficiently"? You speak about performance? But the main goal of the design of Erlang is fault tolerance, not performance. You can still use queues or whatever in Erlang if you want, but having a fault-tolerant system with process tree and process supervision in other languages is a different story.

mapcars 4y ago
Right, I was mainly describing arguments for parallel processing which Erlang is also famous for. In terms of being fault-tolerant I think the modern approach with (micro)services is quite similar, one can have multiple services running and communicating using something like protobuf, having restart strategies, fallbacks and so on. From my experience Erlang doesn't offer any killer features in this case, does it?
- ElectricalUnion 4y ago
  The killer feature is that the fault-tolerant behaviour is something either implicit, builtin or bundled [Erlang/OTP] with the language, instead of something that you need to either reinvent or bolt yourself to both your own code and your dependencies code as a afterthought when things start to fail.
  - mapcars 4y ago
    In Erlang the fault-tolerant behavior is not builtin either, only tools to make it. You still have to make the right supervision tree, dependencies between processes, links, making sure you handle the process termination messages correctly and many other details. In my experience in non-Erlang setups while doing requests to other services you have to check the response status and add some code handling it, so it's not really a complete afterthought. The only difference I see here is that Erlang handles failure in a real-time way, but it also can be done using some periodic task to query important services. And implementing in outside of Erlang gives more flexibility (think of Erlang cluster size and network limitations)
    - throwawaymaths 4y ago
      That's wrong. Fault tolerance is basically the default. Yes you have to build a supervision tree, but unless you're writing a one-off script, you have to build it to anyways to do anything.
      - mapcars 4y ago
        Well you confirming what I said - you have to build it, it's not that Erlang programs automatically never fail and always handle problem correctly as required.
    - hosh 4y ago
      Our team had worked on an Elixir app for a couple years, before splitting off game logic into Dotnet. Scaling the dotnet server was a much different beast: - It wasn't designed to crash on failure. It uses thread pools with no supervision trees. We had to add in liveliness probes to check if it is alive. I've only had to use readiness checks for Elixir - No REPL. With a REPL in production, we can debug things live, even try patches to see if those work. Can't do that with Dotnet. That's also something that contributes to reliability Now, cluster size do matter. The way Erlang and BEAM was designed were for vertical scaling. You can minimize cluster size by biasing towards vertical scaling. That's what we do on our systems. There's a way to do that with Kubernetes so that we scale vertically during our daily traffic cycle. At some point though, you start looking at partial clustering topology for BEAM, or use one of the many process registeries that are better suited for dynamic membership. (The one bitwalker wrote comes to mind).

rozap 4y ago

But isn't the alternative wrangling infrastructure with yaml files?

mapcars 4y ago
It is yes. When you just starting a project Erlang's approach might be favorable but once the whole infrastructure is there the difference is negligible.