7 ms·

Elixir multiple processes: Basics

131 points by ck3g 9y ago

sbuttgereit 9y ago
I'm in the process of learning Elixir and the observation I'm about to make may be completely off-base. With that said... Seems like the section on "When to use processes" is selling processes short a bit. Certain kinds of state management would seem to call for processes to either manage or even hold state... but processes (as I understand them) are also key in designing for reliability. So I would think I may well want to organize my code relative to processes as I'm also figuring out the supervision trees and various failure scenarios. And yes, concurrency issues as well. If I'm wrong on this, I'd be happy to be set straight. Anyway, yes, the section I speak of does get to some of the other parts of what I mention, but the emphasis on state management seems to distort and unbalance the view of what you might want as a process. [edited a touch for clarity]
- macintux 9y ago
  You are correct. That's something I've been a bit worried about with people coming to Elixir thinking it's basically a faster/robust Ruby: are they absorbing the full Erlang Weltanschauung?
  - udfalkso 9y ago
    (I had to look it up) Welt·an·schau·ung ˈveltˌänˌSHouəNG a particular philosophy or view of life; the worldview of an individual or group. -- Personally I came for the faster/robust Ruby first and then over time discovered (and am still discovering) the other powerful pieces. I think that's ok.
    - macintux 9y ago
      Absolutely that's ok. Everything's a journey. (I tend to get lazy about learning, so perhaps I doth project too much.)
  - debacle 9y ago
    Was Elixir designed with Ruby in mind?
    - macintux 9y ago
      It was designed by José Valim, a Rails core team member.
- ck3g 9y ago
  I think you are right. Good point. I am also in the process of learning, so your reply is a valuable for me. Thanks But does reliability in Elixir/Erlang come only from processes or is it result of the "teamwork" of several features of the platform itself?
  - dozzie 9y ago
    You won't have Erlang-style reliability (predictable recovery after localized errors) without links, monitors, and supervisors, which themselves are built with links and monitors.
    - lostcolony 9y ago
      Plus immutability, plus message passing paradigm (and the associated distribution). The immutability prevents entire classes of bugs, and helps enforce predictability on the part of your code. It also is assumed at least in part when it comes to message passing (else, what happens if you change a referenced binary after passing it in a message? Does the message change too, or remain static? Either way, complexity) The message passing paradigm for interprocess communication ensures that the developer is forced to ask "What happens if this isn't received", i.e., if the process is down or missing or etc. Any sort of 'return', via a message sent back, could not happen; this is akin to a called function throwing, which again, requires the developer to ask 'what happens if we never get this response back'? The answer is never to wait indefinitely, it's to...what? Other languages make it very easy to ignore possible exceptions; in Erlang, not only are they likely handled (if you have your supervisor structure in place), but you're forced to decide how such a thing in another process affects the current process (if at all). Because of the message passing paradigm, the distribution story is simplified; it's largely transparent as to whether the process you're sending a message to is local, or remote. This allows you to build in redundancy across nodes without many of the complexities (and thus, room for errors) that other languages give you, nor the lying abstractions many languages give you (such as RMI). And having the distribution Erlang does helps with the reliability side of things, as it makes it comparatively straightforward (still complex, but far less so than most other languages) to get solid handling in the event of machine failure.
- fasteo 9y ago
  Elixir newbie here. My understanding if that Agent[1] was designed specifically to keep state. But then again, an Agent is a GenServer is a process, so I guess the post just served as a introduction to the concept of a Elixir process. [1] https://hexdocs.pm/elixir/Agent.html https://hexdocs.pm/elixir/Agent.html
  - ck3g 9y ago
    Yes. That is correct. I am going to cover Agents, Tasks and OTP in the following articles. I thought the article would be too big if I cover all three topics at once.
redshirt 9y ago
Very cool. I'm adding this to my list of easy concurrency tools. The main thing I like is the statelessness. That's where most people screw up parallel programs. Seems like there are many languages/libraries trying to make concurrency easier to implement in practice. Most notably for C++ (my fav since I have to use it for most of my work projects): Intel TBB (definitely the go to for most things), RaftLib (saw at C++Now last year) is probably the easiest to understand (same theme as this post, super easy concurrency for c++). Even Java seems to make concurrency rather easy with it's thread pools and relatively strait-forward synchronized sections.
- MaxBarraclough 9y ago
  I recommend taking a look at .Net/C#'s concurrency machinery. It makes it terribly easy to use its thread-pool, and its async/await stuff is very cool. Think TBB tasks, but with very nice sugar. My understanding is that Java's current concurrency offerings are much like C#'s, but... worse.
- lostcolony 9y ago
  Okay, taking a step back - this isn't just another concurrency tool. This is a language built for concurrency. Taking another step back - this isn't just a language built for concurrency. This is a language built for -resiliency-. Taking another step back - this isn't just a language built for resiliency, this is a language built atop a VM built for resiliency -over 30 years ago- (the Erlang VM, aka, the BEAM, which Elixir runs on). Okay. Why does this matter? Well, the thing is, resiliency encompasses concurrency and distribution, both. It prioritizes error minimization and, more importantly, the ability to recover from errors. This isn't just a try/catch; this is a "something you completely failed to even expect caused things to fail in a way you can't even imagine, and the system still handled it". It achieves that via immutability, concurrency, and distribution. Ensure your data is immutable, so that state has to be very explicit (it's not stateless...a process has state. But it's very explicit state; as a developer you can't help but handle it and be very aware of it). Ensure bad states are dropped, and the system can recover the execution unit from a good state. If it can't, allow a user defined subsystem to fail, as the intricacies between the entire subsystem are implicitly stateful, and restart the whole thing from a known good state. If even that fails, keep climbing the supervisor tree, restarting larger and larger subsystems, until you restart the entire -application-, assuming that the intricacies across subsystems have gotten into a bad state, and again, restart from a good state. These principles have been around a long time, but Erlang is one of the first languages to put them into practice, again, over 30 years ago. There's been a lot of time since to see they actually work, and to further refine them. The difficulty with concurrency is not actually being concurrent (per your post, there are a LOT of ways to implement concurrency); the difficulty is doing it in a way that it behaves how you want it, even in the face of user's doing things you don't anticipate, external resources doing things you don't anticipate, your own code doing things you don't anticipate, etc. The design decisions that went into Erlang focused on minimizing errors...in so doing, it provides a way that most kinds of errors are handled transparently (from logic bugs to actual machine failure), while making it much harder to do things that it can't recover from (memory leaks are comparatively difficult to cause, as are deadlocks, for instance). To give you an idea, the first commercial product built with Erlang boasted (famously) 9 9s of uptime. Meaning something on the order of ~30ms of downtime a year. That includes planned downtime, visible errors, etc. I've seen Erlang systems in production...even with a rather critical bug in one, the system just -worked- for -years-, before someone noted an oddity in the logs, dug into it, and went "Oh my God" over how severe the issue was. But, again, Erlang's supervisor process just restarted it, and it was never noticed. I helped write CNN's current video ingest system in Erlang. It's been working without issue for years, despite no maintenance or attention (to where even most of the developers have left, but it still just...works). Even much less complex Ruby, Java, and Javascript systems are plagued with constant bugs. I would not say the devs on this project were just that much better (though the process was a little different, with little product owner involvement), but that the language we picked was so much better geared toward fault tolerance.
nickjj 9y ago
Maybe I'm thinking about this incorrectly but when it comes to web application development and concurrency, the things I would typically want to run in a separate process are very important tasks. For example, let's say you're sending emails out. In Rails, Flask, etc. you would typically offload this to a background worker like Sidekiq or Celery. These are dedicated libraries that run in their own process and handle job processing. Both tools allow you to see if that job was successful or failed and deals with retries. They also use Redis as a back-end so your state persists outside of the application code. If you just willynilly spawn a process and start sending stuff through it, how do you ensure everything works as expected and what happens to all of your state if the Erlang VM goes down? I love the idea of doing this, but in real world practice, it sounds like you would still need the Elixir equiv. of Sidekiq / Celery to handle ensuring these spawned tasks are trackable, right?
- narrowtux 9y ago
  Of course you could make such a system in Elixir, and I'd always recommend doing things within the erlang VM (beam) instead of figuring out how to deploy another service (like install sidekiq with all its dependencies). However, processes in Erlang/Elixir are used ubiquitously. Most of the time, they run forever like some sort of service and are restarted when they crash. Sometimes, you won't care about the result. That's the reason why features such as tracking success/failure in a persistent data store is not a core feature. You could easily add those features to your processes though, the necessary API is already there. The alternative is to use message queues that are durable. Only ack the message when you successfully handled it. If your application crashes before the task was completed, the task will be redelivered the next time the application starts.
  - nickjj 9y ago
    I definitely wouldn't want to be responsible for writing this code myself. Something like Sidekiq has been worked on for 5+ years by hundreds of people. Does anything exist in the Elixir world that's as battle hardened and has comparable features to Sidekiq / Celery?

131 points by ck3g 9y ago

I'm in the process of learning Elixir and the observation I'm about to make may be completely off-base. With that said... Seems like the section on "When to use processes" is selling processes short a bit. Certain kinds of state management would seem to call for processes to either manage or even hold state... but processes (as I understand them) are also key in designing for reliability. So I would think I may well want to organize my code relative to processes as I'm also figuring out the supervision trees and various failure scenarios. And yes, concurrency issues as well. If I'm wrong on this, I'd be happy to be set straight. Anyway, yes, the section I speak of does get to some of the other parts of what I mention, but the emphasis on state management seems to distort and unbalance the view of what you might want as a process. [edited a touch for clarity]

macintux 9y ago
You are correct. That's something I've been a bit worried about with people coming to Elixir thinking it's basically a faster/robust Ruby: are they absorbing the full Erlang Weltanschauung?
- udfalkso 9y ago
  (I had to look it up) Welt·an·schau·ung ˈveltˌänˌSHouəNG a particular philosophy or view of life; the worldview of an individual or group. -- Personally I came for the faster/robust Ruby first and then over time discovered (and am still discovering) the other powerful pieces. I think that's ok.
  - macintux 9y ago
    Absolutely that's ok. Everything's a journey. (I tend to get lazy about learning, so perhaps I doth project too much.)
- debacle 9y ago
  Was Elixir designed with Ruby in mind?
  - macintux 9y ago
    It was designed by José Valim, a Rails core team member.
ck3g 9y ago
I think you are right. Good point. I am also in the process of learning, so your reply is a valuable for me. Thanks But does reliability in Elixir/Erlang come only from processes or is it result of the "teamwork" of several features of the platform itself?
- dozzie 9y ago
  You won't have Erlang-style reliability (predictable recovery after localized errors) without links, monitors, and supervisors, which themselves are built with links and monitors.
  - lostcolony 9y ago
    Plus immutability, plus message passing paradigm (and the associated distribution). The immutability prevents entire classes of bugs, and helps enforce predictability on the part of your code. It also is assumed at least in part when it comes to message passing (else, what happens if you change a referenced binary after passing it in a message? Does the message change too, or remain static? Either way, complexity) The message passing paradigm for interprocess communication ensures that the developer is forced to ask "What happens if this isn't received", i.e., if the process is down or missing or etc. Any sort of 'return', via a message sent back, could not happen; this is akin to a called function throwing, which again, requires the developer to ask 'what happens if we never get this response back'? The answer is never to wait indefinitely, it's to...what? Other languages make it very easy to ignore possible exceptions; in Erlang, not only are they likely handled (if you have your supervisor structure in place), but you're forced to decide how such a thing in another process affects the current process (if at all). Because of the message passing paradigm, the distribution story is simplified; it's largely transparent as to whether the process you're sending a message to is local, or remote. This allows you to build in redundancy across nodes without many of the complexities (and thus, room for errors) that other languages give you, nor the lying abstractions many languages give you (such as RMI). And having the distribution Erlang does helps with the reliability side of things, as it makes it comparatively straightforward (still complex, but far less so than most other languages) to get solid handling in the event of machine failure.
fasteo 9y ago
Elixir newbie here. My understanding if that Agent[1] was designed specifically to keep state. But then again, an Agent is a GenServer is a process, so I guess the post just served as a introduction to the concept of a Elixir process. [1] https://hexdocs.pm/elixir/Agent.html https://hexdocs.pm/elixir/Agent.html
- ck3g 9y ago
  Yes. That is correct. I am going to cover Agents, Tasks and OTP in the following articles. I thought the article would be too big if I cover all three topics at once.

redshirt 9y ago

Very cool. I'm adding this to my list of easy concurrency tools. The main thing I like is the statelessness. That's where most people screw up parallel programs. Seems like there are many languages/libraries trying to make concurrency easier to implement in practice. Most notably for C++ (my fav since I have to use it for most of my work projects): Intel TBB (definitely the go to for most things), RaftLib (saw at C++Now last year) is probably the easiest to understand (same theme as this post, super easy concurrency for c++). Even Java seems to make concurrency rather easy with it's thread pools and relatively strait-forward synchronized sections.

MaxBarraclough 9y ago
I recommend taking a look at .Net/C#'s concurrency machinery. It makes it terribly easy to use its thread-pool, and its async/await stuff is very cool. Think TBB tasks, but with very nice sugar. My understanding is that Java's current concurrency offerings are much like C#'s, but... worse.
lostcolony 9y ago
Okay, taking a step back - this isn't just another concurrency tool. This is a language built for concurrency. Taking another step back - this isn't just a language built for concurrency. This is a language built for -resiliency-. Taking another step back - this isn't just a language built for resiliency, this is a language built atop a VM built for resiliency -over 30 years ago- (the Erlang VM, aka, the BEAM, which Elixir runs on). Okay. Why does this matter? Well, the thing is, resiliency encompasses concurrency and distribution, both. It prioritizes error minimization and, more importantly, the ability to recover from errors. This isn't just a try/catch; this is a "something you completely failed to even expect caused things to fail in a way you can't even imagine, and the system still handled it". It achieves that via immutability, concurrency, and distribution. Ensure your data is immutable, so that state has to be very explicit (it's not stateless...a process has state. But it's very explicit state; as a developer you can't help but handle it and be very aware of it). Ensure bad states are dropped, and the system can recover the execution unit from a good state. If it can't, allow a user defined subsystem to fail, as the intricacies between the entire subsystem are implicitly stateful, and restart the whole thing from a known good state. If even that fails, keep climbing the supervisor tree, restarting larger and larger subsystems, until you restart the entire -application-, assuming that the intricacies across subsystems have gotten into a bad state, and again, restart from a good state. These principles have been around a long time, but Erlang is one of the first languages to put them into practice, again, over 30 years ago. There's been a lot of time since to see they actually work, and to further refine them. The difficulty with concurrency is not actually being concurrent (per your post, there are a LOT of ways to implement concurrency); the difficulty is doing it in a way that it behaves how you want it, even in the face of user's doing things you don't anticipate, external resources doing things you don't anticipate, your own code doing things you don't anticipate, etc. The design decisions that went into Erlang focused on minimizing errors...in so doing, it provides a way that most kinds of errors are handled transparently (from logic bugs to actual machine failure), while making it much harder to do things that it can't recover from (memory leaks are comparatively difficult to cause, as are deadlocks, for instance). To give you an idea, the first commercial product built with Erlang boasted (famously) 9 9s of uptime. Meaning something on the order of ~30ms of downtime a year. That includes planned downtime, visible errors, etc. I've seen Erlang systems in production...even with a rather critical bug in one, the system just -worked- for -years-, before someone noted an oddity in the logs, dug into it, and went "Oh my God" over how severe the issue was. But, again, Erlang's supervisor process just restarted it, and it was never noticed. I helped write CNN's current video ingest system in Erlang. It's been working without issue for years, despite no maintenance or attention (to where even most of the developers have left, but it still just...works). Even much less complex Ruby, Java, and Javascript systems are plagued with constant bugs. I would not say the devs on this project were just that much better (though the process was a little different, with little product owner involvement), but that the language we picked was so much better geared toward fault tolerance.

nickjj 9y ago

Maybe I'm thinking about this incorrectly but when it comes to web application development and concurrency, the things I would typically want to run in a separate process are very important tasks. For example, let's say you're sending emails out. In Rails, Flask, etc. you would typically offload this to a background worker like Sidekiq or Celery. These are dedicated libraries that run in their own process and handle job processing. Both tools allow you to see if that job was successful or failed and deals with retries. They also use Redis as a back-end so your state persists outside of the application code. If you just willynilly spawn a process and start sending stuff through it, how do you ensure everything works as expected and what happens to all of your state if the Erlang VM goes down? I love the idea of doing this, but in real world practice, it sounds like you would still need the Elixir equiv. of Sidekiq / Celery to handle ensuring these spawned tasks are trackable, right?

narrowtux 9y ago
Of course you could make such a system in Elixir, and I'd always recommend doing things within the erlang VM (beam) instead of figuring out how to deploy another service (like install sidekiq with all its dependencies). However, processes in Erlang/Elixir are used ubiquitously. Most of the time, they run forever like some sort of service and are restarted when they crash. Sometimes, you won't care about the result. That's the reason why features such as tracking success/failure in a persistent data store is not a core feature. You could easily add those features to your processes though, the necessary API is already there. The alternative is to use message queues that are durable. Only ack the message when you successfully handled it. If your application crashes before the task was completed, the task will be redelivered the next time the application starts.
- nickjj 9y ago
  I definitely wouldn't want to be responsible for writing this code myself. Something like Sidekiq has been worked on for 5+ years by hundreds of people. Does anything exist in the Elixir world that's as battle hardened and has comparable features to Sidekiq / Celery?