4 ms·

Maybe I'm thinking about this incorrectly but when it comes to web application development and concurrency, the things I would typically want to run in a separa

by nickjj 9y ago

Maybe I'm thinking about this incorrectly but when it comes to web application development and concurrency, the things I would typically want to run in a separate process are very important tasks. For example, let's say you're sending emails out. In Rails, Flask, etc. you would typically offload this to a background worker like Sidekiq or Celery. These are dedicated libraries that run in their own process and handle job processing. Both tools allow you to see if that job was successful or failed and deals with retries. They also use Redis as a back-end so your state persists outside of the application code. If you just willynilly spawn a process and start sending stuff through it, how do you ensure everything works as expected and what happens to all of your state if the Erlang VM goes down? I love the idea of doing this, but in real world practice, it sounds like you would still need the Elixir equiv. of Sidekiq / Celery to handle ensuring these spawned tasks are trackable, right?

narrowtux 9y ago
Of course you could make such a system in Elixir, and I'd always recommend doing things within the erlang VM (beam) instead of figuring out how to deploy another service (like install sidekiq with all its dependencies). However, processes in Erlang/Elixir are used ubiquitously. Most of the time, they run forever like some sort of service and are restarted when they crash. Sometimes, you won't care about the result. That's the reason why features such as tracking success/failure in a persistent data store is not a core feature. You could easily add those features to your processes though, the necessary API is already there. The alternative is to use message queues that are durable. Only ack the message when you successfully handled it. If your application crashes before the task was completed, the task will be redelivered the next time the application starts.
- nickjj 9y ago
  I definitely wouldn't want to be responsible for writing this code myself. Something like Sidekiq has been worked on for 5+ years by hundreds of people. Does anything exist in the Elixir world that's as battle hardened and has comparable features to Sidekiq / Celery?
  - brightball 9y ago
    The BEAM itself. Erlang predates Linux. When you're coming from Rails/Sidekiq world a process can seem very fragile because you're usually thinking of it like a thread or a Sidekiq job. With a Sidekiq job, the jobs are kept in Redis and what's actively being worked on is tracked by Sidekiq in Redis. Even if you switch to a different store like Mongo for job details, you still have to have Redis available just for the progress tracking. This background job will usually run through and execute a lot of sequential steps. When writing a Sidekiq job you are supposed to build it so that if it fails at some point along the way, the job can run again. Failures will trigger retries or exhaustion failure cases and log the error out to Redis for your Sidekiq dashboard so that you can do something different with it at that point. The job is running in it's own thread which doesn't have a clean tracking mechanism via Sidekiq, so the only option to retry happens by re-queuing the job. Your database connection pool has to be increase to match your concurrency level in most cases as well and the concurrency has to be closely watched to track resource usage by job type in multiple job queues to ensure that too many heavy jobs don't take over and prevent other things like emails or push notifications from getting out on time. That's with everything that's built in plus a few extensions. Now, compare that to how processes work with Elixir/Erlang. Processes are so small (0.5kb compared with at least 1mb / thread) that each process is created with a supervisor as a standard mode of operation. Each job has it's own personal supervisor that will see if an error happened and immediately restart the job (with backoff). No external dependency, re-queuing or special tracking needed (although there are plenty of options). Because the processes are so small, in most cases you won't have a single large background job, you'll have many smaller ones breaking the entire thing into smaller pieces with other supervisors so at the point of an error, only that piece will need to restart. Worrying about different job sizes or resource usage largely goes out the window as well because the BEAM's scheduler limits the execution time of every process ensuring that a single large job can't takeover a machine. All of the smaller ones will keep executing just as if the big one wasn't running. Because the entire thing is built for concurrency, you also don't have to worry about the database-connection-per-job as the connection will just be checked out to make a query and then checked right back in. Because the scheduler could change context at any point, allowing a single process to hold a connection for longer than the database transaction in progress doesn't make sense. If you really want to use the Sidekiq approach, there is a library for it available (https://github.com/akira/exq https://github.com/akira/exq). But what you get out of the box with Elixir/BEAM is already pretty far ahead of what Sidekiq gives you.
sbuttgereit 9y ago
I don't think so. First, just to make sure there aren't any points of confusion... Erlang/Elixir processes are not OS processes (https://elixir-lang.github.io/getting-started/processes.html https://elixir-lang.github.io/getting-started/processes.html). As such, we really can't speak of processes in Erlang/Elixir and other kinds of systems on any sort of equal footing. In Erlang, processes are much lighter weight than OS processes. And you do expect them to go wrong... so Erlang has the idea of "Supervision Tree" where there are Supervisor processes whose job it is to monitor other processes and to manage their failure when it occurs. There can be multiple Supervisors which influence one another, or not, as you design them too (thus the "tree" bit). Naturally you plan for these sorts of failures in designing what is/is not a process, what dependencies there may be, what any Supervisor watches and how the Supervisors relate to one another. Erlang/Elixir also seems to have one of the strongest availability and concurrency stories out there on its own. Part of this comes from the aforementioned built in assumptions around isolating and managing failed processes, but the other is in terms of the relative ease of starting processes as needed, and accessing the processes of other Erlang VMs (same server or not). It's clear you have to architect and build things correctly to take full advantage of these capabilities, but I think you can get to the place you describe without necessarily taking on a bundle of different applications for different purposes. For example, for many of the use cases I think of Redis, Erlang offers a number of different ways to deal with such task out of the box: https://blog.codeship.com/elixir-ets-vs-redis/ https://blog.codeship.com/elixir-ets-vs-redis/ Again, you have to think about how you architect things, but you can use the same toolbox for much of the work. I'm still new to Elixir/Erlang, but I've been studying capabilities and architecture for several months now. I think there are sufficient differences in Erlang's approach as compared to other more common stacks that it really does benefit one to come at it fresh and not make very many comparative assumptions. In my own learning I've found that to just get the basic feel for what this Erlang/Elixir animal really is, that the Elixir side is more approachable. The documentation tends to be a little more direct and tends to be more beginner friendly. After I got the very basics down, I was able to look at the Erlang side and understand really what they were trying to say.
- sbuttgereit 9y ago
  FYI... I think that https://news.ycombinator.com/item?id=15952005 https://news.ycombinator.com/item?id=15952005 give a more complete and rounded answer in many ways, definitely worth a look.

by nickjj 9y ago

narrowtux 9y ago

Of course you could make such a system in Elixir, and I'd always recommend doing things within the erlang VM (beam) instead of figuring out how to deploy another service (like install sidekiq with all its dependencies). However, processes in Erlang/Elixir are used ubiquitously. Most of the time, they run forever like some sort of service and are restarted when they crash. Sometimes, you won't care about the result. That's the reason why features such as tracking success/failure in a persistent data store is not a core feature. You could easily add those features to your processes though, the necessary API is already there. The alternative is to use message queues that are durable. Only ack the message when you successfully handled it. If your application crashes before the task was completed, the task will be redelivered the next time the application starts.

nickjj 9y ago
I definitely wouldn't want to be responsible for writing this code myself. Something like Sidekiq has been worked on for 5+ years by hundreds of people. Does anything exist in the Elixir world that's as battle hardened and has comparable features to Sidekiq / Celery?
- brightball 9y ago
  The BEAM itself. Erlang predates Linux. When you're coming from Rails/Sidekiq world a process can seem very fragile because you're usually thinking of it like a thread or a Sidekiq job. With a Sidekiq job, the jobs are kept in Redis and what's actively being worked on is tracked by Sidekiq in Redis. Even if you switch to a different store like Mongo for job details, you still have to have Redis available just for the progress tracking. This background job will usually run through and execute a lot of sequential steps. When writing a Sidekiq job you are supposed to build it so that if it fails at some point along the way, the job can run again. Failures will trigger retries or exhaustion failure cases and log the error out to Redis for your Sidekiq dashboard so that you can do something different with it at that point. The job is running in it's own thread which doesn't have a clean tracking mechanism via Sidekiq, so the only option to retry happens by re-queuing the job. Your database connection pool has to be increase to match your concurrency level in most cases as well and the concurrency has to be closely watched to track resource usage by job type in multiple job queues to ensure that too many heavy jobs don't take over and prevent other things like emails or push notifications from getting out on time. That's with everything that's built in plus a few extensions. Now, compare that to how processes work with Elixir/Erlang. Processes are so small (0.5kb compared with at least 1mb / thread) that each process is created with a supervisor as a standard mode of operation. Each job has it's own personal supervisor that will see if an error happened and immediately restart the job (with backoff). No external dependency, re-queuing or special tracking needed (although there are plenty of options). Because the processes are so small, in most cases you won't have a single large background job, you'll have many smaller ones breaking the entire thing into smaller pieces with other supervisors so at the point of an error, only that piece will need to restart. Worrying about different job sizes or resource usage largely goes out the window as well because the BEAM's scheduler limits the execution time of every process ensuring that a single large job can't takeover a machine. All of the smaller ones will keep executing just as if the big one wasn't running. Because the entire thing is built for concurrency, you also don't have to worry about the database-connection-per-job as the connection will just be checked out to make a query and then checked right back in. Because the scheduler could change context at any point, allowing a single process to hold a connection for longer than the database transaction in progress doesn't make sense. If you really want to use the Sidekiq approach, there is a library for it available (https://github.com/akira/exq https://github.com/akira/exq). But what you get out of the box with Elixir/BEAM is already pretty far ahead of what Sidekiq gives you.

sbuttgereit 9y ago

I don't think so. First, just to make sure there aren't any points of confusion... Erlang/Elixir processes are not OS processes (https://elixir-lang.github.io/getting-started/processes.html https://elixir-lang.github.io/getting-started/processes.html). As such, we really can't speak of processes in Erlang/Elixir and other kinds of systems on any sort of equal footing. In Erlang, processes are much lighter weight than OS processes. And you do expect them to go wrong... so Erlang has the idea of "Supervision Tree" where there are Supervisor processes whose job it is to monitor other processes and to manage their failure when it occurs. There can be multiple Supervisors which influence one another, or not, as you design them too (thus the "tree" bit). Naturally you plan for these sorts of failures in designing what is/is not a process, what dependencies there may be, what any Supervisor watches and how the Supervisors relate to one another. Erlang/Elixir also seems to have one of the strongest availability and concurrency stories out there on its own. Part of this comes from the aforementioned built in assumptions around isolating and managing failed processes, but the other is in terms of the relative ease of starting processes as needed, and accessing the processes of other Erlang VMs (same server or not). It's clear you have to architect and build things correctly to take full advantage of these capabilities, but I think you can get to the place you describe without necessarily taking on a bundle of different applications for different purposes. For example, for many of the use cases I think of Redis, Erlang offers a number of different ways to deal with such task out of the box: https://blog.codeship.com/elixir-ets-vs-redis/ https://blog.codeship.com/elixir-ets-vs-redis/ Again, you have to think about how you architect things, but you can use the same toolbox for much of the work. I'm still new to Elixir/Erlang, but I've been studying capabilities and architecture for several months now. I think there are sufficient differences in Erlang's approach as compared to other more common stacks that it really does benefit one to come at it fresh and not make very many comparative assumptions. In my own learning I've found that to just get the basic feel for what this Erlang/Elixir animal really is, that the Elixir side is more approachable. The documentation tends to be a little more direct and tends to be more beginner friendly. After I got the very basics down, I was able to look at the Erlang side and understand really what they were trying to say.

sbuttgereit 9y ago
FYI... I think that https://news.ycombinator.com/item?id=15952005 https://news.ycombinator.com/item?id=15952005 give a more complete and rounded answer in many ways, definitely worth a look.