3 ms·

How do you process a million jobs in parallel across many machines? How do you schedule a job to run in a week? I'm the author of Sidekiq and not trying to de

by mperham 10y ago

How do you process a million jobs in parallel across many machines? How do you schedule a job to run in a week? I'm the author of Sidekiq and not trying to debate; I'm simply ignorant of how Erlang, BEAM and the ecosystem solve these common problems, which usually persist no matter which language you use.

querulous 10y ago
(caveat: in practice you'd do this differently, making sure applications were prestarted/preregistered on remote nodes, ensuring your network was already connected and wrapping most of these calls in nicer function calls but for the sake of illustration i've described the naive way to do this) %% ping a remote node to connect to it pong = net_adm:ping('foo@127.0.0.1'), %% start my application on the remote node _ = spawn('foo@127.0.0.1', application, start, [my_app]), %% my_app is a gen_server, an otp process that abstracts %% async request/response and makes it look more like %% synchronous function calls %% this example assumes the gen_server accepts the JobArgs %% and immediately returns 'ok' as acknowledgement, but does %% the work asynchronously ok = gen_server:call({my_app, 'foo@127.0.0.1'}, {do_some_work, JobArgs}) if you wanted to distribute a number of jobs over a number of nodes you'd just repeat this process for each node and distribute your jobs however you choose, either by round robin or something more complicated like rendezvous hashing or something this example assumes you're uninterested in the responses, but if you wanted to receive the results you could either send a pid (process identifier) along with the request that would be setup to collect the results and do any further processing or you could spawn a local process that made the actual call and blocked until it received the results as for scheduling a job a week in the future, the naive way would be to setup a timer that, when it expires, spawns a process that can execute an arbitrary function. no one would do this, normally, however. you'd probably want to use something like sidekiq that can store the job off node and poll regularly to see if there's any jobs you should run
biokoda 10y ago
> How do you process a million jobs in parallel across many machines? Start the processes that execute the code, get notified when they are finished. Literally a couple of lines of Erlang/Elixir. Not some large framework that does all that for you, it's in the language. It is built to run a lot of tasks in parallel and across many machines. > How do you schedule a job to run in a week? Like anyone else, put it in some permanent storage like redis. You can also keep it in memory, you can also use mnesia which is a distributed database that comes with Erlang.
pdimitar 10y ago
That's a really good question and I'd be delighted if somebody answers it adequately. I heard that Erlang/OTP (and thus Elixir) can transparently distribute work no matter where it gets physically executed but I'd just love to see an example -- freely accessible code, demo site, everything.
sbrother 10y ago
For your first question, I use the toniq library: https://github.com/joakimk/toniq https://github.com/joakimk/toniq. As biokoda mentioned, there's nothing stopping you from doing this by literally just spawning a ton of processes, but you do get a lot of nice things out of the box with toniq or another task queue framework. For example, toniq gives you automatic task retrying with exponential backoff, a max-concurrency option, and task backup with redis. It doesn't have a scheduled job API yet, but there is a note in the readme that explains how to do this with quantum-elixir: https://github.com/joakimk/toniq#how-do-i-run-scheduled-or-recurring-jobs https://github.com/joakimk/toniq#how-do-i-run-scheduled-or-r....
di4na 10y ago
Let's go for that one. Processing a million jobs in parallel is not hard my_list_of_job |> distribute_tasks() |> Enum.map(fn x -> Task.Supervisor.async(x, my_list_of_job) end) |> Enum.map(fn x -> Task.await(x, your_timeout) end) That is will distribute your worload on all the nodes you have that implement a runner for that. Of course you may want to make things a bit different, and i did not covered your load balancing that i hidden in distribute task, nor any error handling, but that would not be far harder imho. Until you enter the partition and idempotent part, but even that would not be that hard. There is a long running project to work on that to produce a nice job queue lib on Elixir Slack but noone on the interested group have time for now. To schedule a job to run in a week you have a couple way. An easy first approach is to use Process.send_after/3 to send yourself a message after a week and handle that message in a week. Of course if you want persistence, etc, you will have to do it a bit differently, but not so much.... edit : now use the fn form instead of the shorthand & for ease of comprehension for non seasoned Elixir devs

by mperham 10y ago

querulous 10y ago

(caveat: in practice you'd do this differently, making sure applications were prestarted/preregistered on remote nodes, ensuring your network was already connected and wrapping most of these calls in nicer function calls but for the sake of illustration i've described the naive way to do this) %% ping a remote node to connect to it pong = net_adm:ping('foo@127.0.0.1'), %% start my application on the remote node _ = spawn('foo@127.0.0.1', application, start, [my_app]), %% my_app is a gen_server, an otp process that abstracts %% async request/response and makes it look more like %% synchronous function calls %% this example assumes the gen_server accepts the JobArgs %% and immediately returns 'ok' as acknowledgement, but does %% the work asynchronously ok = gen_server:call({my_app, 'foo@127.0.0.1'}, {do_some_work, JobArgs}) if you wanted to distribute a number of jobs over a number of nodes you'd just repeat this process for each node and distribute your jobs however you choose, either by round robin or something more complicated like rendezvous hashing or something this example assumes you're uninterested in the responses, but if you wanted to receive the results you could either send a pid (process identifier) along with the request that would be setup to collect the results and do any further processing or you could spawn a local process that made the actual call and blocked until it received the results as for scheduling a job a week in the future, the naive way would be to setup a timer that, when it expires, spawns a process that can execute an arbitrary function. no one would do this, normally, however. you'd probably want to use something like sidekiq that can store the job off node and poll regularly to see if there's any jobs you should run

biokoda 10y ago

> How do you process a million jobs in parallel across many machines? Start the processes that execute the code, get notified when they are finished. Literally a couple of lines of Erlang/Elixir. Not some large framework that does all that for you, it's in the language. It is built to run a lot of tasks in parallel and across many machines. > How do you schedule a job to run in a week? Like anyone else, put it in some permanent storage like redis. You can also keep it in memory, you can also use mnesia which is a distributed database that comes with Erlang.

pdimitar 10y ago

That's a really good question and I'd be delighted if somebody answers it adequately. I heard that Erlang/OTP (and thus Elixir) can transparently distribute work no matter where it gets physically executed but I'd just love to see an example -- freely accessible code, demo site, everything.

sbrother 10y ago

For your first question, I use the toniq library: https://github.com/joakimk/toniq https://github.com/joakimk/toniq. As biokoda mentioned, there's nothing stopping you from doing this by literally just spawning a ton of processes, but you do get a lot of nice things out of the box with toniq or another task queue framework. For example, toniq gives you automatic task retrying with exponential backoff, a max-concurrency option, and task backup with redis. It doesn't have a scheduled job API yet, but there is a note in the readme that explains how to do this with quantum-elixir: https://github.com/joakimk/toniq#how-do-i-run-scheduled-or-recurring-jobs https://github.com/joakimk/toniq#how-do-i-run-scheduled-or-r....

di4na 10y ago

Let's go for that one. Processing a million jobs in parallel is not hard my_list_of_job |> distribute_tasks() |> Enum.map(fn x -> Task.Supervisor.async(x, my_list_of_job) end) |> Enum.map(fn x -> Task.await(x, your_timeout) end) That is will distribute your worload on all the nodes you have that implement a runner for that. Of course you may want to make things a bit different, and i did not covered your load balancing that i hidden in distribute task, nor any error handling, but that would not be far harder imho. Until you enter the partition and idempotent part, but even that would not be that hard. There is a long running project to work on that to produce a nice job queue lib on Elixir Slack but noone on the interested group have time for now. To schedule a job to run in a week you have a couple way. An easy first approach is to use Process.send_after/3 to send yourself a message after a week and handle that message in a week. Of course if you want persistence, etc, you will have to do it a bit differently, but not so much.... edit : now use the fn form instead of the shorthand & for ease of comprehension for non seasoned Elixir devs