3 ms·

Why couldn't parallel execution in single process be combined with persistence? Store the job data, the continue execution of the job and at the end, clean up o

Why couldn't parallel execution in single process be combined with persistence? Store the job data, the continue execution of the job and at the end, clean up or mark it completed. One gets both persistence as well as an easy, single-process deployment.

vidarh 5y ago
Nothing stops you from doing that, but you're then in a relatively small niche of processing messages that you want persistence for, but where persistence is not important enough to put it in a more resilient external queuing system. There's absolutely apps that fit in that niche. E.g. somewhat time-consuming processing that fit on a single machine and that can still be re-run from scratch if a queue is lost, can be a use-case for it. I've built crawlers where some of the queues fit that model, for example - e.g. things we'd re-crawl 2-3 times a day, so worst case if a queue went and some things wouldn't update for a few hours.
- mathnmusic 5y ago
  You are right that an external queuing system might be useful in some cases. I was thinking of simple background jobs like sending emails while processing an HTTP request. Once persistence is available, why would this be any less resilient than an external queue? IIRC, this is the standard way of dealing with background jobs in Erlang/Elixir projects . Ractor should bring similar capability to Ruby ecosystem.
  - vidarh 5y ago
    Depends what you mean by "persistence". If you store it to a replicated database, then sure, it's just as resilient, but that is an external queue. If you store it to a replicated filesystem, then sure, it's just as resilient, but almost nobody runs a setup like that (I have; I ran GlusterFS in production without downtime for a decade - it was a great piece of software, but for this kind of usage pattern it'd kill performance and just stuffing it in a database would be operationally easier and much faster). But if you just store it to a local disk, then if the individual server fails your queue fails, that is why it would usually be less resilient than a properly set up external queue based on a replication storage mechanism. Note that you certainly can set up external queues that are just as, or even more, brittle. But at least the more serious queueing systems at least have options that will allow you to make the queue resilient. Note that all of the above takes away the advantage of "just" having a single process deployment unless you already have that infrastructure set up. I'd much rather set up a database + replica than GlusterFS for example (GlusterFS is great, but it's a pain to manage compared to e.g. just Postgres or a Redis cluster to get resilience for Sidekiq) > Ractor should bring similar capability to Ruby ecosystem. Ractor is 100% orthogonal to that. I did what you describe first time in Ruby in 2005 with processes. It took a few dozen lines of code + sqlite (run one thread to stuff new jobs into sqlite; fork as many times as you need workers, and use a pipe to tell them which job to pick up, or serialize the whole job). What Ractor brings is more fine-grained isolation (more sharing than a separate process, less than Thread), and running Ruby code on multiple CPU cores without resorting to fork(). That's all it's trying to do. You could implement what I described above slightly cleaner with Ractor and wrap it up into a gem, but form vs. Ractor is unlikely to make a big difference for that kind of scenario, because the overheads of the sharing is dwarfed by the disk io. This may matter more for other Ruby implementations than MRI.

vidarh 5y ago

Nothing stops you from doing that, but you're then in a relatively small niche of processing messages that you want persistence for, but where persistence is not important enough to put it in a more resilient external queuing system. There's absolutely apps that fit in that niche. E.g. somewhat time-consuming processing that fit on a single machine and that can still be re-run from scratch if a queue is lost, can be a use-case for it. I've built crawlers where some of the queues fit that model, for example - e.g. things we'd re-crawl 2-3 times a day, so worst case if a queue went and some things wouldn't update for a few hours.

mathnmusic 5y ago
You are right that an external queuing system might be useful in some cases. I was thinking of simple background jobs like sending emails while processing an HTTP request. Once persistence is available, why would this be any less resilient than an external queue? IIRC, this is the standard way of dealing with background jobs in Erlang/Elixir projects . Ractor should bring similar capability to Ruby ecosystem.
- vidarh 5y ago
  Depends what you mean by "persistence". If you store it to a replicated database, then sure, it's just as resilient, but that is an external queue. If you store it to a replicated filesystem, then sure, it's just as resilient, but almost nobody runs a setup like that (I have; I ran GlusterFS in production without downtime for a decade - it was a great piece of software, but for this kind of usage pattern it'd kill performance and just stuffing it in a database would be operationally easier and much faster). But if you just store it to a local disk, then if the individual server fails your queue fails, that is why it would usually be less resilient than a properly set up external queue based on a replication storage mechanism. Note that you certainly can set up external queues that are just as, or even more, brittle. But at least the more serious queueing systems at least have options that will allow you to make the queue resilient. Note that all of the above takes away the advantage of "just" having a single process deployment unless you already have that infrastructure set up. I'd much rather set up a database + replica than GlusterFS for example (GlusterFS is great, but it's a pain to manage compared to e.g. just Postgres or a Redis cluster to get resilience for Sidekiq) > Ractor should bring similar capability to Ruby ecosystem. Ractor is 100% orthogonal to that. I did what you describe first time in Ruby in 2005 with processes. It took a few dozen lines of code + sqlite (run one thread to stuff new jobs into sqlite; fork as many times as you need workers, and use a pipe to tell them which job to pick up, or serialize the whole job). What Ractor brings is more fine-grained isolation (more sharing than a separate process, less than Thread), and running Ruby code on multiple CPU cores without resorting to fork(). That's all it's trying to do. You could implement what I described above slightly cleaner with Ractor and wrap it up into a gem, but form vs. Ractor is unlikely to make a big difference for that kind of scenario, because the overheads of the sharing is dwarfed by the disk io. This may matter more for other Ruby implementations than MRI.