4 ms·

You are right that an external queuing system might be useful in some cases. I was thinking of simple background jobs like sending emails while processing an HT

You are right that an external queuing system might be useful in some cases. I was thinking of simple background jobs like sending emails while processing an HTTP request. Once persistence is available, why would this be any less resilient than an external queue? IIRC, this is the standard way of dealing with background jobs in Erlang/Elixir projects . Ractor should bring similar capability to Ruby ecosystem.

vidarh 5y ago
Depends what you mean by "persistence". If you store it to a replicated database, then sure, it's just as resilient, but that is an external queue. If you store it to a replicated filesystem, then sure, it's just as resilient, but almost nobody runs a setup like that (I have; I ran GlusterFS in production without downtime for a decade - it was a great piece of software, but for this kind of usage pattern it'd kill performance and just stuffing it in a database would be operationally easier and much faster). But if you just store it to a local disk, then if the individual server fails your queue fails, that is why it would usually be less resilient than a properly set up external queue based on a replication storage mechanism. Note that you certainly can set up external queues that are just as, or even more, brittle. But at least the more serious queueing systems at least have options that will allow you to make the queue resilient. Note that all of the above takes away the advantage of "just" having a single process deployment unless you already have that infrastructure set up. I'd much rather set up a database + replica than GlusterFS for example (GlusterFS is great, but it's a pain to manage compared to e.g. just Postgres or a Redis cluster to get resilience for Sidekiq) > Ractor should bring similar capability to Ruby ecosystem. Ractor is 100% orthogonal to that. I did what you describe first time in Ruby in 2005 with processes. It took a few dozen lines of code + sqlite (run one thread to stuff new jobs into sqlite; fork as many times as you need workers, and use a pipe to tell them which job to pick up, or serialize the whole job). What Ractor brings is more fine-grained isolation (more sharing than a separate process, less than Thread), and running Ruby code on multiple CPU cores without resorting to fork(). That's all it's trying to do. You could implement what I described above slightly cleaner with Ractor and wrap it up into a gem, but form vs. Ractor is unlikely to make a big difference for that kind of scenario, because the overheads of the sharing is dwarfed by the disk io. This may matter more for other Ruby implementations than MRI.

vidarh 5y ago

Depends what you mean by "persistence". If you store it to a replicated database, then sure, it's just as resilient, but that is an external queue. If you store it to a replicated filesystem, then sure, it's just as resilient, but almost nobody runs a setup like that (I have; I ran GlusterFS in production without downtime for a decade - it was a great piece of software, but for this kind of usage pattern it'd kill performance and just stuffing it in a database would be operationally easier and much faster). But if you just store it to a local disk, then if the individual server fails your queue fails, that is why it would usually be less resilient than a properly set up external queue based on a replication storage mechanism. Note that you certainly can set up external queues that are just as, or even more, brittle. But at least the more serious queueing systems at least have options that will allow you to make the queue resilient. Note that all of the above takes away the advantage of "just" having a single process deployment unless you already have that infrastructure set up. I'd much rather set up a database + replica than GlusterFS for example (GlusterFS is great, but it's a pain to manage compared to e.g. just Postgres or a Redis cluster to get resilience for Sidekiq) > Ractor should bring similar capability to Ruby ecosystem. Ractor is 100% orthogonal to that. I did what you describe first time in Ruby in 2005 with processes. It took a few dozen lines of code + sqlite (run one thread to stuff new jobs into sqlite; fork as many times as you need workers, and use a pipe to tell them which job to pick up, or serialize the whole job). What Ractor brings is more fine-grained isolation (more sharing than a separate process, less than Thread), and running Ruby code on multiple CPU cores without resorting to fork(). That's all it's trying to do. You could implement what I described above slightly cleaner with Ractor and wrap it up into a gem, but form vs. Ractor is unlikely to make a big difference for that kind of scenario, because the overheads of the sharing is dwarfed by the disk io. This may matter more for other Ruby implementations than MRI.