6 ms·
If this was part of reducing operational overhead, why not implement something functionally like GCP Cloud Tasks [0]? Since this is part of Rails, all you woul
by latchkey 1y ago
If this was part of reducing operational overhead, why not implement something functionally like GCP Cloud Tasks [0]?
Since this is part of Rails, all you would need to do is implement regular http endpoints, no need for workers/listeners. Submit a "job" to the queue (which itself is just a POST) and the message details: the endpoint and some data to POST to said endpoint.
The queue "server" processes the list of jobs, hits the specified endpoint and when it gets a 200 response, it deletes it. Otherwise, it just keeps retrying.
[0] https://cloud.google.com/tasks/docs https://cloud.google.com/tasks/docs
- pythonaut_16 1y agoWhy would GCP Cloud Tasks be part of the default libraries? If it’s that simple surely a community maintained gem would be fine?
- latchkey 1y agoTo clarify: I'm talking about implementing the core functionality of GCP Cloud Tasks, not using that specific product.
- ryeguy 1y agoHow does this reduce operational overhead? You still need a queue and a worker to dispatch the tasks to your api.
- latchkey 1y agoThe end user doesn't need to implement the queue or the worker. They just need to implement the http api for receiving POST messages. The queue "server" is effectively just a http client that reads messages from the database and sends them off.
- ryeguy 1y agoThat makes sense as a general pattern, but probably less so in the context of rails which is typically a self contained monolith. It would be adding another hop, more indirection, and more complexity. It would introduce new problems like the need to segment your "real" api from your worker api for the purpose of load isolation.
- latchkey 1y agoIf it is a self contained monolith, that's perfect for this. You have a long running process, which scans a database for new work and then posts to localhost.
- ARama 1y agoWe use to do this by making use of the cloudtasker gem. However we've recently just migrated to Sidekiq. We regularly found that the time to enqueue a job with cloud tasks was anywhere from 200-800ms. We often have workflows from users where they need do something that would involve a batch of say 100+ jobs. The time to process each job was minuscule, but actually enqueuing them took a long time. Time to enqueue with Sidekiq / redis was brought down to under 20ms. We can also make use of bulk enqueuing and enqueue all 100+ jobs in one redis call. However when we were first starting out, it was a godsend. Made our infrastructure significantly more easy to manage and not have to worry.
- latchkey 1y agoCloudtasker is a wrapper around the actual GCP Cloud Tasks service, that's not what I'm talking about here. What I'm talking about is the implementation detail of Solid Queue itself.