5 ms·
Until you hit scale, the database you're already using is fine. If that's Postgres, look up SELECT FOR UPDATE SKIP LOCKED. The major convenience here - aside fr
by stickfigure 2y ago
Until you hit scale, the database you're already using is fine. If that's Postgres, look up SELECT FOR UPDATE SKIP LOCKED. The major convenience here - aside from operational simplicity - is transactional task enqueueing.
For hosted, SQS or Google Cloud Tasks. Google's approach is push-based (as opposed to pull-based) and is far and above easier to use than any other queueing system.
- vrosas 2y agoCloud Tasks is one of the most undervalued tools in the GCP ecosystem, but mostly because PubSub gets all the attention. I've been using it since it was baked in the AppEngine and love it for 1-to-1 queues or delayed job handling.
- kamikaz1k 2y agohow do you recommend working with Cloud Tasks? raw dogging gcloud? Terraform? or something more manageable? I've been curious for one of my smaller projects, but I am worried about adopting more GCPisms.
- vrosas 2y agoTerraform is definitely for the best. Any AI tool should be able to spit it out well enough, but if you do rawdog it in the console or gcloud you might be able to export the terraform with: gcloud beta resource-config bulk-export --resource-format=terraform
- stickfigure 2y agoBack when I was all-in on GCP, I had a queue.yaml file which the appengine deployer syncs to cloud tasks (creates/disabled queues, changes the rate limits, concurrency, etc). Now that I'm mostly on AWS... I still use the same system. I have a thin little project that deploys to GAE and has a queue.yaml file. It sets up the cloud tasks queues. They hit my EB endpoints just like they used to hit my GAE endpoints. As a bonus, my thin little GAE app also has a cron.yaml that it proxies to my AWS app. Appengine's cron is also better than Amazon's overcomplicated eventbridge system. It's great.
- monksy 2y agoFamious last words. There are database as a queue antipattern warnings about this.
- reval 2y agoWhy is that an anti-pattern? Databases have added `SKIP LOCKED` and `SELECT FOR UPDATE` to handle these use cases. What are the downsides?
- deleted 2y ago[deleted]
- maniacalhack0r 2y agoas with everything, it depends on how you're processing the queue. eg we built a system at my last company to process 150 million objects / hour, and we modeled this using a postgres-backed queue with multiple processes pulling from the queue. we observed that, whenever there were a lot of locked rows (ie lots of work being done), Postgres would correctly SKIP these rows, but having to iterate over and skip that many locked rows did have a noticeable impact on CPU utilization. we worked around this by partitioning the queue, indexing on partition, and assigning each worker process a partition to pull from upon startup. this reduced the # of locked rows that postgres would have to skip over because our queries would contain a `WHERE partition=X` clause. i had some great graphs on how long `SELECT FOR UPDATE ... SKIP LOCKED` takes as the number of locked rows in the queue increases, and how this partiton work around reduced the time to execute the SKIP LOCKED query, but unfortunately they are in the hands of my previous employer :(
- ninju 2y agoI believe the article and parent comment were discussing queue solutions for low-volume situations.
- maniacalhack0r 2y agocompletely missed this. apologies.
- crabbone 2y agoI'm probably biased, but in the number of cases where I had to work with Kafka, I'd really prefer to simply have an SQL database. In all of those cases I struggled to understand why developers wanted Kafka, what problem was it solving better than the database they already had, and for the life of me, there just wasn't one. I'm not saying that configuring and deploying databases is easy, but it's probably going to happen anyway. Deploying and configuring Kafka is a huge headache: bad documentation, no testing tools, no way to really understand performance in the light of durability guarantees (which are also obscured by the poor quality documentation). It's just an honestly bad product (from the infra perspective): poor UX, poor design... and worst of all, it's kind of useless from the developer standpoint. Not 100% useless, but whatever it offers can be replaced by other existing tools with a tiny bit of work.
- boruto 2y agoHow could I solve the problem of in-order processing based on a key using skip locked? Basically all records having the key to be processed one after other.
- wmfiv 2y agoWork jobs in the order they were submitted within a partition key. This selects the next partition key that isn't locked. You could make it smarter to select a subset of the jobs checking for partition keys where all of the rows are still unlocked. SELECT * FROM jobs WHERE partition_key = ( SELECT partition_key FROM jobs ORDER BY partition_key LIMIT 1 SKIP LOCKED ) ORDER BY submitted_at FOR UPDATE SKIP LOCKED;
- boruto 2y agoYes, something along the lines could work. But I am not sure if the above query itself would work if rows are appended to the table in parallel. Also if events for a partition gets processed quick would the last partition get an equal chance?