5 ms·
The author of this post, Chris Hanks, created the Que queueing library for Ruby: https://github.com/que-rb/que https://github.com/que-rb/que It’s changed signi
by bgentry 7y ago
The author of this post, Chris Hanks, created the Que queueing library for Ruby: https://github.com/que-rb/que https://github.com/que-rb/que
It’s changed significantly since this post as the 1.x betas use a very different structure which should actually be more efficient, use fewer Postgres connections, cause less lock contention, and cause less table bloat.
Not sure if the benchmarks have been run recently or not but I’m definitely curious how things stack up to this post from 6 years ago :)
- aldoushuxley001 7y agoWish I could use that in a Django app. Doesn't seem to be a viable python queueing library that allows using postgresql
- deleted 7y ago[deleted]
- dimino 7y agorq is pretty good, have you checked it out?
- elamje 7y agoRecently was wishing this myself. I’ve been using rq, which although I hear is much faster than an rdbms is also another dependency and service to worry about failing. Trying to keep things simple is nice, I am already having trouble trying to grok what kind of weird state I can have in my application if Redis goes down, or my dB goes down but Redis remains
- codetrotter 7y ago> I am already having trouble trying to grok what kind of weird state I can have in my application if Redis goes down, or my dB goes down but Redis remains If it is possible to set up a smaller test environment with one or a few instances of your DB and Redis, and you have time to do it, you could do some testing where you purposely shut down each of them at various points in time and inspect what happens to your application state compared to how it behaves when all is well. Perhaps you could even outfit the testing version of your application with two proxies, one that will proxy the DB connection and one that will proxy the Redis connection, and have these proxies randomly decide within some threshold whether to pass the data along to their upstreams, or to simulate a broken connection. Then test the application with some different threshold values, for example, 1% probability of failure, 5% probability of failure, 50% probability of failure and 100% probability of failure. Repeat the tests some number of times for each threshold value and observe the behavior of the application each time. Also, make sure that you log the decisions made by the proxies each run, so that you are able to look at these afterwards.
- subleq 7y agoI wrote django-postgres-queue for this purpose. It uses postgres transactions to keep queue and application state in sync. It also uses SKIP LOCKED to avoid some of the typical issues with using a database as a queue. https://github.com/gavinwahl/django-postgres-queue https://github.com/gavinwahl/django-postgres-queue
- elnygren 7y agoThanks for making this, I've been using it to help me send emails, push notifications etc. to 20-30k users in a Django project :)
- aldoushuxley001 7y agoThat actually does look like exactly what I need. Beautiful. Thank you! Great work.
- michelpp 7y agoThere is a Python library that uses SKIP LOCKED https://github.com/malthe/pq https://github.com/malthe/pq
- StavrosK 7y agoYeah, it would be nice if Dramatiq supported it. django-postgres-queue sounds promising, though.
- ukd1 7y agoI actually ran them in July 2019 against QueueClassic using the Que benchmark; queue_classic jobs per second: avg = 1879.1, max = 2072.1, min = 1779.4, stddev = 128.3 que jobs per second: avg = 1500.5, max = 1550.8, min = 1405.2, stddev = 58.1 Full deets: https://github.com/QueueClassic/queue_classic/pull/303#issuecomment-512970655 https://github.com/QueueClassic/queue_classic/pull/303#issue...