3 ms·

Isn't celery plug and play nowadays? If you need a something simpler take a look at django-rq. I don't think a serious job queue is possbile with RDBMS as backe

Isn't celery plug and play nowadays? If you need a something simpler take a look at django-rq. I don't think a serious job queue is possbile with RDBMS as backend.

mrfusion 11y ago
What's so special about a job queue that it can't use an rdbms? I never understood that.
- anentropic 11y ago
  are you asking why RabbitMQ exists?
  - LaurensBER 11y ago
    Yeah, or any documentation that explains the fundamental differences. I get that it's required and I can see the advantages for large scale operations but I never understood why you can run a simpel DB backend for small scale setup. Just trying to learn here!
    - fleetfox 11y ago
      You can use relation database but sane implementation is ugly and doesn't scale. There is little reason to choose bunch complexity and unpredictable behavior over dedicated queue dependency. Sorry, i don't have any good material on the topic. Maybe you can use this to see how RDBMS implementation looks and what are the consequences. https://brandur.org/postgres-queues https://brandur.org/postgres-queues
- rajivm 11y ago
  One reason: Most RDBMS's aren't designed for "waiting for newly queued items". You literally have to have each worker poll at some time interval for new items: `SELECT * FROM jobs WHERE status = 'new'`, transactionally claim the job, and if a worker dies, un-claim it. It's not that it's not possible to use an RDBMS and work around these problems, but generally, it won't lead to the most scalable and robust solution for the problem. Message brokers / AMQP are better designed for this problem set. If you have a small site, with a low volume queue, your RDBMS might be just fine though.
  - houzi 11y ago
    This is not the case if you're using Postgres, which will signal/notify listening clients, instead of having the clients poll for changes.
collyw 11y ago
It wasn't (for me) a year ago when I did an upgrade. I basically had to go through the docs and start from scratch. :(

Isn't celery plug and play nowadays? If you need a something simpler take a look at django-rq. I don't think a serious job queue is possbile with RDBMS as backend.

mrfusion 11y ago

What's so special about a job queue that it can't use an rdbms? I never understood that.

anentropic 11y ago
are you asking why RabbitMQ exists?
- LaurensBER 11y ago
  Yeah, or any documentation that explains the fundamental differences. I get that it's required and I can see the advantages for large scale operations but I never understood why you can run a simpel DB backend for small scale setup. Just trying to learn here!
  - fleetfox 11y ago
    You can use relation database but sane implementation is ugly and doesn't scale. There is little reason to choose bunch complexity and unpredictable behavior over dedicated queue dependency. Sorry, i don't have any good material on the topic. Maybe you can use this to see how RDBMS implementation looks and what are the consequences. https://brandur.org/postgres-queues https://brandur.org/postgres-queues
rajivm 11y ago
One reason: Most RDBMS's aren't designed for "waiting for newly queued items". You literally have to have each worker poll at some time interval for new items: `SELECT * FROM jobs WHERE status = 'new'`, transactionally claim the job, and if a worker dies, un-claim it. It's not that it's not possible to use an RDBMS and work around these problems, but generally, it won't lead to the most scalable and robust solution for the problem. Message brokers / AMQP are better designed for this problem set. If you have a small site, with a low volume queue, your RDBMS might be just fine though.
- houzi 11y ago
  This is not the case if you're using Postgres, which will signal/notify listening clients, instead of having the clients poll for changes.

collyw 11y ago

It wasn't (for me) a year ago when I did an upgrade. I basically had to go through the docs and start from scratch. :(