4 ms·

Why this and not celery? And why Postgres and not rabbitmq or redis?

derefr 5y ago
Re: why Postgres, two common answers: 1. For anyone whose stack is currently a simple three-tier architecture that currently has Postgres and nothing else in the DB tier, adding anything else would incur 100+% operational complexification (e.g. now you have to figure out how to do backups for two stateful components.) Far more than 100%, even, if they get the Postgres DB "for free" as part of a virtual LAMP-stack appliance, or built into the base offering of some PaaS service, and currently don't need to do any ops work as the framework/platform handles their DB's care and feeding for them. (Ideally, such setups would do the same with a built-in MQ as well — but sadly, that's much rarer.) 2. You can be clever with a job queue that’s in your DB, by making “taking the job” and “doing the queries that comprise the job” part of the same atomic transaction, such that a ROLLBACK due to a constraint validation error in "the queries that comprise the job" will also implicitly “put back” the job immediately (even if the worker crashed in response to seeing the error.)
- TameAntelope 5y ago
  Generally yes, adding new tech can complicate your infrastructure, but specifically redis is one of those rare gems that, “just works”. If you use Celery, the time saved dropping Celery will more than cover the operational costs of redis. Really, one of the magic tools of the 21st century.
emptysea 5y ago
Not parent but in my experience celery has a number of long standing bugs / poor defaults, like prefetching tasks so they can get stuck behind long running tasks. Next project id probably try a simplistic project that’s easier to understand like arq / rq.
- aidos 5y ago
  That prefetching is still an issue? I recall it was caused by rabbitmq though? I switched to rq over celery years ago because it was a deal breaker. We have just long running tasks and it didn’t work for us at all. Rq has been great btw.
simonw 5y ago
They address that on this page: https://procrastinate.readthedocs.io/en/stable/discussions.html https://procrastinate.readthedocs.io/en/stable/discussions.h...
41b696ef1113 5y ago
While I respect Celery, it is complicated (some amount is intrinsic to the problem space). I used to use it, but it felt heavyweight and debugging errors was so quite challenging. Have since switched to Dramatiq - which may not be web scale, but works fine for an internal service. I love the idea of re-using infrastructure "for free".
- ungawatkt 5y ago
  +1 for dramatiq over celery, dramatiq was much easier to understand and extend, even if it's not as "complete". It handled most anything I threw at it for an internal task runner system with some custom scheduling logic. I'd give it a flask out of 10 compared to celery being a Django.
cultofmetatron 5y ago
> why Postgres and not rabbitmq or redis? I chose postrges for our mq stack over rabbitmq and redis. Since our team is small, we are trying to be efficient in our use of manpower. our architecture is as dirt simple as it gets. (loadbalancer -> api-server-nodes -> aws aurora). adding another external dependency just adds one more thing that can potentially go wrong. Thats one more thing we need to watch and maintain accross staging and production envrionemnts and one more thingto get running on developer machines. There may come a day where we switch over to rabbitmq or kafka but postgres has proven to be fast enough. when writes are too much of a load, we can shard the writes to its own dedicated machine and buy more time. When our traffic is sufficiently high that even that isn't enough, we'll have enough revenue to pay someone to configure kafka/rabbitmq and deal with the configuration and maintenance fulltime. Until then, if you want to survive as a startup, KEEP IT SIMPLE. Any complexity you adopt needs to justify itself by providing a competitive advantage.
aaroninsf 5y ago
Came here to ask this as well :) I have a production system running on Celery (on Redis). It's been fine, once I tuned it. Time for a refactor; wondering: should I consider migration?

Re: why Postgres, two common answers: 1. For anyone whose stack is currently a simple three-tier architecture that currently has Postgres and nothing else in the DB tier, adding anything else would incur 100+% operational complexification (e.g. now you have to figure out how to do backups for two stateful components.) Far more than 100%, even, if they get the Postgres DB "for free" as part of a virtual LAMP-stack appliance, or built into the base offering of some PaaS service, and currently don't need to do any ops work as the framework/platform handles their DB's care and feeding for them. (Ideally, such setups would do the same with a built-in MQ as well — but sadly, that's much rarer.) 2. You can be clever with a job queue that’s in your DB, by making “taking the job” and “doing the queries that comprise the job” part of the same atomic transaction, such that a ROLLBACK due to a constraint validation error in "the queries that comprise the job" will also implicitly “put back” the job immediately (even if the worker crashed in response to seeing the error.)

TameAntelope 5y ago
Generally yes, adding new tech can complicate your infrastructure, but specifically redis is one of those rare gems that, “just works”. If you use Celery, the time saved dropping Celery will more than cover the operational costs of redis. Really, one of the magic tools of the 21st century.

emptysea 5y ago

Not parent but in my experience celery has a number of long standing bugs / poor defaults, like prefetching tasks so they can get stuck behind long running tasks. Next project id probably try a simplistic project that’s easier to understand like arq / rq.

aidos 5y ago
That prefetching is still an issue? I recall it was caused by rabbitmq though? I switched to rq over celery years ago because it was a deal breaker. We have just long running tasks and it didn’t work for us at all. Rq has been great btw.

simonw 5y ago

They address that on this page: https://procrastinate.readthedocs.io/en/stable/discussions.html https://procrastinate.readthedocs.io/en/stable/discussions.h...

41b696ef1113 5y ago

While I respect Celery, it is complicated (some amount is intrinsic to the problem space). I used to use it, but it felt heavyweight and debugging errors was so quite challenging. Have since switched to Dramatiq - which may not be web scale, but works fine for an internal service. I love the idea of re-using infrastructure "for free".

ungawatkt 5y ago
+1 for dramatiq over celery, dramatiq was much easier to understand and extend, even if it's not as "complete". It handled most anything I threw at it for an internal task runner system with some custom scheduling logic. I'd give it a flask out of 10 compared to celery being a Django.

cultofmetatron 5y ago

> why Postgres and not rabbitmq or redis? I chose postrges for our mq stack over rabbitmq and redis. Since our team is small, we are trying to be efficient in our use of manpower. our architecture is as dirt simple as it gets. (loadbalancer -> api-server-nodes -> aws aurora). adding another external dependency just adds one more thing that can potentially go wrong. Thats one more thing we need to watch and maintain accross staging and production envrionemnts and one more thingto get running on developer machines. There may come a day where we switch over to rabbitmq or kafka but postgres has proven to be fast enough. when writes are too much of a load, we can shard the writes to its own dedicated machine and buy more time. When our traffic is sufficiently high that even that isn't enough, we'll have enough revenue to pay someone to configure kafka/rabbitmq and deal with the configuration and maintenance fulltime. Until then, if you want to survive as a startup, KEEP IT SIMPLE. Any complexity you adopt needs to justify itself by providing a competitive advantage.

aaroninsf 5y ago

Came here to ask this as well :) I have a production system running on Celery (on Redis). It's been fine, once I tuned it. Time for a refactor; wondering: should I consider migration?