4 ms·

I wish there was a paragraph up the top that made two points: 1. Quite often you don't (I've built dozens of websites without needing Celery) 2. Even if you t

I wish there was a paragraph up the top that made two points: 1. Quite often you don't (I've built dozens of websites without needing Celery) 2. Even if you think you do there's often a much simpler solution that is enough for most needs (Use cron, spawn a process etc) Celery is a big, heavy lump of code to add to most websites and it increased the deployment complexity.

mattbillenstein 7y ago
I prefer rq - Celery is too complex imho.
- ptype 7y ago
  Also worth checking out nameko, I’ve had good experiences with it
  - cepp 7y ago
    Huey is also worth a shot.
    - mattbillenstein 7y ago
      I like it - uses redis as a broker - supports crontab style periodic tasks. I've usually had to build a small python cron runner using croniter in previous systems - which I think is a pretty clean solution - it just deferred tasks to rq workers. But having direct support in the lib might be nice.
  - mattbillenstein 7y ago
    * depends on rabbitmq
danpalmer 7y ago
It’s worth scoping out what your site will need to do up front to some extent. Spending a couple of days setting up a basic background processing system (whether that’s celery, rq, or a home grown system) makes it easy to make great engineering decisions later down the line. Questions like: should I send this email in-line in the web request? Get a very easy answer: no, just stick do it later. Sure, sending an email is probably fine to do in-line for now, but months in you may realise that things are slow, that you’re sending emails and rolling back transactions later, or committing the transaction but losing the email that needed to be sent, or all manner of other annoying edge cases. Queues don’t solve everything, but they can be an ok answer to a lot of stuff for a long time. For basic sites, yeah maybe not necessary, but a reliable background processing system has always been a significant accelerator in my projects.
- 1337shadow 7y ago
  Same in my experience: emails should always be sent in a background process. Luckily for me, using uWSGI to deploy anything in any languages: it builds in a nice little spooler that's going to let me spool emails without adding a single new software to my stack: not even having to start another process.
raverbashing 7y ago
Completely agree Several developers like to overengineer and "go for celery" (also applies to other technologies with other uses) even for small things. You don't need Celery to run a batch job every day for example. Or to even do some parallel processing. "Oh but python multithreading sucks" do you know when it does not suck? When your thread is waiting on something else. Also there's the multiprocessing module with a lot of "batteries included" for basic use cases. Not to mention (my biggest pet-peeve of Celery) is that it "forces" you to work with a task queue model. Not a data queue model. Sure, it helps a lot when you need that, but sometimes you just need a queue.

mattbillenstein 7y ago

I prefer rq - Celery is too complex imho.

ptype 7y ago
Also worth checking out nameko, I’ve had good experiences with it
- cepp 7y ago
  Huey is also worth a shot.
  - mattbillenstein 7y ago
    I like it - uses redis as a broker - supports crontab style periodic tasks. I've usually had to build a small python cron runner using croniter in previous systems - which I think is a pretty clean solution - it just deferred tasks to rq workers. But having direct support in the lib might be nice.
- mattbillenstein 7y ago
  * depends on rabbitmq

danpalmer 7y ago

It’s worth scoping out what your site will need to do up front to some extent. Spending a couple of days setting up a basic background processing system (whether that’s celery, rq, or a home grown system) makes it easy to make great engineering decisions later down the line. Questions like: should I send this email in-line in the web request? Get a very easy answer: no, just stick do it later. Sure, sending an email is probably fine to do in-line for now, but months in you may realise that things are slow, that you’re sending emails and rolling back transactions later, or committing the transaction but losing the email that needed to be sent, or all manner of other annoying edge cases. Queues don’t solve everything, but they can be an ok answer to a lot of stuff for a long time. For basic sites, yeah maybe not necessary, but a reliable background processing system has always been a significant accelerator in my projects.

1337shadow 7y ago
Same in my experience: emails should always be sent in a background process. Luckily for me, using uWSGI to deploy anything in any languages: it builds in a nice little spooler that's going to let me spool emails without adding a single new software to my stack: not even having to start another process.

raverbashing 7y ago

Completely agree Several developers like to overengineer and "go for celery" (also applies to other technologies with other uses) even for small things. You don't need Celery to run a batch job every day for example. Or to even do some parallel processing. "Oh but python multithreading sucks" do you know when it does not suck? When your thread is waiting on something else. Also there's the multiprocessing module with a lot of "batteries included" for basic use cases. Not to mention (my biggest pet-peeve of Celery) is that it "forces" you to work with a task queue model. Not a data queue model. Sure, it helps a lot when you need that, but sometimes you just need a queue.