5 ms·
> We use Celery for the following general cases: > > Communicating with 3rd party services (sending emails, notifications, etc.) > Offloading heavier c
by andybak 4y ago
> We use Celery for the following general cases:
>
> Communicating with 3rd party services (sending emails, notifications, etc.)
> Offloading heavier computational tasks outside the HTTP cycle.
> Periodic tasks (using Celery beat)
Sigh. No mention of the trade-offs. There's simpler ways to do all these things. Celery is a big complex beast and it always pains me to see it as the default suggestion for simple tasks.
- andrewingram 4y agoI've been using the beta of Temporal's Python SDK with Django, and aside from some minor teething issues, it's very nice.
- 69Represente 4y agoI have a bug with celery i can't solve. when I send an async job that get data from various APIs and write all in a DB, in the case of lot there is lot of data, the celery task finish properly to but my flask app becomes unresponsive. I have to restart flask to get back to a normal state. Anyone would know where I should check?
- roflyear 4y agosounds like a resource issue, maybe you are opening application contexts and not handling them properly?
- aequitas 4y agoA lot of people use Django with uWSGI, which also comes with queues, cron, workers, cache and lots more. I've been stuck with Celery on previous projects for reasons. But I've been dying to try out uWSGI's built in features for this. Hearing great things about it. https://uwsgi-docs.readthedocs.io/en/latest/Spooler.html https://uwsgi-docs.readthedocs.io/en/latest/Spooler.html https://uwsgi-docs.readthedocs.io/en/latest/Cron.html https://uwsgi-docs.readthedocs.io/en/latest/Cron.html https://uwsgi-docs.readthedocs.io/en/latest/Mules.html https://uwsgi-docs.readthedocs.io/en/latest/Mules.html https://uwsgi-docs.readthedocs.io/en/latest/Caching.html https://uwsgi-docs.readthedocs.io/en/latest/Caching.html
- asalahli 4y agoFellow uWSGI fan here. Unfortunately uWSGI is now in maintenance mode, not because is complete which would've been fine, but because the maintainers are not able to dedicate time to it[0]
- collyw 4y agoDjango subreddit. Someone asks for a way to run a simple cron job once a week. Everyone jumps in and shouts "Celery". I chip in and say, nah, just set up a cron job, it will be simpler. Get downvoted by the hivemind.
- hdjjhhvvhga 4y agoWhat would be your first choices for each of the above?
- roflyear 4y agoThis is good for really simple scheduling of stuff: https://schedule.readthedocs.io/en/stable/ https://schedule.readthedocs.io/en/stable/
- true_religion 4y agoMy choice for periodic jobs is cronjobs.
- VWWHFSfQ 4y agoDo you mean a cronjob that calls a Django manager command to do the work? Or invokes an API method? From my experience cronjobs have a lot of downsides as well. They're great for doing tasks local to the server the job is running on. Not so great for the kinds of tasks (periodic or transactional) that Celery/a real queue is designed for.
- true_religion 4y agoI use them for invoking django commands on the same server. I do use celery for transactional jobs though. It’s only periodic jobs that get called with cron. For the context, I do this on small web apps with less than 20 dedicated servers (real servers not VPS), so there is a “manager” server that does nothing but run periodic tasks, management interventions, and cleanup operations.
- danjac 4y agoCrons are fine for running Django commands that are just talking to the same services as your web views (databases, cache, email services etc). It should be fine to let them run in their dedicated VM/containers, using Ansible etc to keep the crontab configurations in the repo. Celerybeat has the advantage of better visibility e.g. you can configure them in the Django admin and check when they are running. If you are not using Celery though and your needs are simple, it's easier to just use plain crons.
- notdarkyet 4y agoDo you have additional examples of those simpler ways? I totally understand how Celery can be a hammer and everything is a nail type situation.
- nerdbaggy 4y agoI am a big fan of Huey. And they have a Django module https://huey.readthedocs.io/en/latest/ https://huey.readthedocs.io/en/latest/
- j4mie 4y agohttps://github.com/dabapps/django-db-queue https://github.com/dabapps/django-db-queue (I am the author) EDIT: oh hi, Andy :)
- nicolaslem 4y agoSince everyone suggests alternatives to Celery, may I plug my own healthier celery? https://github.com/NicolasLM/spinach https://github.com/NicolasLM/spinach
- giancarlostoro 4y agoI would just use serverless functions to achieve the same thing personally, if you're in the cloud already, chances are high you can trigger functions based on new records in a database or new file uploaded, or what have you. Then you don't need to import much outside of the serverless SDK which should typically be pretty minimal. That's how I did a timed function for a Django project we were hosting in Azure anyway.
- roflyear 4y agoServerless functions are good for some stuff: clear this cache on this schedule, run this task when XYZ happens, etc.. But that is not the popular usecase for celery. Often you want "some code" (that you likely already have written) to be executed async. Sure, you can create a public interface for "some code" then write a record, that the serverless function is looking for, that then calls back to that interface (but is it a web interface???? then you have a problem where if the job takes too long to complete, what about http timeouts ... ) and now you're really creating a big circle for something that should be simple: execute some code outside of the request (send an email, hit an api, whatever). Serverless functions really shouldn't contain much logic either, because it's too complicated to test.
- nickjj 4y agoA long time ago I wrote a blog post about Celery use cases at https://nickjanetakis.com/blog/4-use-cases-for-when-to-use-celery-in-a-flask-application https://nickjanetakis.com/blog/4-use-cases-for-when-to-use-c.... It applies to Django, Flask or any Python system using it. All of it still applies today. It covers a few use cases on the before vs after of using Celery and touches base on why I'd consider using Celery over other solutions such as async / await. The TL;DR is Celery brings a lot to the table around tracking and retrying jobs. It's also nice to separate your web and worker workloads into different processes since they have much different requirements usually.
- emptysea 4y agoI think as a concept a task queue with some workers makes a lot of sense but having used Celery in production, it leaves a lot to be desired We’ve run into various bugs and weird performance gotchas (like the workers prefetch jobs which is terrible if they aren’t all the same size)
- roflyear 4y agoAgreed. I have had more luck writing my own "jobs" engine that does stuff that I need that celery doesn't have anyway (retries, some record of execution, rate limiting). I'm sure if I really tried, I can get celery to be very reliable... but I never really got there. Also for whatever reason I have NEVER been able to get celery to be reliable for its scheduling/cron stuff. It just starts to fail. I use this library for that, which I have never had problems with: https://schedule.readthedocs.io/en/stable/ https://schedule.readthedocs.io/en/stable/
- nickjj 4y agoWhat type of situations did you run into while using it? I've been using Celery for a long time in production now. Nothing crazy and it's a fairly basic set up of "execute job, thanks!" along with a beat server. Over the last 6-7 years an off the top of my head guess would be that there's been at least 5 million jobs processed. It's not huge volume when measured over years but it's been stable. One server was running for 6 months without being updated. That's a Celery worker process running for ~180 days uninterrupted. It served hundreds of thousands of jobs without maintenance. A lot of these were pretty beefy tasks too like performing HTTP requests that got 400mb XML responses and then parsed them. I didn't even have things like `worker_max_tasks_per_child` set either.
- Spivak 4y agoBecause it’s mature, well integrated with Django and is a path so well-trodden there’s a McDonald’s on the way. Any possible use-case you can imagine for a job queue has been done in Celery and documented. Celery being complicated is also entirely on the operational side, once you actually have Celery using it from within your app is simple enough. Cron is awful for this use-case. You end up just inventing Celery but worse when you decide how your app and the cron scripts communicate. If you wanted just scheduled tasks but simpler use something like APScheduler.
- andybak 4y ago> Celery being complicated is also entirely on the operational side, once you actually have Celery using it from within your app is simple enough. Yeah and to some degree improvements in devops quality of life in the last few years has softened my view. (I used to mainly use Webfaction without access to apt-get and my own hand-rolled scripted deployment. Ugh...) But I'd still usually prefer a pure-Python solution without additional persistent processes - assuming there is one and it's fairly well-documented. Huey is pretty good from recollection. > Cron is awful for this use-case. You end up just inventing Celery but worse when you decide how your app and the cron scripts communicate. If you wanted just scheduled tasks but simpler use something like APScheduler. I was advocating for something like this. There's django-cron etc which solves issues around communicating with scripts. I do see both sides of the debate between "complexity" and "solves problems out of the box". I'm generally on the Django side when the flask vs Django discussion happens. There's always trade-offs.
- hbrn 4y ago> Because it’s mature, well integrated with Django and is a path so well-trodden > Cron is awful for this use-case. You end up just inventing Celery Isn't it the other way around? Crons are way more mature, well integrated (mgmt commands don't require 3rd party modules), and extremely well trodden. Crons are super predictable, have sensible defaults and plenty of tooling. Which you will have to reinvent with Celery. There are some benefits to programmatic crons, but the downsides are huge.
- ccanassa 4y agoI spent 3 years building a high scale crawler on top of Celery. I can't recommend it. We found many bugs in the more advanced features of Celery (like Canvas) we also ran into some really weird issues like tasks getting duplicated for no reason [1]. The most concerning problem is that the project was abandoned. The original creator is not working on it anymore and all issues that we raised were ignored. We had to fork the project and apply our own fixes to it. This was 4 years ago so maybe things improved since them. Celery is also extremely complex. I would recommend https://dramatiq.io/ https://dramatiq.io/ instead. [1]: https://github.com/celery/celery/issues/4426 https://github.com/celery/celery/issues/4426