4 ms·
I ran Celery in many projects in production over the past 10 years; I would not recommend it. It is mostly a constant fight even though it is the first thing mo
by tluyben2 5y ago
I ran Celery in many projects in production over the past 10 years; I would not recommend it. It is mostly a constant fight even though it is the first thing most people grab when they have a Django stack.
- gregsadetsky 5y agoWhat would you recommend instead?
- ungawatkt 5y agoI had a good time with dramatiq+apecheduler, much more bare bones and less out of the box, but when something went wrong is was pretty easy to understand and work around.
- staticautomatic 5y agoDramatiq is great.
- wryun 5y agoI'm also in the 'anything but Celery' camp: https://dramatiq.io/motivation.html https://dramatiq.io/motivation.html If you're backing onto something like SQS or Rabbit (Redis is a bit trickier), it's not the worst thing to hand-roll either. Then you understand exactly what trade-offs you're making.
- porker 5y agoBeanstalkd? These comments say better than I can why I like it https://news.ycombinator.com/item?id=27483167 https://news.ycombinator.com/item?id=27483167 https://news.ycombinator.com/item?id=24714198 https://news.ycombinator.com/item?id=24714198
- 411111111111111 5y agoCelery is not a broker daemon, it's an abstraction for them. You can use it with various brokers and some kv databases such as redis. And yes, celery also supports beanstalk, so it's clearly not an alternative to it http://docs.celeryproject.org/en/latest/getting-started/brokers/beanstalk.html http://docs.celeryproject.org/en/latest/getting-started/brok...
- rtpg 5y agoWe have transitioned over to Dramatiq. It has its own set of problems (mostly from starting at the set of "we will remove all of Celery features" and then going from there), in particular its logging policies are extremely frustrating and we have had to work around them. But the code is so simple in general that we get good compensation for that. I personally would have liked to use dq instead but that's mostly an aesthetics thing.
- akvadrako 5y agoFor under 1000 messages per second, PostgreSQL is great at queues.
- johtso 5y agoCurrently exploring using arq, a successor to rq for my next project.
- the_cat_kittles 5y agoman... couldnt agree more.
- js2 5y agoI've been running four separate installs of Celery since 2014 or so processing about 10-20 million jobs a day, using reddit as a broker, and with about 100 worker instances per install. It just works, but I don't use Celery results or any esoteric features.
- loloquwowndueo 5y agoReddit :)
- sergiomattei 5y agoDid we just witness a Freudian slip? :)
- Operyl 5y agoOr overzealous phone auto correct. Redis used to auto correct to Reddit on an old android device of mine heh.
- hoesephgerrible 5y agoI can't stop giggling. ;)
- Scarblac 5y agoPosts to a subreddit as things in the queue, comments for results and feedback like progress percentages, up and down votes for priorities...
- js2 5y agoThat's embarrassing! I'd like to blame autocorrect but I doubt it...
- crucialfelix 5y agoI used it for 10 years myself. Also learned the hard way not to use esoteric features. I did have to build my own solutions for things like priority queues, but it was much clearer and testable.
- alesso_x 5y agoI totally agree, I would not recommend using Celery. The last time I used Celery was in 2019 and it was a mess. The memory leak issue is still open https://github.com/celery/celery/issues/4843 https://github.com/celery/celery/issues/4843 to this day.
- postpawl 5y agoThere has been a lot of progress on fixing memory leaks recently: https://www.paulsprogrammingnotes.com/2021/12/python-memory-leaks.html https://www.paulsprogrammingnotes.com/2021/12/python-memory-...
- sillycube 5y agoDo you have any suggestions for alternatives?
- matsemann 5y agoIMO using python at all for this kinda setup with lots of workers is kinda crazy. As there is no usable threading, each worker need its own process. And as the article says, that's a couple of hundred mb of ram per worker. But in other languages, you could have hundreds of threads doing work with the same resources.
- ddorian43 5y agoYou can have async workers or gevent workers in python.
- odiroot 5y agoWhat have you switched to?