13 ms·
Celery in production: Three more years of fixing bugs
- iamcreasy 5y agoCan any explain why we need task queue? It is unique to Python because of it lack of real threading?
- shagmin 5y agoEven if there were real threading, it has features that make it convenient for scheduling tasks, chaining them in different ways and so on. You could write your own thing of course. But also it operates over different processes and those processes don't even necessarily have to be on the same machine.
- shoo 5y agoTask queue pattern is not unique to python. Consider where you see physical queues of people in real world: e.g. people lining up to be served at supermarket, to board aircraft, to enter a concert. The purpose of the queue is an orderly way to manage situations where there is more demand for a service than can be immediately processed. If the system tries to serve all the demand simultaneously, perhaps that leads to chaos and no one getting served. Similar situations happen in computer systems, in some of those cases queues can be used to manage load. E.g. When you make a purchase to order something online, there is a good chance there is a task queue or something very similar as part of the machinery helping that to happen. When you click "purchase" your order or request is persisted in some queue or DB, before any attempt is made to fulfill it. The UI can give you immediate feedback telling you that your order was received, here's your order ID, sit back and relax, we'll send you an update. When some machine (or person) is available with free capacity, and no higher priority orders to process, they will grab your order and start processing it. Some worker queue implementations offer additional benefits in terms of increasing robustness of system. e.g. if worker node A begins processing task T, then worker node A explodes before T is complete, the task T may be returned to the queue after some timeout, and surviving worker B may be able to grab T and process it successfully. This can be very valuable in situations where you want to guarantee that task T is done correctly and not half-done or forgotten -- e.g. processing and provisioning customer orders. This gets considerably more challenging to implement correctly if task T involves performing side effects and modifying the state of the real world (transferring money, dispatching vehicles) and the failure occurs after some but not all of the side effects have already happened. Queues introduce new failure modes: if something stops workers from processing tasks, and queue gets longer and longer, how do you find out and fix it? Queue needs to be stored somewhere, there is some max capacity, what happens when that is exceeded -- does the queue give up and drop all the tasks on the floor, or explode, or so on.
- geraneum 5y agoWe have used it to run different CPU bound tasks on worker machines (to not run them on the app server). For example imagine you want to receive a file from the user and extract data from it. Using such system also allowed us to horizontally scale by adding more worker machines. All this, of course, depends on the tasks you want to schedule.
- aetimmes 5y agoThreading is local to a machine; task queues are generally a solution to distributing a large number of tasks across many distinct compute nodes by using some remote network service (redis, rmq, etc) to track the progress of jobs.
- zbentley 5y ago> Supervisor should've killed celery's forked workers once the parent dies. Either it is not reliable or we haven't been able to make this happen in at least a few attempts now. Use systemd instead of supervisord for misbehaving forking daemons like this. If you set "KillMode=mixed" you get a nice compromise between "signal the parent and give things a configurable grace period to shut themselves down" and "zero risk of orphaned child (or stuck parent) processes being left around after service stop/restart". Without weighing in on the systemd-vs-other-init debate in general, I do strongly believe that systemd fully obsoletes supervisord. The (slightly) tighter coupling supervisord offers with Python isn't worth the deficient feature set.
- syro 5y agosupervisord still has a place in scenarios where you can't or don't want to use the init system. WSL2 and docker containers running multiple processes come to mind. In most linux environments, this is a non-issue. And some will argue you are using containers wrong if you are running multiple services in a single container, but containers as a reproducible environment and isolation unit is a valid use case (for test/dev environments especially).
- boardwaalk 5y agoJust want to point out there are options other than supervisord. You can even use systemd in a container, though I would avoid it. s6 and runit are both options. Using something Python-based for this sort of thing makes me slightly nervous.
- js2 5y agoI find supervisor's event listener functionality useful for implementing watchdog processes. I have the supervised process write an event whenever it successfully completes a task. Meanwhile, a separate event listener process has a watchdog timer which asks supervisor to restart the supervised process if it hasn't checked in recently enough. Does systemd have anything like supervisor events? http://supervisord.org/events.html#events http://supervisord.org/events.html#events
- tluyben2 5y agoI ran Celery in many projects in production over the past 10 years; I would not recommend it. It is mostly a constant fight even though it is the first thing most people grab when they have a Django stack.
- gregsadetsky 5y agoWhat would you recommend instead?
- ungawatkt 5y agoI had a good time with dramatiq+apecheduler, much more bare bones and less out of the box, but when something went wrong is was pretty easy to understand and work around.
- staticautomatic 5y agoDramatiq is great.
- wryun 5y agoI'm also in the 'anything but Celery' camp: https://dramatiq.io/motivation.html https://dramatiq.io/motivation.html If you're backing onto something like SQS or Rabbit (Redis is a bit trickier), it's not the worst thing to hand-roll either. Then you understand exactly what trade-offs you're making.
- porker 5y agoBeanstalkd? These comments say better than I can why I like it https://news.ycombinator.com/item?id=27483167 https://news.ycombinator.com/item?id=27483167 https://news.ycombinator.com/item?id=24714198 https://news.ycombinator.com/item?id=24714198
- 411111111111111 5y agoCelery is not a broker daemon, it's an abstraction for them. You can use it with various brokers and some kv databases such as redis. And yes, celery also supports beanstalk, so it's clearly not an alternative to it http://docs.celeryproject.org/en/latest/getting-started/brokers/beanstalk.html http://docs.celeryproject.org/en/latest/getting-started/brok...
- nerdbaggy 5y agoI love Huey [1]. I switched everything over from Celery to it. It’s so simple it’s a godsend, even supports SQLite queue [1] https://github.com/coleifer/huey https://github.com/coleifer/huey
- GSGBen 5y agoI implemented this for its Windows support and it's been great.
- jteppinette 5y agoThoughts on Huey vs RQ?
- imp 5y agoI have been using Celery a lot for several projects and haven't really run into any significant issues. I like it as part of my tech stack. Sorry to hear others don't feel the same way. I find it easy to use and maintain.
- gilad 5y agoRQ [0] (Python-based with Redis backend) is another option; works great. [0] https://python-rq.org/ https://python-rq.org/
- leetrout 5y agoI love RQ. Nothing but good experiences using it several times over the past ~6 years. However if you are doin mission critical work in your queue that cannot be replayed Celery is still the best python flavored option IMO
- tux1968 5y agoOthers above have been suggesting Huey as an alternative. Do you know of any reason that it falls short of Celery in a mission critical situation?
- leetrout 5y agoAnything using redis will be riskier than AMQP by default. Redis can be tuned to be safer (such as persisting on every key change) but this all comes with tradeoffs. Redis such a great piece of tech I use it first as often as I can.
- azinman2 5y agoSorry why is Redis riskier?
- leetrout 5y agoBecause it's default configuration and (most) naive setups offer no redundancy? It is almost entirely in-memory with no easy eviction controls? If you are going to snapshot on every change because the thing you are doing needs to happen another purpose built queue like rabbitmq is going to perform better in this situation, especially during failures. Clustering is another topic entirely.
- 5y ago
- captaintobs 5y agoI wrote an extremely performant and simple async worker framework called SAQ because I couldn't find any that fit my use case. https://github.com/tobymao/saq https://github.com/tobymao/saq
- johtso 5y agoYou said you drew inspiration from arq. What was lacking with arq that made you want to write something new?
- jefficient 5y agoI've been using Celery on a small scale for years now and generally haven't had any serious issues except 1: reliably cancelling scheduled (future) tasks. Even using persistent revokes has been unreliable in my experience: https://docs.celeryproject.org/en/latest/userguide/workers.html#worker-persistent-revokes https://docs.celeryproject.org/en/latest/userguide/workers.h... Fwiw though, the alternatives (huey, dramatiq, rq) don't even seem to offer this feature.
- memco 5y agoI’ve been using Celery and it has been working okay for our needs but it is not very friendly to make changes depending on how you write tasks. I feel okay when a task is a wrapper around some other functions that you can just run and test with any tools you’d like but we have some tasks that go all in by binding all kids of data and functionality to the task that isn’t guaranteed to be there if you just launch a task with .delay or similar so you can’t just spin up a REPL or debugger and call the function and have it work.
- darkoob12 5y agoi am using it for four months. its configuration is really complicated and many default values do not make sense. for instance when you are using solo there should be a default hard task timeout smaller than effective heartbeat value otherwise it will fail and retry the same task forever.
- true_religion 5y agoMy advise is to just pay someone to run your celery server for you. Celery is hard to maintain, but the benefits are definitely worth it, and it’s the most mature cross language stack. I can have a celery job sent or received from any major language. I think the only reason people pick another option is that that start of by only using python in the first place.
- samirsd 5y agoi gave up on celery and switched to huey
- jhgg 5y agoCelery is not great. At my place of work we begrudgingly are using it right now, with plans to replace it with something else this year. I am at the very least amazed we got this much use out of it. Considering replacing it with a much much more simple version that just uses google cloud pub sub and some simple python routing of the messages to functions.
- mattbillenstein 5y agoNever deployed Celery as I got stuck at the docs - it seems way too complicated. Ran beanstalkd in small deployments, rq in others, and nsq for real queueing but at a small scale.
- andrewstuart 5y agoI suspect in many cases people use Celery only to send emails or for simple queueing type problems. At risk of being overly negative, my experience with celery has been that it is complex and hard to configure and debug. I chose a simple queueing mechanism instead of using Celery, and for sending emails I was so frustrated by Celery that I wrote the Arnie SMTP buffer server (in 100 lines of Python): https://github.com/bootrino/arniesmtpbufferserver https://github.com/bootrino/arniesmtpbufferserver
- sz3 5y agoI would put celery in that class of software that's good enough to be useful, but enough trouble that you'll never be truly happy with it. We got a lot of mileage out of it (in a past life) before finally moving our job scheduling to a custom solution built on RQ. Celery caused more than a few headaches -- which is why we ditched it -- but its flexibility probably helped us scale up the service to begin with...
- johtso 5y agoI've been mulling over using arq for a new project, a successor to rq written by the author of Pydantic. https://arq-docs.helpmanual.io/ https://arq-docs.helpmanual.io/ Does anyone have any experience with it? I like the pessimistic execution concept..
- e-brake 5y agoWe switched our python task processing from celery to pubsub (on GCP) backed by redis for message storage, and it's much better. Celery is full of bugs and quirks, nobody ever really knew what it was doing when it would fail. For simple task processing, we find it helpful to have a task queue that you can fully understand. Just a hundred lines of code for the subscriber and publisher logic, which we wrote ourselves. If you can get by without workflow and orchestration, pubsub works well at high volume and it's cheap. Something to consider!
- qeternity 5y agoI'll take the other side of all this hate. We use celery for quite a bit, and I struggle to sympathize with TFA. Much of what's written comes back to using old python2 libraries or just complaining that celery doesn't have defaults that better suit you. The very first issue mentioned, about not manually setting the concurrency level...this is basic stuff. It always boggles my mind when people deploy something to prod with the copy/paste "get started" instructions from github. Why do people deploy critical infra layers without spending a lot of time to understand the thing they're depending on? There are genuinely bugs in celery, and task queues are particularly hard to get right, especially with pluggable backends. But if you RTFD and follow good devops, you will have a perfectly fine time.
- ascendantlogic 5y agoThe moment I saw mention of Python 2 I was like "well, yeah". I'm sure that won't solve all, or even many of the ills, but if you're not reasonably up to date with your language and libraries then yeah, you're probably gonna have a harder time than you need to.
- nickjj 5y agoI can't say I've encountered the same things and I've been using Celery since around 2015. In one case of using it I am routinely throwing work at it. Its main job is to contact third party APIs and parse up to 600MB of XML responses. It's not high traffic but it's active enough to be running jobs often. This box has been really stable, often times going 6 months without having to restart Flask or Celery because it's humming along without issues and the service itself is fairly stable so it's not getting new versions rolled out (just security patches as needed). It's not a legacy system either, it's just feature complete. In the past I've had other sites where Celery was doing around a million jobs a month to massage a small amount of data and write it to a Postgres database, it also had no issues and was super stable. I mainly use standard Celery features backed by Redis on a low'ish end VPS. I would like to know more details about what folks are doing where they're encountering all of these problems.