This is fascinating. Doordash seems to have encountered many of the same problems we (Klaviyo) did using a RabbitMQ/Celery stack, but arrived at totally differe

This is fascinating. Doordash seems to have encountered many of the same problems we (Klaviyo) did using a RabbitMQ/Celery stack, but arrived at totally different answers. Klaviyo doubled down on making the tools work for us, rather than jumping to a new and untested (for us) broker/job framework technology and arrived at some pretty nice solutions: Problems with observability? We added lots of custom StatsD and text logging instrumentation (Celery "signal" middleware), so that we could get e.g. accurate "how long do tasks like this spend waiting in the queue?" answers. Other than the inherent limitation of RabbitMQ being a strict queue (unlike Kafka, you can't "peek" at things in the middle of a topic without consuming that topic--we could do something hyper-complicated with exchanges and deliberate duplication of messages to address this limitation, but that doesn't sound worth it to me at all), observability of the brokers themselves seems pretty good. Coupled with Sentry reporting issues that occur during task processing, and some custom decorators to retry/delay/store tasks affected by common classes of systems issues, our visibility tends to be better than I've seen in any other asynchronous queue/message-bus system I've worked on. Sneaky task disappearances turned out to be mostly bugs in the way we were starting/stopping workers, related to signal handling and rabbitmq "redelivery". By really understanding the Celery worker start/stop lifecycle and coupling that with how Systemd chooses to kill processes, we were able to reduce those to zero. Celery also had a couple of "bugs" (questionable behavior choices) in this area which were resolved in 4.4. Celery ETA/countown task induced RabbitMQ load turned out to be because Celery made the questionable decision to queue ETA tasks on every single (eventual executor destination) worker node. We customized the celery task-dispatching code to route all ETA tasks to a set of workers which only buffer tasks, and re-deliver them to the actual work queues when their time is up. As a result, the domain of cases in which RabbitMQ had to "take back" (unack -> ready) large amounts of tasks went from "every time every worker restarts (and we deploy to them a lot!)" to "every time a very specific, single-purpose worker crashes", which reduced issues with that system to zero. A lack of scale-out in RabbitMQ was addressed by adding additional separate brokers (and therefore celery "apps") along two axes: sometimes we peel off specific task workloads to their own broker, and in other cases we run "fleets" of identical brokers that tasks round-robin across, with a (currently manual, moving in the direction of automatic) circuit breaker to take a broker out of rotation if it ever has issues. We wrote a whole blog post[1] about scaling that specific set of RabbitMQs and workers. Totally agree with Doordash that rabbit's "HA" generally isn't, and that scale-out needs to happen across brokers. Connection churn-related broker instability was addressed partially by scaling the number of brokers, but also on the consumer side, by carefully tuning per-worker concurrency to minimize connection counts while doing as much work as possible on a given piece of hardware, and by disabling the Celery remote control channel (pidbox queues). While that means that nice tools like e.g. Flower aren't as useful to us, it also means that the cost to a RabbitMQ broker of losing a whole bunch of consumer connections is much lower. When it comes to the "harakiri" churn of publisher connections discussed in the article, we haven't encountered connection issues due to our publisher tier. Doordash's web tier is almost certainly bigger than ours, but I'd make a deeply uneducated guess that we're at most an order of magnitude apart. I'd be curious to learn more about the story there, since, even at a reduced size, we regularly run 10ks of connections on a single broker with a pretty high flap-rate due to e.g. recycling webserver processes or restarting consumers due to code releases. In general, I agree with the article and yesterday's Celery 5.0 release post comments, that Celery is a quirky piece of tech, and that RabbitMQ is far from simple to run. However, I'm generally pretty pleased with Klaviyo's approach to go "through" the problem by diving deep on issues we had and fixing them in the stack we chose, rather than tossing large parts of it and re-learning the foibles of some other piece of software. At present, we run dozens of brokers and process 100ks of tasks per second at peak volume. While nobody considers our setup simple or issue-free, it's one of the most fully understood pieces of technology we run. While it's not out of the question for us to adopt it at some point in the future, Kafka was discouraging to us when hardening our RabbitMQ/Celery setup for a few reasons (though we do use it for some other pieces of our infrastructure which require it): As the Doordash folks indicated in the article, Kafka is really not well-integrated with the Celery stack at all, so building in things like front-vs-back-of-queue retries (both of which are extremely useful in different situations), deferred delivery, and the ability to rapidly change the number of consumers on a topic all take effort. Each of those problems has a solution, or at least a response, in the Kafka ecosystem, but Python task-processing frameworks which integrate those behaviors are both unfamiliar to us, and significantly younger than Celery. As with any clustered (rather than sharded) system, we lack expertise in understanding why publishes fail when Kafka is in a partially degraded state. With our existing Kafka workloads, many failure waves (consume or publish) happen without a full grasp of what's wrong/how to fix it. That's most definitely an "us" problem, and we are learning, but it's likely going to be quite awhile before we're at the comfort level that we currently have with, say, recovering message data from a data volume in the aftermath of a massive AWS-induced broker crash, or replacing RabbitMQ nodes that are experiencing elevated latency or flow control. Lastly, we were unpleasantly surprised by Kafka publishers habit of lying to their clients and saying a message was published when it was in fact buffered in-memory, pending a periodic (or volume-initiated) flush operation. Our processes crash a lot, usually when we least want them to, and having those crashes cause the data loss of an entire pre-publish Kafka buffer has been extremely unpleasant for us. When we reduced "batch.size" to 1, we discovered, to our dismay, that Kafka's vaunted "way better than RabbitMQ" publish time and volume numbers were entirely dependent on batch-wise optimizations, and that publishes were tens of times slower with batch.size=1 Kafka than they were with pub-confirms-enabled RabbitMQ (RabbitMQ also has batch-wise optimizations with publish confirms, which I'd argue have vastly better reliability semantics than Kafka, but that's another story and we're not using those...yet; ask me if interested). Again, that's partly an "us" problem (our crash rate is high, and dropped Kafka-destined batches could be recovered in other ways if we spent the time), but one that we don't have to worry about in the Celery/RabbitMQ setup. 1. https://klaviyo.tech/load-testing-our-event-pipeline-2019-42c984b90aae https://klaviyo.tech/load-testing-our-event-pipeline-2019-42... Edits: a few for clarity and removed a few things that the Doordash folks already covered and that my initial less-than-careful read didn't catch. The substance of my post didn't change.