4 ms·

What I find really interesting is that their decision making process. At least the way the article is written suggests they thought of five possible solutions.

What I find really interesting is that their decision making process. At least the way the article is written suggests they thought of five possible solutions. However, they decided to go all-in on a single solution without actually doing any significant investigation or actually trying the others. Option 1: Redis broker. Did they ever setup a test environment with a Redis broker to see how it compares to a RabbitMQ broker? Seems like they didn’t. Option 2: Same as option 1. Did they actually try to setup a kafka broker for celery to test it out and see how it goes? Option 3: Multiple brokers. They could have attempted a very simple setup starting with just two RabbitMQs and dividing the tasks roughly in half between them. This would allow scaling of RabbitMQ horizontally. At least try it and see. Option 4: Upgrade versions. Did they even try setting up an environment with upgraded versions. They say there’s no guarantee it fixes their observed bugs. How about trying it out and doing some more observation? Option 5: We decided to go all-in on this one without even trying the other 4 and doing any sort of comparison with actual tests or benchmarks. Maybe this was the best solution for them, and they chose wisely, but how do they know? The big thing that gets me is they talk about the observed issue of celery workers that stop processing tasks as well as the limited observability of celery workers and RabbitMQ. These aren’t black boxes. They are open source. You can debug them yourselves, report the bugs, fix the bugs and submit patches, add features, fork it yourself if necessary, etc. The fact that they don’t know if new versions will fix their observed bugs makes it clear they never identified those bugs. What’s the point of running on an open source stack if you’re going to treat things like a black box anyway?

rad_gruchalski 6y ago
> The fact that they don’t know if new versions will fix their observed bugs makes it clear they never identified those bugs. What’s the point of running on an open source stack if you’re going to treat things like a black box anyway? This will probably not be a popular opinion here but the business of DoorDash is not to fix open source software. I get your point. I use a lot of OSS in my business and we do contribute back. But not every business has the opportunity to observe the stack for days or weeks while the real clients suffer due to outages. Each client who is affected by an outage is most likely churning immediately and you're not going to win them back easily. In essence, the migration to Kafka will only benefit them long term and they have ultimately made the right business choice.
- toast0 6y ago
  > This will probably not be a popular opinion here but the business of DoorDash is not to fix open source software. It probably isn't. But as their business depends on software, it's worrisome that rather than understanding what was wrong, they just switched software. That doesn't seem inspiring to me. Nobody said they had to fix the problems, but identifying them, and checking if they were fixed upstream seems like a good idea. Sometimes upgrading is a nightmare and doesn't fix anything, sometimes it fixes a major issue and drops cpu usage from 90% to 5%. If you know what the problem areas are, it should take less than a day to look at the changes and see if that area was addressed. Even if not, reading the changelogs and version notes should give some idea. It's fine if they didn't test upgrading, but again, I'm not inspired about their engineering talent and processes when it looks like they didn't even look to see if it would have been likely too help.
  - xupybd 6y ago
    Sometimes you're burnt by bugs. You lose trust in a product. Its not that it can't be fixed it's that it was broken when you expected it to work. It's not rational it's emotional.
remote_phone 6y ago
A lot of people at DoorDash are from Uber. Uber is one of the largest installations of Kafka in the world, so a lot of key processes there are based of Kafka. There was probably inhouse expertise from ex-Uber engineers that made it easier to transition.
- ckdarby 6y ago
  This is a typical pattern for large compensation payout with little effort after the large initial investment. Engineering Daily podcast has various podcasts that come up around this. One that comes to mind was the one with Slack. Slack had hyper-growth, they went out and hired individuals who had already tackled these problems at the same numbers or larger. You get hired at a company, and you look around at what FAANG is using, you implement that because you can always say, "But it works at Netflix". You build up immense expertise in the project, the tool, etc. Switch jobs, ignore everything the existing company is doing, where the landscape is going and shove your agenda of implementing the same thing such as Kafka. Measure everything. Look like the saviour of the company. Point to this, ask for massive stock compensation, bonus, etc. based on the metrics that make you look good.
  - fsociety 6y ago
    Ah-ha! So true, never thought of it that way. I’ve seen a startup BURN through money with Kafka clusters. The kicker was that they didn’t need it.
    - remote_phone 6y ago
      There are only a handful of companies who need the true power of Kafka. It is a burden on devops and expensive. If you’re too small you definitely will waste money pursuing this aspiration.
rbanffy 6y ago
One of the nice things of Celery is how it abstracts away the queues. This is unfortunate if you want to see the queues from up close. I don't have the full picture, but I'd move to multiple brokers first, which seems like a low effort move, then away from Celery, in order to split worker code from the rest of the application. In the meantime, I'd instrument the hell out of Celery to see what was going on and push those changes back into upstream.
jdotjdot 6y ago
Quite a few of these issues too strike me as Celery problems rather than RabbitMQ. I’ve run into many of these similar issues and in every case it was due to Celery’s implementation not using RabbitMQ properly and was fixed with an internal patch to Celery. The most blatant example is the countdown tasks. Celery has a very strange implementation of these (meant to be broker agnostic) where it consumes the task from queue, sees in the task custom headers (which is meaningless to RabbitMQ) that it should be delayed and then sits on the task and takes a new task. That results in heavy memory load on your celery client holding all these tasks in memory, and if you have acks_late set, RabbitMQ will be sitting on many tasks that are claimed by a client but not acked and _also_ have to sit in memory. But that is 100% a celery problem, not Rabbit, and we solved it by overriding countdowns to use DLX queues instead so that we could use Rabbit-native features. Not surprisingly, Rabbit performs a lot better when you’re using native built-in features.
- zbentley 6y ago
  With your countdown implementation, how did you solve the resolution issue? I.e. if I delay a task for 111 seconds, I can publish it to RabbitMQ with a queue with a per-queue TTL of 100 seconds, after which it'll fall out to a DLX. What happens with the remaining 11 seconds? Does the DLX deliver (somehow) to a series of increasingly shorter per-queue TTL queues? I, too, would love an implementation of arbitrary-granularity delays without buffering (potentially huge, mem-wise) tasks in "unacknowledged" in a giant binary heap: that's expensive for consumers and terrifying for RabbitMQ stability when a broker has to e.g. re-"ready" millions of messages because a delay-buffer OOMed.
  - avinassh 6y ago
    One solution which we did was to use the official delay plugin provided by the Rabbit MQ. It’s working good, so far https://www.rabbitmq.com/blog/2015/04/16/scheduling-messages-with-rabbitmq/ https://www.rabbitmq.com/blog/2015/04/16/scheduling-messages...
    - zbentley 6y ago
ww520 6y ago
It sounds like they tested a MVP with Kafka and it’s so good that there’s no need to try the other options.
maxdo 6y ago
Redis is good only for a limited set of use cases. 1. Cache it's amazing... unless you need keys something more complex than key value, e.g. multiple keys, invalidate by pattern etc... So ... not so good actually. 2. Pubsub doesn't sale really good, and that'a a real limitation 3. The new pubsub is purely a copy of Kafka ( they admit that in docs) but in a much less supported, and feature rich version. So despite I'm still using Redis in my Day today, as of 2020 I don't see redis in any new setup.
tyfon 6y ago
Pardon my ignorance here, I have not worked with these kinds of large scale systems. But the first stack I would reach for to handle a queue would be erlang/otp (or maybe elixir), is that not designed to handle large queues without failure? Again I am absolutely no expert but I'd like to hear the views on this.
- dgritsko 6y ago
  RabbitMQ is written in Erlang.
nitrogen 6y ago
limited observability of celery workers and RabbitMQ RabbitMQ has very good observability. It has built-in web dashboards as well as an API. Every day that I had to bang my head against the opacity of an org-wide Kafka or ActiveMQ setup at one job, I missed the simplicity and transparency of RabbitMQ from a previous job. Somehow Kafka earned the "trendy" badge though, so everyone just uses it by default.
quazar987 6y ago
Indeed In absence of any significant stats its safe to assume their decision were driver by this two statements from article. "There were no in-house Celery or RabbitMQ experts at DoorDash who we could lean on to help devise a scaling strategy for this technology." "DoorDash had in-house Kafka expertise"

rad_gruchalski 6y ago

> The fact that they don’t know if new versions will fix their observed bugs makes it clear they never identified those bugs. What’s the point of running on an open source stack if you’re going to treat things like a black box anyway? This will probably not be a popular opinion here but the business of DoorDash is not to fix open source software. I get your point. I use a lot of OSS in my business and we do contribute back. But not every business has the opportunity to observe the stack for days or weeks while the real clients suffer due to outages. Each client who is affected by an outage is most likely churning immediately and you're not going to win them back easily. In essence, the migration to Kafka will only benefit them long term and they have ultimately made the right business choice.

toast0 6y ago
> This will probably not be a popular opinion here but the business of DoorDash is not to fix open source software. It probably isn't. But as their business depends on software, it's worrisome that rather than understanding what was wrong, they just switched software. That doesn't seem inspiring to me. Nobody said they had to fix the problems, but identifying them, and checking if they were fixed upstream seems like a good idea. Sometimes upgrading is a nightmare and doesn't fix anything, sometimes it fixes a major issue and drops cpu usage from 90% to 5%. If you know what the problem areas are, it should take less than a day to look at the changes and see if that area was addressed. Even if not, reading the changelogs and version notes should give some idea. It's fine if they didn't test upgrading, but again, I'm not inspired about their engineering talent and processes when it looks like they didn't even look to see if it would have been likely too help.
- xupybd 6y ago
  Sometimes you're burnt by bugs. You lose trust in a product. Its not that it can't be fixed it's that it was broken when you expected it to work. It's not rational it's emotional.

remote_phone 6y ago

A lot of people at DoorDash are from Uber. Uber is one of the largest installations of Kafka in the world, so a lot of key processes there are based of Kafka. There was probably inhouse expertise from ex-Uber engineers that made it easier to transition.

ckdarby 6y ago
This is a typical pattern for large compensation payout with little effort after the large initial investment. Engineering Daily podcast has various podcasts that come up around this. One that comes to mind was the one with Slack. Slack had hyper-growth, they went out and hired individuals who had already tackled these problems at the same numbers or larger. You get hired at a company, and you look around at what FAANG is using, you implement that because you can always say, "But it works at Netflix". You build up immense expertise in the project, the tool, etc. Switch jobs, ignore everything the existing company is doing, where the landscape is going and shove your agenda of implementing the same thing such as Kafka. Measure everything. Look like the saviour of the company. Point to this, ask for massive stock compensation, bonus, etc. based on the metrics that make you look good.
- fsociety 6y ago
  Ah-ha! So true, never thought of it that way. I’ve seen a startup BURN through money with Kafka clusters. The kicker was that they didn’t need it.
  - remote_phone 6y ago
    There are only a handful of companies who need the true power of Kafka. It is a burden on devops and expensive. If you’re too small you definitely will waste money pursuing this aspiration.

rbanffy 6y ago

One of the nice things of Celery is how it abstracts away the queues. This is unfortunate if you want to see the queues from up close. I don't have the full picture, but I'd move to multiple brokers first, which seems like a low effort move, then away from Celery, in order to split worker code from the rest of the application. In the meantime, I'd instrument the hell out of Celery to see what was going on and push those changes back into upstream.

jdotjdot 6y ago

Quite a few of these issues too strike me as Celery problems rather than RabbitMQ. I’ve run into many of these similar issues and in every case it was due to Celery’s implementation not using RabbitMQ properly and was fixed with an internal patch to Celery. The most blatant example is the countdown tasks. Celery has a very strange implementation of these (meant to be broker agnostic) where it consumes the task from queue, sees in the task custom headers (which is meaningless to RabbitMQ) that it should be delayed and then sits on the task and takes a new task. That results in heavy memory load on your celery client holding all these tasks in memory, and if you have acks_late set, RabbitMQ will be sitting on many tasks that are claimed by a client but not acked and _also_ have to sit in memory. But that is 100% a celery problem, not Rabbit, and we solved it by overriding countdowns to use DLX queues instead so that we could use Rabbit-native features. Not surprisingly, Rabbit performs a lot better when you’re using native built-in features.

zbentley 6y ago
With your countdown implementation, how did you solve the resolution issue? I.e. if I delay a task for 111 seconds, I can publish it to RabbitMQ with a queue with a per-queue TTL of 100 seconds, after which it'll fall out to a DLX. What happens with the remaining 11 seconds? Does the DLX deliver (somehow) to a series of increasingly shorter per-queue TTL queues? I, too, would love an implementation of arbitrary-granularity delays without buffering (potentially huge, mem-wise) tasks in "unacknowledged" in a giant binary heap: that's expensive for consumers and terrifying for RabbitMQ stability when a broker has to e.g. re-"ready" millions of messages because a delay-buffer OOMed.
- avinassh 6y ago
  One solution which we did was to use the official delay plugin provided by the Rabbit MQ. It’s working good, so far https://www.rabbitmq.com/blog/2015/04/16/scheduling-messages-with-rabbitmq/ https://www.rabbitmq.com/blog/2015/04/16/scheduling-messages...
  - zbentley 6y ago

ww520 6y ago

It sounds like they tested a MVP with Kafka and it’s so good that there’s no need to try the other options.

maxdo 6y ago

Redis is good only for a limited set of use cases. 1. Cache it's amazing... unless you need keys something more complex than key value, e.g. multiple keys, invalidate by pattern etc... So ... not so good actually. 2. Pubsub doesn't sale really good, and that'a a real limitation 3. The new pubsub is purely a copy of Kafka ( they admit that in docs) but in a much less supported, and feature rich version. So despite I'm still using Redis in my Day today, as of 2020 I don't see redis in any new setup.

tyfon 6y ago

Pardon my ignorance here, I have not worked with these kinds of large scale systems. But the first stack I would reach for to handle a queue would be erlang/otp (or maybe elixir), is that not designed to handle large queues without failure? Again I am absolutely no expert but I'd like to hear the views on this.

dgritsko 6y ago
RabbitMQ is written in Erlang.

nitrogen 6y ago

limited observability of celery workers and RabbitMQ RabbitMQ has very good observability. It has built-in web dashboards as well as an API. Every day that I had to bang my head against the opacity of an org-wide Kafka or ActiveMQ setup at one job, I missed the simplicity and transparency of RabbitMQ from a previous job. Somehow Kafka earned the "trendy" badge though, so everyone just uses it by default.

quazar987 6y ago

Indeed In absence of any significant stats its safe to assume their decision were driver by this two statements from article. "There were no in-house Celery or RabbitMQ experts at DoorDash who we could lean on to help devise a scaling strategy for this technology." "DoorDash had in-house Kafka expertise"