5 ms·

Wow, great comment. I ran into similar issues that DoorDash had with Celery/RabbitMQ a while back (~2013-2014) so I'm surprised the plague still continues. It's

Wow, great comment. I ran into similar issues that DoorDash had with Celery/RabbitMQ a while back (~2013-2014) so I'm surprised the plague still continues. It's for that reason that I pushed for going with SQS after that point. The way I saw it, a message queue is a really fundamental piece of reliable distributed systems. It's fundamentally stateful and I would like it to never go down and certainly never result in dropped messages/tasks. Like with an RDBMS, if I can pay someone else to solve that problem for a reasonable fee, I will rush to give them my money. The next time I built a queued task management system, I used SQS. In the past 7 years I've used it, it's generally worked like a dream. I can understand if for performance, cost optimization, transparency or portability reasons you still want to run your own RabbitMQ cluster. But my default bias is against that. So with that said, I'd like to ask a follow up question to this excellent comment: did you consider a managed queue like SQS? If so, why did you elect not to go with it? SQS even has a celery broker now (although certainly not as cleanly integrated as Redis or RabbitMQ).

zbentley 6y ago
Given the rate and size of messages we process, SQS presents a compelling "fallback" option (i.e. we need to route traffic when an irreplaceable RabbitMQ is down, pending human-initiated recovery), but not a compelling happy-path option because of cost, which you mentioned. Edit: additionally, there was an inertial component to that decision. Not only do some folks have pretty deep expertise with RabbitMQ, but by the time we started to hit significant limitations with it, we had devised pretty good (manual and automated) ways around those limitations, like the multi-broker approach. I can't predict what would have happened had we jumped to a totally different messaging technology when we ran up against those things, but I'm reasonably happy with where we ended up.
- yowlingcat 6y ago
  Yeah, that was my suspicion. From a cost optimization perspective, I don't think SQS will ever hit the level of price per instruction that RabbitMQ does. SQS also doesn't get anywhere near spitting distance on latency either. For my needs, though, it's always done a great job on reliable throughput. If you have folks with deep expertise with RabbitMQ, I can see why you'd go with it. It means that you can essentially neutralize its operational risks and maximize its strength, which is performance (as well as great integration codebase maturity).

zbentley 6y ago

Given the rate and size of messages we process, SQS presents a compelling "fallback" option (i.e. we need to route traffic when an irreplaceable RabbitMQ is down, pending human-initiated recovery), but not a compelling happy-path option because of cost, which you mentioned. Edit: additionally, there was an inertial component to that decision. Not only do some folks have pretty deep expertise with RabbitMQ, but by the time we started to hit significant limitations with it, we had devised pretty good (manual and automated) ways around those limitations, like the multi-broker approach. I can't predict what would have happened had we jumped to a totally different messaging technology when we ran up against those things, but I'm reasonably happy with where we ended up.

yowlingcat 6y ago
Yeah, that was my suspicion. From a cost optimization perspective, I don't think SQS will ever hit the level of price per instruction that RabbitMQ does. SQS also doesn't get anywhere near spitting distance on latency either. For my needs, though, it's always done a great job on reliable throughput. If you have folks with deep expertise with RabbitMQ, I can see why you'd go with it. It means that you can essentially neutralize its operational risks and maximize its strength, which is performance (as well as great integration codebase maturity).