3 ms·

I sympathise - we’ve also experienced a bunch of issues with celery and rabbitmq and observing what’s actually happening is awful. I actually tweeted out some

I sympathise - we’ve also experienced a bunch of issues with celery and rabbitmq and observing what’s actually happening is awful. I actually tweeted out some similar issues a week ago: https://twitter.com/jarshwah/status/1310820638655877120 https://twitter.com/jarshwah/status/1310820638655877120 There are ways to scale rabbitmq past 1 node without giving up too much throughput. - use multiple queues as each queue is single threaded - use sharding plugins which present a single queue externally but use multiple queues across a cluster internally - use lazy queues to get messages straight to disk, lower throughput but higher message count It doesn’t really sound like the other options were seriously considered and it was more about moving away from celery. That’s fine - just be honest about it.

jojo2000 6y ago
+1. I also think that their problem was more celery-related that rabbitmq related, but that's just a guess. We also evaluated kafka vs. rabbitmq, and chose rabbitmq. For people hitting performance degradation with rabbitmq this is a good reference [0] RabbitMQ should be used as-is without a middleman as much as possible, to make the best possible use of all of its concepts. [0] https://www.cloudamqp.com/blog/2018-01-19-part4-rabbitmq-13-common-errors.html https://www.cloudamqp.com/blog/2018-01-19-part4-rabbitmq-13-...
zbentley 6y ago
Per your tweet, yes, running without publish confirmations is a major unpleasant surprise that folks eventually run into. However, that's less Celery's fault and more RabbitMQ's: pub confirms are a Rabbit-specific extension to their protocol, and are left off in most example implementations. MongoDB, also, chooses horrible default settings; however, I wouldn't blame a Mongo-backed product entirely for being hackable. Additionally (I mentioned this in a sibling thread), Kafka's publish behavior is even worse. The vast majority of Kafka publishers' default settings don't even send the data to the broker without waiting for a response (like RabbitMQ without publish confirms); they send the data to a local buffer which is asynchronously flushed periodically/volumetrically. If your process crashes, you lose the buffer. Unpleasantly, if you force a buffer flush on every publish, Kafka's ingest rate goes from 10-100x RabbitMQ's to 0.01-0.1x RabbitMQ's.
- synthc 6y ago
  >Kafka's publish behavior is even worse. The vast majority of Kafka publishers' default settings don't even send the data to the broker without waiting for a response (like RabbitMQ without publish confirms); they send the data to a local buffer which is asynchronously flushed periodically/volumetrically. When you publish a message to Kafka, it stores the message in a buffer and immediately returns a future that completes once the data is sent to the broker. If you forget to wait on completion of the future that is on you, this behaviour is well documented.
  - zbentley 6y ago
    > If you forget to wait on completion of the future that is on you Absolutely. However, data loss and edge cases emerge where process crashes and abrupt exits are concerned. With RabbitMQ, even with pub confirms disabled, the odds are that all of your publishes (sans, perhaps, the one you were in the middle of at the instant of exit) will make it to the broker. With Kafka, every message not yet flushed from the buffer will be dropped. I get that this is expected/documented, and I get why it exists (batchwise operations allow way higher net throughput into Kafka). However, I've encountered many groups of engineers from many companies who were surprised by and unable to effectively work around this behavior. Workarounds include performance sacrifices (flushing per-publish); complex signal handling semantics to coordinate flushes (complicated further by errors that occur if a signal arrives during flush); and atexit(3) flush hooks (terrifying, given the way some higher level programming languages' garbage collection interacts with atexit). At the end of the day, it matters how friendly a tool is when it's held by a novice. While there are many good reasons to use Kafka, that's not one of them.
    - deleted 6y ago
      [deleted]
- jsmeaton 6y ago
  Agree that lack of confirms is less a celery problem and more a rabbit problem. My main gripe is that it’s not documented within celery - anywhere. Not even how to enable them. If celery is going to go to the lengths it does to abstract brokers then it should also mention required configuration for reliability.

jojo2000 6y ago

+1. I also think that their problem was more celery-related that rabbitmq related, but that's just a guess. We also evaluated kafka vs. rabbitmq, and chose rabbitmq. For people hitting performance degradation with rabbitmq this is a good reference [0] RabbitMQ should be used as-is without a middleman as much as possible, to make the best possible use of all of its concepts. [0] https://www.cloudamqp.com/blog/2018-01-19-part4-rabbitmq-13-common-errors.html https://www.cloudamqp.com/blog/2018-01-19-part4-rabbitmq-13-...

zbentley 6y ago

Per your tweet, yes, running without publish confirmations is a major unpleasant surprise that folks eventually run into. However, that's less Celery's fault and more RabbitMQ's: pub confirms are a Rabbit-specific extension to their protocol, and are left off in most example implementations. MongoDB, also, chooses horrible default settings; however, I wouldn't blame a Mongo-backed product entirely for being hackable. Additionally (I mentioned this in a sibling thread), Kafka's publish behavior is even worse. The vast majority of Kafka publishers' default settings don't even send the data to the broker without waiting for a response (like RabbitMQ without publish confirms); they send the data to a local buffer which is asynchronously flushed periodically/volumetrically. If your process crashes, you lose the buffer. Unpleasantly, if you force a buffer flush on every publish, Kafka's ingest rate goes from 10-100x RabbitMQ's to 0.01-0.1x RabbitMQ's.

synthc 6y ago
>Kafka's publish behavior is even worse. The vast majority of Kafka publishers' default settings don't even send the data to the broker without waiting for a response (like RabbitMQ without publish confirms); they send the data to a local buffer which is asynchronously flushed periodically/volumetrically. When you publish a message to Kafka, it stores the message in a buffer and immediately returns a future that completes once the data is sent to the broker. If you forget to wait on completion of the future that is on you, this behaviour is well documented.
- zbentley 6y ago
  > If you forget to wait on completion of the future that is on you Absolutely. However, data loss and edge cases emerge where process crashes and abrupt exits are concerned. With RabbitMQ, even with pub confirms disabled, the odds are that all of your publishes (sans, perhaps, the one you were in the middle of at the instant of exit) will make it to the broker. With Kafka, every message not yet flushed from the buffer will be dropped. I get that this is expected/documented, and I get why it exists (batchwise operations allow way higher net throughput into Kafka). However, I've encountered many groups of engineers from many companies who were surprised by and unable to effectively work around this behavior. Workarounds include performance sacrifices (flushing per-publish); complex signal handling semantics to coordinate flushes (complicated further by errors that occur if a signal arrives during flush); and atexit(3) flush hooks (terrifying, given the way some higher level programming languages' garbage collection interacts with atexit). At the end of the day, it matters how friendly a tool is when it's held by a novice. While there are many good reasons to use Kafka, that's not one of them.
  - deleted 6y ago
    [deleted]
jsmeaton 6y ago
Agree that lack of confirms is less a celery problem and more a rabbit problem. My main gripe is that it’s not documented within celery - anywhere. Not even how to enable them. If celery is going to go to the lengths it does to abstract brokers then it should also mention required configuration for reliability.