3 ms·

Yes, this makes little to me. It seems that a slow task-executor can still cause the local queue to fill up. Each task-executor should have its own queue.

rad_gruchalski 6y ago
The difference being that in Kafka the topic partition is isolated and the ingest is separate from consumption. Slow consumer does not slow down ingest which means that your task executor is simply that - processing is slower but no data is lost because ingest is still operational. Kafka is not a queue. It's a distributed log. The difference is significant because in a queue, the message can be processed by a single worker within the single unit of time - message is picked, locked, nobody else can see it, after processing, message is either acked or nacked. In a log, multiple consumers can process the same message in parallel. If a message is bad, it is usually skipped by simply advancing the offset to the next offset. The log has the advantage that the reason for slow processing is easy to identify and isolate - it's the slow consumer. It's also very easy to monitor the consumption lag - simply look at how fast the maximum offset grows vs what's the current consumed offset (high / low watermark). If the difference grows, your consumer is too slow.
- XorNot 6y ago
  This seems very much like saying that if you shoot the hostage in a hostage situation, you no longer have a hostage situation. Like...the whole point of having messages work that way is that you're not having to bake event processing and duplicate transaction handling and compensating transactions and all the other message handling logic into literally every process which comes after your message queue. It's a non-solution - you're just leaking the problem out to another part of the stack.
  - Joeri 6y ago
    It isn’t actually that complicated. The kafka client abstracts away the complexity of tracking offsets. You fetch batches of messages through the kafka client in a loop, process them one by one, and commit the batch’s offset if everything went well. If something goed wrong, after restarting the consumer the kafka client will make sure it gets offered the same batch again. So you’re in an at least once scenario where as long as you process messages in an idempotent way and skip the bad ones it is basically foolproof.
    - rad_gruchalski 6y ago
      Indeed. Plus it automatically handles consumer group rebelancing and partition leader changes. From the user perspective, one only needs to ensure having enough consumers in a consumer group. At Klarrio, we have a rather small cluster in production but we can easily push 2.4Gbps with <100ms end to end latency for dozens of consumers. It’s working really well.

rad_gruchalski 6y ago

The difference being that in Kafka the topic partition is isolated and the ingest is separate from consumption. Slow consumer does not slow down ingest which means that your task executor is simply that - processing is slower but no data is lost because ingest is still operational. Kafka is not a queue. It's a distributed log. The difference is significant because in a queue, the message can be processed by a single worker within the single unit of time - message is picked, locked, nobody else can see it, after processing, message is either acked or nacked. In a log, multiple consumers can process the same message in parallel. If a message is bad, it is usually skipped by simply advancing the offset to the next offset. The log has the advantage that the reason for slow processing is easy to identify and isolate - it's the slow consumer. It's also very easy to monitor the consumption lag - simply look at how fast the maximum offset grows vs what's the current consumed offset (high / low watermark). If the difference grows, your consumer is too slow.

XorNot 6y ago
This seems very much like saying that if you shoot the hostage in a hostage situation, you no longer have a hostage situation. Like...the whole point of having messages work that way is that you're not having to bake event processing and duplicate transaction handling and compensating transactions and all the other message handling logic into literally every process which comes after your message queue. It's a non-solution - you're just leaking the problem out to another part of the stack.
- Joeri 6y ago
  It isn’t actually that complicated. The kafka client abstracts away the complexity of tracking offsets. You fetch batches of messages through the kafka client in a loop, process them one by one, and commit the batch’s offset if everything went well. If something goed wrong, after restarting the consumer the kafka client will make sure it gets offered the same batch again. So you’re in an at least once scenario where as long as you process messages in an idempotent way and skip the bad ones it is basically foolproof.
  - rad_gruchalski 6y ago
    Indeed. Plus it automatically handles consumer group rebelancing and partition leader changes. From the user perspective, one only needs to ensure having enough consumers in a consumer group. At Klarrio, we have a rather small cluster in production but we can easily push 2.4Gbps with <100ms end to end latency for dozens of consumers. It’s working really well.