4 ms·

I enjoyed the read, but it glossed over some really important details: > Failovers took more than 20 minutes to complete and would often get stuck requiring m

I enjoyed the read, but it glossed over some really important details: > Failovers took more than 20 minutes to complete and would often get stuck requiring manual intervention. Messages were often lost in the process as well. Data loss while RabbitMQ is down is certainly a problem. > The Kafka-consumer process is responsible for fetching messages from Kafka, and placing them on a local queue that is read by the task-execution processes. This is a bit confusing, so now instead of a centralized outage, grinding everything to a halt, the risk is distributed data loss as local queues go down? As long as they stay down, it should be limited data loss to the size of the queues times the number of consumers, but if an instance is flagging it could eat through a whole lot of messages before someone notices the problem. I guess it’s really dependent on the types of messages being processed and if they are idempotent enough to be replayed without consequence.

secondcoming 6y ago
Yes, this makes little to me. It seems that a slow task-executor can still cause the local queue to fill up. Each task-executor should have its own queue.
- rad_gruchalski 6y ago
  The difference being that in Kafka the topic partition is isolated and the ingest is separate from consumption. Slow consumer does not slow down ingest which means that your task executor is simply that - processing is slower but no data is lost because ingest is still operational. Kafka is not a queue. It's a distributed log. The difference is significant because in a queue, the message can be processed by a single worker within the single unit of time - message is picked, locked, nobody else can see it, after processing, message is either acked or nacked. In a log, multiple consumers can process the same message in parallel. If a message is bad, it is usually skipped by simply advancing the offset to the next offset. The log has the advantage that the reason for slow processing is easy to identify and isolate - it's the slow consumer. It's also very easy to monitor the consumption lag - simply look at how fast the maximum offset grows vs what's the current consumed offset (high / low watermark). If the difference grows, your consumer is too slow.
  - XorNot 6y ago
    This seems very much like saying that if you shoot the hostage in a hostage situation, you no longer have a hostage situation. Like...the whole point of having messages work that way is that you're not having to bake event processing and duplicate transaction handling and compensating transactions and all the other message handling logic into literally every process which comes after your message queue. It's a non-solution - you're just leaking the problem out to another part of the stack.
    - Joeri 6y ago
      It isn’t actually that complicated. The kafka client abstracts away the complexity of tracking offsets. You fetch batches of messages through the kafka client in a loop, process them one by one, and commit the batch’s offset if everything went well. If something goed wrong, after restarting the consumer the kafka client will make sure it gets offered the same batch again. So you’re in an at least once scenario where as long as you process messages in an idempotent way and skip the bad ones it is basically foolproof.
      - rad_gruchalski 6y ago
        Indeed. Plus it automatically handles consumer group rebelancing and partition leader changes. From the user perspective, one only needs to ensure having enough consumers in a consumer group. At Klarrio, we have a rather small cluster in production but we can easily push 2.4Gbps with <100ms end to end latency for dozens of consumers. It’s working really well.

secondcoming 6y ago

Yes, this makes little to me. It seems that a slow task-executor can still cause the local queue to fill up. Each task-executor should have its own queue.

rad_gruchalski 6y ago
The difference being that in Kafka the topic partition is isolated and the ingest is separate from consumption. Slow consumer does not slow down ingest which means that your task executor is simply that - processing is slower but no data is lost because ingest is still operational. Kafka is not a queue. It's a distributed log. The difference is significant because in a queue, the message can be processed by a single worker within the single unit of time - message is picked, locked, nobody else can see it, after processing, message is either acked or nacked. In a log, multiple consumers can process the same message in parallel. If a message is bad, it is usually skipped by simply advancing the offset to the next offset. The log has the advantage that the reason for slow processing is easy to identify and isolate - it's the slow consumer. It's also very easy to monitor the consumption lag - simply look at how fast the maximum offset grows vs what's the current consumed offset (high / low watermark). If the difference grows, your consumer is too slow.
- XorNot 6y ago
  This seems very much like saying that if you shoot the hostage in a hostage situation, you no longer have a hostage situation. Like...the whole point of having messages work that way is that you're not having to bake event processing and duplicate transaction handling and compensating transactions and all the other message handling logic into literally every process which comes after your message queue. It's a non-solution - you're just leaking the problem out to another part of the stack.
  - Joeri 6y ago
    It isn’t actually that complicated. The kafka client abstracts away the complexity of tracking offsets. You fetch batches of messages through the kafka client in a loop, process them one by one, and commit the batch’s offset if everything went well. If something goed wrong, after restarting the consumer the kafka client will make sure it gets offered the same batch again. So you’re in an at least once scenario where as long as you process messages in an idempotent way and skip the bad ones it is basically foolproof.
    - rad_gruchalski 6y ago
      Indeed. Plus it automatically handles consumer group rebelancing and partition leader changes. From the user perspective, one only needs to ensure having enough consumers in a consumer group. At Klarrio, we have a rather small cluster in production but we can easily push 2.4Gbps with <100ms end to end latency for dozens of consumers. It’s working really well.