4 ms·
Kafka consumer groups do not block. Kafka uses partitions to scale out a single stream/topic and a consumer group allows consumers to exclusively read a partiti
by manigandham 5y ago
Kafka consumer groups do not block. Kafka uses partitions to scale out a single stream/topic and a consumer group allows consumers to exclusively read a partition. Multiple consumers can read from multiple partitions simultaneously, as long as there are enough partitions available.
Redis doesn't have partitions, and instead has a single host managing a stream, while taking on all the consumer tracking functionality. It's fast, but not as scalable, and eventually if you keep growing then scale is how you get speed.
- yeswecatan 5y agoIf a topic is split across multiple partitions and we have multiple consumers, won't messages be read out of order?
- cmorgan31 5y agoOrder guarantees is not part of the promise with Kafka. If your consumer drops or times out the broker will pause all active consumers for the topic, revoke assignments and issue their new assigned blocks based on the new consumer group acknowledgement count.
- manigandham 5y agoMessages within a partition are read in order. Partitions themselves may be read simultaneously or out of order. This is the trade-off for scalability. You can compromise by using hashing or other logic to group related messages into the same partition. For example, hashing by user-id lets you process events for a particular user in order, while processing all events by users in parallel. If you can divide into logically consistent but globally isolated boundaries then this setup works very well.