4 ms·

From my limited research, Redis streams offers pretty good durability if you run with AOF and sync frequently. Performance should be great since it's all in mem

From my limited research, Redis streams offers pretty good durability if you run with AOF and sync frequently. Performance should be great since it's all in memory. As for availability, there is sentintel and cluster mode which I know nothing about. edit: the differences in consumer groups confuse me as well. With Redis streams, if you have a single stream with a single consumer group with multiple consumers, each consumer will get a new message at different times so processing of each message may happen out of order. That seems... fine to me. Apparently that is not the case with Kafka which makes me ask, why have multiple consumers in a single group if one will be blocked on the other?

manigandham 5y ago
Kafka consumer groups do not block. Kafka uses partitions to scale out a single stream/topic and a consumer group allows consumers to exclusively read a partition. Multiple consumers can read from multiple partitions simultaneously, as long as there are enough partitions available. Redis doesn't have partitions, and instead has a single host managing a stream, while taking on all the consumer tracking functionality. It's fast, but not as scalable, and eventually if you keep growing then scale is how you get speed.
- yeswecatan 5y ago
  If a topic is split across multiple partitions and we have multiple consumers, won't messages be read out of order?
  - cmorgan31 5y ago
    Order guarantees is not part of the promise with Kafka. If your consumer drops or times out the broker will pause all active consumers for the topic, revoke assignments and issue their new assigned blocks based on the new consumer group acknowledgement count.
  - manigandham 5y ago
    Messages within a partition are read in order. Partitions themselves may be read simultaneously or out of order. This is the trade-off for scalability. You can compromise by using hashing or other logic to group related messages into the same partition. For example, hashing by user-id lets you process events for a particular user in order, while processing all events by users in parallel. If you can divide into logically consistent but globally isolated boundaries then this setup works very well.

manigandham 5y ago

Kafka consumer groups do not block. Kafka uses partitions to scale out a single stream/topic and a consumer group allows consumers to exclusively read a partition. Multiple consumers can read from multiple partitions simultaneously, as long as there are enough partitions available. Redis doesn't have partitions, and instead has a single host managing a stream, while taking on all the consumer tracking functionality. It's fast, but not as scalable, and eventually if you keep growing then scale is how you get speed.

yeswecatan 5y ago
If a topic is split across multiple partitions and we have multiple consumers, won't messages be read out of order?
- cmorgan31 5y ago
  Order guarantees is not part of the promise with Kafka. If your consumer drops or times out the broker will pause all active consumers for the topic, revoke assignments and issue their new assigned blocks based on the new consumer group acknowledgement count.
- manigandham 5y ago
  Messages within a partition are read in order. Partitions themselves may be read simultaneously or out of order. This is the trade-off for scalability. You can compromise by using hashing or other logic to group related messages into the same partition. For example, hashing by user-id lets you process events for a particular user in order, while processing all events by users in parallel. If you can divide into logically consistent but globally isolated boundaries then this setup works very well.