3 ms·

Kafka is more focused on distributed use-case, I think. It is built around being able to write to fault-tolerant queues, and having multiple consumers in a grou

by saynay 5y ago

Kafka is more focused on distributed use-case, I think. It is built around being able to write to fault-tolerant queues, and having multiple consumers in a group reading from it; you can have it configured so that losing a queue node will not lose availability of any of the data in the queue, and losing a consumer will not drop any messages. All of that while being able to handle very high throughput, even with persistence. The tradeoff is in the baseline complexity of Kafka is much higher. As far as I know, Redis Streams does not offer all of that on its own. You could certainly build a lot of that for Redis yourself (and someone probably has already), but then you start getting back that complexity. So if you have high requirements for performance, durability and availability, are don't mind the added complexity then Kafka is worth a look.

yeswecatan 5y ago
From my limited research, Redis streams offers pretty good durability if you run with AOF and sync frequently. Performance should be great since it's all in memory. As for availability, there is sentintel and cluster mode which I know nothing about. edit: the differences in consumer groups confuse me as well. With Redis streams, if you have a single stream with a single consumer group with multiple consumers, each consumer will get a new message at different times so processing of each message may happen out of order. That seems... fine to me. Apparently that is not the case with Kafka which makes me ask, why have multiple consumers in a single group if one will be blocked on the other?
- manigandham 5y ago
  Kafka consumer groups do not block. Kafka uses partitions to scale out a single stream/topic and a consumer group allows consumers to exclusively read a partition. Multiple consumers can read from multiple partitions simultaneously, as long as there are enough partitions available. Redis doesn't have partitions, and instead has a single host managing a stream, while taking on all the consumer tracking functionality. It's fast, but not as scalable, and eventually if you keep growing then scale is how you get speed.
  - yeswecatan 5y ago
    If a topic is split across multiple partitions and we have multiple consumers, won't messages be read out of order?
    - cmorgan31 5y ago
      Order guarantees is not part of the promise with Kafka. If your consumer drops or times out the broker will pause all active consumers for the topic, revoke assignments and issue their new assigned blocks based on the new consumer group acknowledgement count.
    - manigandham 5y ago
      Messages within a partition are read in order. Partitions themselves may be read simultaneously or out of order. This is the trade-off for scalability. You can compromise by using hashing or other logic to group related messages into the same partition. For example, hashing by user-id lets you process events for a particular user in order, while processing all events by users in parallel. If you can divide into logically consistent but globally isolated boundaries then this setup works very well.

by saynay 5y ago

yeswecatan 5y ago

From my limited research, Redis streams offers pretty good durability if you run with AOF and sync frequently. Performance should be great since it's all in memory. As for availability, there is sentintel and cluster mode which I know nothing about. edit: the differences in consumer groups confuse me as well. With Redis streams, if you have a single stream with a single consumer group with multiple consumers, each consumer will get a new message at different times so processing of each message may happen out of order. That seems... fine to me. Apparently that is not the case with Kafka which makes me ask, why have multiple consumers in a single group if one will be blocked on the other?

manigandham 5y ago
Kafka consumer groups do not block. Kafka uses partitions to scale out a single stream/topic and a consumer group allows consumers to exclusively read a partition. Multiple consumers can read from multiple partitions simultaneously, as long as there are enough partitions available. Redis doesn't have partitions, and instead has a single host managing a stream, while taking on all the consumer tracking functionality. It's fast, but not as scalable, and eventually if you keep growing then scale is how you get speed.
- yeswecatan 5y ago
  If a topic is split across multiple partitions and we have multiple consumers, won't messages be read out of order?
  - cmorgan31 5y ago
    Order guarantees is not part of the promise with Kafka. If your consumer drops or times out the broker will pause all active consumers for the topic, revoke assignments and issue their new assigned blocks based on the new consumer group acknowledgement count.
  - manigandham 5y ago
    Messages within a partition are read in order. Partitions themselves may be read simultaneously or out of order. This is the trade-off for scalability. You can compromise by using hashing or other logic to group related messages into the same partition. For example, hashing by user-id lets you process events for a particular user in order, while processing all events by users in parallel. If you can divide into logically consistent but globally isolated boundaries then this setup works very well.