5 ms·

Can someone please explain why to use Kafka over Redis streams? Is it basically use Kafka if you need your data long term and/or if it's way too big for an in-m

Can someone please explain why to use Kafka over Redis streams? Is it basically use Kafka if you need your data long term and/or if it's way too big for an in-memory store? Redis seems much, much simpler to setup. Even if you do need to keep your events long term, why not use something like Eventstore?

xgb84j 5y ago
4 years ago at work we compared Kafka to alternatives and ended up using it. In my opinion the only reason for using Kafka is that you need a very very highly performant message queue. It had bad usability at the time and some very annoying undocumented behaviour (e.g. there was no official guide or feature to permanently remove a node). It feels like whenever Kafka developers had to make any tradeoffs they chose performance over everything else. I would only ever use it again if I the sheer data volume makes it impossible to use an alternative.
- thibauts 5y ago
  With a friend we’ve built this [1]. Usability has been our main focus, though we ended up having very good perfs (millions of mps out of the box, usually 2x to 10x kafka). We provide REST and WebSocket APIs, + a simple to implement binary protocol for the most demanding workloads. It’s in Go so very light and rock solid in stress tests. We provide trivial to implement and understandable usage patterns for exactly once processing. We’re currently working on HA and cloud integration. And we’re looking for community feedback BTW ! :) [1] https://github.com/dataptive/styx https://github.com/dataptive/styx
saynay 5y ago
Kafka is more focused on distributed use-case, I think. It is built around being able to write to fault-tolerant queues, and having multiple consumers in a group reading from it; you can have it configured so that losing a queue node will not lose availability of any of the data in the queue, and losing a consumer will not drop any messages. All of that while being able to handle very high throughput, even with persistence. The tradeoff is in the baseline complexity of Kafka is much higher. As far as I know, Redis Streams does not offer all of that on its own. You could certainly build a lot of that for Redis yourself (and someone probably has already), but then you start getting back that complexity. So if you have high requirements for performance, durability and availability, are don't mind the added complexity then Kafka is worth a look.
- yeswecatan 5y ago
  From my limited research, Redis streams offers pretty good durability if you run with AOF and sync frequently. Performance should be great since it's all in memory. As for availability, there is sentintel and cluster mode which I know nothing about. edit: the differences in consumer groups confuse me as well. With Redis streams, if you have a single stream with a single consumer group with multiple consumers, each consumer will get a new message at different times so processing of each message may happen out of order. That seems... fine to me. Apparently that is not the case with Kafka which makes me ask, why have multiple consumers in a single group if one will be blocked on the other?
  - manigandham 5y ago
    Kafka consumer groups do not block. Kafka uses partitions to scale out a single stream/topic and a consumer group allows consumers to exclusively read a partition. Multiple consumers can read from multiple partitions simultaneously, as long as there are enough partitions available. Redis doesn't have partitions, and instead has a single host managing a stream, while taking on all the consumer tracking functionality. It's fast, but not as scalable, and eventually if you keep growing then scale is how you get speed.
    - yeswecatan 5y ago
      If a topic is split across multiple partitions and we have multiple consumers, won't messages be read out of order?
      - cmorgan31 5y ago
        Order guarantees is not part of the promise with Kafka. If your consumer drops or times out the broker will pause all active consumers for the topic, revoke assignments and issue their new assigned blocks based on the new consumer group acknowledgement count.
      - manigandham 5y ago
        Messages within a partition are read in order. Partitions themselves may be read simultaneously or out of order. This is the trade-off for scalability. You can compromise by using hashing or other logic to group related messages into the same partition. For example, hashing by user-id lets you process events for a particular user in order, while processing all events by users in parallel. If you can divide into logically consistent but globally isolated boundaries then this setup works very well.

xgb84j 5y ago

4 years ago at work we compared Kafka to alternatives and ended up using it. In my opinion the only reason for using Kafka is that you need a very very highly performant message queue. It had bad usability at the time and some very annoying undocumented behaviour (e.g. there was no official guide or feature to permanently remove a node). It feels like whenever Kafka developers had to make any tradeoffs they chose performance over everything else. I would only ever use it again if I the sheer data volume makes it impossible to use an alternative.

thibauts 5y ago
With a friend we’ve built this [1]. Usability has been our main focus, though we ended up having very good perfs (millions of mps out of the box, usually 2x to 10x kafka). We provide REST and WebSocket APIs, + a simple to implement binary protocol for the most demanding workloads. It’s in Go so very light and rock solid in stress tests. We provide trivial to implement and understandable usage patterns for exactly once processing. We’re currently working on HA and cloud integration. And we’re looking for community feedback BTW ! :) [1] https://github.com/dataptive/styx https://github.com/dataptive/styx

saynay 5y ago

Kafka is more focused on distributed use-case, I think. It is built around being able to write to fault-tolerant queues, and having multiple consumers in a group reading from it; you can have it configured so that losing a queue node will not lose availability of any of the data in the queue, and losing a consumer will not drop any messages. All of that while being able to handle very high throughput, even with persistence. The tradeoff is in the baseline complexity of Kafka is much higher. As far as I know, Redis Streams does not offer all of that on its own. You could certainly build a lot of that for Redis yourself (and someone probably has already), but then you start getting back that complexity. So if you have high requirements for performance, durability and availability, are don't mind the added complexity then Kafka is worth a look.

yeswecatan 5y ago
From my limited research, Redis streams offers pretty good durability if you run with AOF and sync frequently. Performance should be great since it's all in memory. As for availability, there is sentintel and cluster mode which I know nothing about. edit: the differences in consumer groups confuse me as well. With Redis streams, if you have a single stream with a single consumer group with multiple consumers, each consumer will get a new message at different times so processing of each message may happen out of order. That seems... fine to me. Apparently that is not the case with Kafka which makes me ask, why have multiple consumers in a single group if one will be blocked on the other?
- manigandham 5y ago
  Kafka consumer groups do not block. Kafka uses partitions to scale out a single stream/topic and a consumer group allows consumers to exclusively read a partition. Multiple consumers can read from multiple partitions simultaneously, as long as there are enough partitions available. Redis doesn't have partitions, and instead has a single host managing a stream, while taking on all the consumer tracking functionality. It's fast, but not as scalable, and eventually if you keep growing then scale is how you get speed.
  - yeswecatan 5y ago
    If a topic is split across multiple partitions and we have multiple consumers, won't messages be read out of order?
    - cmorgan31 5y ago
      Order guarantees is not part of the promise with Kafka. If your consumer drops or times out the broker will pause all active consumers for the topic, revoke assignments and issue their new assigned blocks based on the new consumer group acknowledgement count.
    - manigandham 5y ago
      Messages within a partition are read in order. Partitions themselves may be read simultaneously or out of order. This is the trade-off for scalability. You can compromise by using hashing or other logic to group related messages into the same partition. For example, hashing by user-id lets you process events for a particular user in order, while processing all events by users in parallel. If you can divide into logically consistent but globally isolated boundaries then this setup works very well.