14 ms·

> Furthermore, in order to have multiple workers, Kinesis Streams require you to use multiple shards. Each worker will make claim to a shard. It helps if you t

> Furthermore, in order to have multiple workers, Kinesis Streams require you to use multiple shards. Each worker will make claim to a shard. It helps if you think of Kinesis as AWS Kafkaesque rather than a message queue, because then shards = partitions and how you work with a Kinesis stream makes a lot of sense. Multiple concurrent consumers? You're going to need shards/partitions. Now, question is, as a replacement for homerolled Kafka in EC2, or AWS's Managed Kafka, is it cheaper? So far, magic 8-ball says "benchmarking for your given use case needed". So far my experiments for our workload and patterns say - maybe, but more investigation is needed. I plan to role out a side-by-side Kafka and Kinesis experiment on a given topic and ascertain the costs. Although ultimately, like any distributed messaging system, you end up engineering to the foibles of the system - in Kinesis' case, it's the fact that it rounds each sent message (which could be a batch of records, or a single record) up to the nearest 5KB for billing that you have to engineer for.

jpgvm 7y ago
Kinesis is just a very very poor Kafka. They wanted to have a Kafka but they also had this silly idea that everything needed to be HTTP so they built a poor clone with a HTTP interface. Unless forced into it by some other system there is never any reason to consider Kinesis over Kafka or Pulsar.
- EdwardDiego 7y ago
  Only reason I'm looking into it is cost engineering tbh. As to the Kafka vs. Pulsar, I've been running a Kafka cluster for several years now, and was interested in Pulsar as a Kafka++, and have been evaluating it for a client who wants to choose between one of the two, and at the moment, I think Pulsar needs about another year before I'd recommend it. It's exciting tech, but as is normal of recently open sourced projects, there's a lot of bugs being surfaced, and the documentation has a lot of unanswered questions, like, if I'm consuming a multi-partitioned topic in an exclusive subscription, what does that mean for ordering? I think it has some great ideas (especially decoupling brokers from storage), but yeah, it feels a bit like Kafka pre 0.8, interesting, but you're taking on a lot of work adopting it at the moment.
  - jpgvm 7y ago
    I think saying it's analogous to Kafka 0.8 is reasonably apt. It's very stable and performant in the verticals it has been used in at Yahoo but not as widely integrated, understood and optimised as Kafka across a broader set of use cases. That said if what you are trying to do fits squarely in the box of things that do work well it's considerably better than Kafka in a few keys ways. The first is the obvious separation of storage and compute/broker responsibilities, the benefits of which also led to the same design being used in LogDevice - a similar system designed and built completely independently and at roughly the same time at Facebook. The second is selective acknowledgement, i.e the ability to acknowledge messages as processed by a consumer out of order rather than merely the offset of the latest message processed. This allows Pulsar to be more easily used as a workqueue without the multitude of hacks and layered infrastructure required to get the same out of Kafka. Shared subscriptions/partitioning model. Compared to Kafka its more flexible, less punishing and less beholden to the architecture of consumers. Finally I would say tiering, for some it means nothing but depending on your use case it can be defining. Pulsar can offload historical segements to long term stable storage but still present a unified offset like API to consume historical data. So I think for some Pulsar provides enough benefit to make up for any shortcomings in integration as long as you have sufficient engineers able to debug/patch issues.
    - EdwardDiego 7y ago
      I agree wholeheartedly, especially with the last paragraph :)
- enitihas 7y ago
  What do you think are the major advantages of Kafka over Kinesis?
  - EdwardDiego 7y ago
    As I'm evaluating the other way, I'd be curious if you see any major advantages of Kinesis over Kafka. I'm not wedded to one or the other, just interested, as I said in another comment, I'm looking at Kinesis from a cost engineering basis alone, but if there's something I've overlooked, would be keen to know :)
  - jpgvm 7y ago
    It's actually awfully simple. Take everything Kinesis does and Kafka does it better. Yes, everything. Then ontop of that add Consumer Groups which basically deal with the issues in the OP w.r.t consuming a topic from multiple processes along with providing administrative APIs to reset application offsets, inspect lag etc. Also a bunch of extra features like transactions etc but if you are comparing on the basis of Kinesis like features they aren't likely to matter as much as the core functionality - which is where Kinesis really gets destroyed anyway. Normally I wouldn't take such an absolutist position but when it comes to Kinesis let me repeat in no uncertain terms. Never use Kinesis.
    - saberience 7y ago
      You're missing something major. I have experience using Kafka and Kinesis. My curreny company only uses Kinesis and zero Kafka. Why? Because Kafka takes significant knowledge, experience, time to setup, understand, and support. And if it fucks up and you don't have one or more "Kafka experts" on-call, well, you're screwed. Kinesis on the other hand, just works. Yes, it just works. I don't have to have a Kinesis expert on hand, I don't have to configure clusters myself or write Ansible/Puppet, etc. I have a few basic lines of Terraform to create my Kinesis streams and I push data to them and we got it working in minutes and we've had no issues. Contract this to my previous job where we literally had to hire multiple Kafka experts at high salaries to maintain our Kafka clusters. This is why you only use Kafka if you ABSOLUTELY NEED it.

jpgvm 7y ago

Kinesis is just a very very poor Kafka. They wanted to have a Kafka but they also had this silly idea that everything needed to be HTTP so they built a poor clone with a HTTP interface. Unless forced into it by some other system there is never any reason to consider Kinesis over Kafka or Pulsar.

EdwardDiego 7y ago
Only reason I'm looking into it is cost engineering tbh. As to the Kafka vs. Pulsar, I've been running a Kafka cluster for several years now, and was interested in Pulsar as a Kafka++, and have been evaluating it for a client who wants to choose between one of the two, and at the moment, I think Pulsar needs about another year before I'd recommend it. It's exciting tech, but as is normal of recently open sourced projects, there's a lot of bugs being surfaced, and the documentation has a lot of unanswered questions, like, if I'm consuming a multi-partitioned topic in an exclusive subscription, what does that mean for ordering? I think it has some great ideas (especially decoupling brokers from storage), but yeah, it feels a bit like Kafka pre 0.8, interesting, but you're taking on a lot of work adopting it at the moment.
- jpgvm 7y ago
  I think saying it's analogous to Kafka 0.8 is reasonably apt. It's very stable and performant in the verticals it has been used in at Yahoo but not as widely integrated, understood and optimised as Kafka across a broader set of use cases. That said if what you are trying to do fits squarely in the box of things that do work well it's considerably better than Kafka in a few keys ways. The first is the obvious separation of storage and compute/broker responsibilities, the benefits of which also led to the same design being used in LogDevice - a similar system designed and built completely independently and at roughly the same time at Facebook. The second is selective acknowledgement, i.e the ability to acknowledge messages as processed by a consumer out of order rather than merely the offset of the latest message processed. This allows Pulsar to be more easily used as a workqueue without the multitude of hacks and layered infrastructure required to get the same out of Kafka. Shared subscriptions/partitioning model. Compared to Kafka its more flexible, less punishing and less beholden to the architecture of consumers. Finally I would say tiering, for some it means nothing but depending on your use case it can be defining. Pulsar can offload historical segements to long term stable storage but still present a unified offset like API to consume historical data. So I think for some Pulsar provides enough benefit to make up for any shortcomings in integration as long as you have sufficient engineers able to debug/patch issues.
  - EdwardDiego 7y ago
    I agree wholeheartedly, especially with the last paragraph :)
enitihas 7y ago
What do you think are the major advantages of Kafka over Kinesis?
- EdwardDiego 7y ago
  As I'm evaluating the other way, I'd be curious if you see any major advantages of Kinesis over Kafka. I'm not wedded to one or the other, just interested, as I said in another comment, I'm looking at Kinesis from a cost engineering basis alone, but if there's something I've overlooked, would be keen to know :)
- jpgvm 7y ago
  It's actually awfully simple. Take everything Kinesis does and Kafka does it better. Yes, everything. Then ontop of that add Consumer Groups which basically deal with the issues in the OP w.r.t consuming a topic from multiple processes along with providing administrative APIs to reset application offsets, inspect lag etc. Also a bunch of extra features like transactions etc but if you are comparing on the basis of Kinesis like features they aren't likely to matter as much as the core functionality - which is where Kinesis really gets destroyed anyway. Normally I wouldn't take such an absolutist position but when it comes to Kinesis let me repeat in no uncertain terms. Never use Kinesis.
  - saberience 7y ago
    You're missing something major. I have experience using Kafka and Kinesis. My curreny company only uses Kinesis and zero Kafka. Why? Because Kafka takes significant knowledge, experience, time to setup, understand, and support. And if it fucks up and you don't have one or more "Kafka experts" on-call, well, you're screwed. Kinesis on the other hand, just works. Yes, it just works. I don't have to have a Kinesis expert on hand, I don't have to configure clusters myself or write Ansible/Puppet, etc. I have a few basic lines of Terraform to create my Kinesis streams and I push data to them and we got it working in minutes and we've had no issues. Contract this to my previous job where we literally had to hire multiple Kafka experts at high salaries to maintain our Kafka clusters. This is why you only use Kafka if you ABSOLUTELY NEED it.