5 ms·
Ask HN: Why should one ever use AWS Kinesis when Kafka is out there?
I mean apart from the fact that Kinesis is a hosted service and Kafka would most likely require you setting up a cluster. Are there any other technical advantages in terms of features, performance, handling volume at scale, stream ordering, authentication etc.?
- anilgulecha 10y agoHosted vs unhosted really. I mean you can ask, why use EC2 instead of your own machine at the local data-center? Managed services typically already have taken design considerations for failover, DR, and scaling, so you don't have to.
- htn 10y agoWell, you can get hosted Kafka as a service as well. My company Aiven offers one such service in AWS/Azure/Google Cloud Platform, DigitalOcean and UpCloud (https://aiven.io/kafka https://aiven.io/kafka). CloudKarafka is an another vendor with managed offering and Heroku too just launched Kafka service as well. I think Kinesis is an excellent service, but if you have an existing stack or preference on OpenSource solutions, it's good to have choice.
- anilgulecha 10y agoHosted Kafka by a third party is not equivalent to (potential) Hosted Kafka by AWS. There's a lot of signalling/data at the back when AWS implements it, which you will never have access to.
- barracoda 10y agoAnother major upside to using Kinesis is how seamlessly it integrates with AWS based consumers such as S3/DynamoDB etc. On the technical side of things, for Kinesis "each shard can support up to 5 transactions per second for reads, up to a maximum total data read rate of 2 MB per second and up to 1,000 records per second for writes, up to a maximum total data write rate of 1 MB per second (including partition keys)" (from AWS documentation), i'm not sure if a hard limit exists for Kafka. W.r.t stream ordering kafka preserves ordering of messages within the same shard i guess which Kinesis also preserves. Again authentication is made easier with the rest of AWS provisioning (IAM) so that's easier to build into Kinesis based application. I think the bottom line is if you've other AWS components and are willing to be less hands-on about tuning with ease of setup and scalability, Kinesis is your best option. Else go with Kafka
- eb0la 10y agoNot a kinesis expert here, but think the need to reprocess stuff could be the key. With kafka you can reprocess data just creating a different consumer group. You pay for instances. I guess in kinesis you need to move data from kinsesis to s3 and back to kinsesis to reprocess it, and it wil surely have a cost.
- imauld 10y ago>> I mean apart from the fact that Kinesis is a hosted service and Kafka would most likely require you setting up a cluster. AFAIK that's the only we use Kinesis over something like Kafka or Spark (much to the chagrin of of our data team). We are planning on moving to Kafka I believe but in the meantime while the Data folks sort all that out we are using Kinesis.