3 ms·

You're missing something major. I have experience using Kafka and Kinesis. My curreny company only uses Kinesis and zero Kafka. Why? Because Kafka takes signifi

You're missing something major. I have experience using Kafka and Kinesis. My curreny company only uses Kinesis and zero Kafka. Why? Because Kafka takes significant knowledge, experience, time to setup, understand, and support. And if it fucks up and you don't have one or more "Kafka experts" on-call, well, you're screwed. Kinesis on the other hand, just works. Yes, it just works. I don't have to have a Kinesis expert on hand, I don't have to configure clusters myself or write Ansible/Puppet, etc. I have a few basic lines of Terraform to create my Kinesis streams and I push data to them and we got it working in minutes and we've had no issues. Contract this to my previous job where we literally had to hire multiple Kafka experts at high salaries to maintain our Kafka clusters. This is why you only use Kafka if you ABSOLUTELY NEED it.

jpgvm 6y ago
I am about 90% the way ridding a larger company of Kinesis. You might not have needed a Kafka expert but instead you had to create (likely poorly implemented) frameworks for consuming from Kinesis (when you could have just used Consumer Groups). Also Kinesis is insanely expensive, to the point that if your throughput is anything but trivial (in which case it doesn't matter which option you use, both will -just work-) that is just becomes untennable. I actually did miss something but it's not complexity, you can easily use hosted Kafka (AWS even provides it with MSK now). It's actually authentication. Kafka does have authentication but it doesn't easily tie into AWS or GCP authentication mechanisms without the help of a tool like Vault. That said.. I don't think that minor benefit is enough to ever justify Kinesis over Kafka.
- crescentfresh 6y ago
  > Kinesis is insanely expensive No, it is not. Also, an equivalent cluster of EC2 instances running zookeeper/Kafka/etc will outprice Kinesis.
  - jpgvm 6y ago
    At tiny throughput maybe? Pretty sure you could run Kafka on small ec2 instances and still get great throughput, especially if you don't need much retention. (reducing retention reduces need for disk I/O as consumers can't fall far behind causing disks to seek). I think for any reasonable workload, i.e 10k/s+ and/or throughput over 100mb/s Kinesis gets dumpstered for price every time.

jpgvm 6y ago

I am about 90% the way ridding a larger company of Kinesis. You might not have needed a Kafka expert but instead you had to create (likely poorly implemented) frameworks for consuming from Kinesis (when you could have just used Consumer Groups). Also Kinesis is insanely expensive, to the point that if your throughput is anything but trivial (in which case it doesn't matter which option you use, both will -just work-) that is just becomes untennable. I actually did miss something but it's not complexity, you can easily use hosted Kafka (AWS even provides it with MSK now). It's actually authentication. Kafka does have authentication but it doesn't easily tie into AWS or GCP authentication mechanisms without the help of a tool like Vault. That said.. I don't think that minor benefit is enough to ever justify Kinesis over Kafka.

crescentfresh 6y ago
> Kinesis is insanely expensive No, it is not. Also, an equivalent cluster of EC2 instances running zookeeper/Kafka/etc will outprice Kinesis.
- jpgvm 6y ago
  At tiny throughput maybe? Pretty sure you could run Kafka on small ec2 instances and still get great throughput, especially if you don't need much retention. (reducing retention reduces need for disk I/O as consumers can't fall far behind causing disks to seek). I think for any reasonable workload, i.e 10k/s+ and/or throughput over 100mb/s Kinesis gets dumpstered for price every time.