4 ms·
To the author (or anyone else with experience): Any insight on why you guys chose Kafka + Secor over Kinesis?
by matsur 11y ago
To the author (or anyone else with experience): Any insight on why you guys chose Kafka + Secor over Kinesis?
- jonathankoren 11y agoI worked with Anuj at LinkedIn, so I'm thinking the most pedestrian answer is that Kafka works, and it's a familiar tool.
- goyalanuj 11y agoThank you Jonathan! As Jonathan mentioned, we made this decision around 9 months back and at that time Kinesis wasn't as mature and had less flexibility around retention period etc. Kafka is very reliable (as I had seen it handling billions of events a day at LinkedIn) and has a huge open-source community around it. At IFTTT, we always prefer to use and contribute to open source ( http://engineering.ifttt.com/oss/2015/07/23/open-source/ http://engineering.ifttt.com/oss/2015/07/23/open-source/ ).
- dekayed 11y agoI'm assuming that you run Kafka within AWS. Much of the hardware requirements/suggestions I've seen for Kafka are all for non-virtualized environments. If you can get into it, could you share some details... - What is the size of your Kafka cluster - What instances types do you use? - Do you use EBS or use ephemeral storage? - How much do you over-provision to deal with instance loss? - Any other gotchas/considerations? Thanks!
- Splendor 11y agoThings might be different now that Kinesis Firehose has been announced. But Kinesis alone is not easier to manage than Kafka.
- lenguyenthedat 11y agoAmazon Kinesis is a great tool, the main problem is that it's relatively new while Kafka have been around for a few years already.