3 ms·
Interesting; I didn't know that Kafka had order-of-magnitude-better latency. Thanks for the data point. I wonder if this is something inherent to Kinesis's des
by stephen 11y ago
Interesting; I didn't know that Kafka had order-of-magnitude-better latency. Thanks for the data point.
I wonder if this is something inherent to Kinesis's design, or if Amazon will magically make it faster at some point in the future. Do you have a suspicion/indication either way?
- tylertreat 11y agoWell, for one, there is no streaming socket API, just HTTP. I suspect the 5 reads/sec limit on shards is mainly due to multitenancy, but I could see this increasing in the future. Amazon has already improved the latency of Kinesis a fair amount, but it still has a ways to go before it matches Kafka. We also have cases where we need many consumers reading a stream, so having that effectively limited to 5 concurrent consumers is a pretty tough limitation. Their solution to this is to just have consumers sleep on a failed poll, but that doesn't really help if you want to scale out your ingestion. You don't have that problem with Kafka.
- cbsmith 11y agoKinesis has gotten better about it's latencies, but it is inherently an HTTP protocol, which already gets you in pretty big trouble for serious real-time applications.