3 ms·

What's been your expiriance with regard to producer latency with Kinesis? My limited experimentation with it so far seems to point to that as a potential pain

by erikcw 11y ago

What's been your expiriance with regard to producer latency with Kinesis? My limited experimentation with it so far seems to point to that as a potential pain point for high throughout low latency systems.

tylertreat 11y ago
In my experience, Kinesis' performance makes it a nonstarter for low-latency systems and use cases with many consumers. Shards are limited to 5 reads/sec, so consumers are heavily throttled. Producers also have significant put latency. Kafka's latency and throughput are OOM better but of course come with the operational overhead. Depends on what your needs are, but Kafka is a better choice for "real-time" systems.
- stephen 11y ago
  Interesting; I didn't know that Kafka had order-of-magnitude-better latency. Thanks for the data point. I wonder if this is something inherent to Kinesis's design, or if Amazon will magically make it faster at some point in the future. Do you have a suspicion/indication either way?
  - tylertreat 11y ago
    Well, for one, there is no streaming socket API, just HTTP. I suspect the 5 reads/sec limit on shards is mainly due to multitenancy, but I could see this increasing in the future. Amazon has already improved the latency of Kinesis a fair amount, but it still has a ways to go before it matches Kafka. We also have cases where we need many consumers reading a stream, so having that effectively limited to 5 concurrent consumers is a pretty tough limitation. Their solution to this is to just have consumers sleep on a failed poll, but that doesn't really help if you want to scale out your ingestion. You don't have that problem with Kafka.
  - cbsmith 11y ago
    Kinesis has gotten better about it's latencies, but it is inherently an HTTP protocol, which already gets you in pretty big trouble for serious real-time applications.
stephen 11y ago
Generally 10s of seconds to low minutes. Two caveats: 1) we batch/queue writes on the producer side, and 2) our systems are more near-line than true low-latency, e.g. we scale the cluster up/down instead of keeping machines running to keep latency as low as possible. So YMMV. I'd be interested in your results, if you were shooting for a specific number and couldn't hit it, just for future reference.
keyzor 11y ago
I only ran some perf testing and haven't used Kinesis in production yet but perf testing I ran end to end was < 1 sec end to end for a couple of thousand records per second using 3 shards. The reads per second are a lot lower than the writes per second because readers are expected to batch reads - you get 1000 puts/sec/shard vs. 5 reads/sec/shard, but 1 MB/sec read throughput and 2 MB /sec write throughput in shard, so you can read faster than you can write (and catch up if you fall behind). You won't get low millisec or microsec responses because Kinesis is replicating messages to multiple stores for resiliency.

by erikcw 11y ago

tylertreat 11y ago

In my experience, Kinesis' performance makes it a nonstarter for low-latency systems and use cases with many consumers. Shards are limited to 5 reads/sec, so consumers are heavily throttled. Producers also have significant put latency. Kafka's latency and throughput are OOM better but of course come with the operational overhead. Depends on what your needs are, but Kafka is a better choice for "real-time" systems.

stephen 11y ago
Interesting; I didn't know that Kafka had order-of-magnitude-better latency. Thanks for the data point. I wonder if this is something inherent to Kinesis's design, or if Amazon will magically make it faster at some point in the future. Do you have a suspicion/indication either way?
- tylertreat 11y ago
  Well, for one, there is no streaming socket API, just HTTP. I suspect the 5 reads/sec limit on shards is mainly due to multitenancy, but I could see this increasing in the future. Amazon has already improved the latency of Kinesis a fair amount, but it still has a ways to go before it matches Kafka. We also have cases where we need many consumers reading a stream, so having that effectively limited to 5 concurrent consumers is a pretty tough limitation. Their solution to this is to just have consumers sleep on a failed poll, but that doesn't really help if you want to scale out your ingestion. You don't have that problem with Kafka.
- cbsmith 11y ago
  Kinesis has gotten better about it's latencies, but it is inherently an HTTP protocol, which already gets you in pretty big trouble for serious real-time applications.

stephen 11y ago

Generally 10s of seconds to low minutes. Two caveats: 1) we batch/queue writes on the producer side, and 2) our systems are more near-line than true low-latency, e.g. we scale the cluster up/down instead of keeping machines running to keep latency as low as possible. So YMMV. I'd be interested in your results, if you were shooting for a specific number and couldn't hit it, just for future reference.

keyzor 11y ago

I only ran some perf testing and haven't used Kinesis in production yet but perf testing I ran end to end was < 1 sec end to end for a couple of thousand records per second using 3 shards. The reads per second are a lot lower than the writes per second because readers are expected to batch reads - you get 1000 puts/sec/shard vs. 5 reads/sec/shard, but 1 MB/sec read throughput and 2 MB /sec write throughput in shard, so you can read faster than you can write (and catch up if you fall behind). You won't get low millisec or microsec responses because Kinesis is replicating messages to multiple stores for resiliency.