3 ms·
Great post! In the last part of your blog you've mentioned: "To scale out we can increase number of partition in kafka topic and add more clockwork nodes." Abo
by palansh 7y ago
Great post! In the last part of your blog you've mentioned:
"To scale out we can increase number of partition in kafka topic and add more clockwork nodes."
Above has a limit. In general, more partitions in a Kafka cluster leads to higher throughput. However, one does have to be aware of the potential impact of having too many partitions in total or per broker on things like availability and latency.
How do we handle too many kafka partitions?
- uttpal 7y agoAdding more partitions will definitely have impact on latency and availability, and you will always have a trade-off between throughput and latency however, Clockwork will work fine for most of use cases as kafka supports upto 200k partitions per cluster (https://blogs.apache.org/kafka/entry/apache-kafka-supports-more-partitions https://blogs.apache.org/kafka/entry/apache-kafka-supports-m...). Beyond that we will need to have multi-cluster support, definitely in the pipeline wink