3 ms·
I'm not the OP, but changing the number of Kafka partitions isn't a super graceful operation. You would be wise to add as many as you could reasonably need assu
by ryanworl 9y ago
I'm not the OP, but changing the number of Kafka partitions isn't a super graceful operation. You would be wise to add as many as you could reasonably need assuming one consumer thread per partition. But not too many because each one is at least two files on disk!
- tedmiston 9y agoI haven't used Kafka yet, but a Kafka partition is roughly the same as a Kinesis shard, right?
- ryanworl 9y agoYes, but you have to set them up upfront and there is no API to split and merge partitions. If you add more partitions, it doesn't automatically re-shard for you either. I don't think it should automatically re-shard as it would cause a ton of disk and network IO, but just something to be aware of.