3 ms·
Yes I agree if using extremely small partition size ~100mb and using Cruise Control it should be able to auto heal quickly. But you will still have broker with
by skyde 6y ago
Yes I agree if using extremely small partition size ~100mb and using Cruise Control it should be able to auto heal quickly.
But you will still have broker with like 100% cpu usage and high network usage while other are mostly idle.
Cruise control try to patch around the bad design of kafka but there is only little it can do.
Any broker in ISR going down still cause big latency spike , and if broker was leader for several partition all those partition will be down until the single cluster controller finish doing leader switch sequentially for all those partitions.
Typically on cluster with many partition of topic, each broker would be leader for ~1000 partition which mean it will take up to 15 minutes for leader switch to complete for all partitions.
- EdwardDiego 6y agoYep, agree that Pulsar is far better for dealing with large amounts of topics than Kafka. Doesn't mean that Kafka is broken, it just means it has limitations.