It does not, I've lost alot of time profiling Kafka perf issues against clusters on the exact same hardware with exact same traffic but with a 3000% throughput

It does not, I've lost alot of time profiling Kafka perf issues against clusters on the exact same hardware with exact same traffic but with a 3000% throughput difference. The root cause was one cluster had a lot of empty test topics Try benchmarking Kafka from 0 partitions to a few thousand partitions in 100 partition increments. The benchmark only needs to write to a single topic, using their provided producer perf tool while all other topics are inactive with zero data. As the partitions increase there is a very noticeable drop in throughout that looks to be linear. Kafka does not handle a large number of partitions well currently, large even being low thousands. It's easy to hit with just a few hundred topics. Reading between the lines ehen Linkdin and Netflix advertise several clusters, i am predicting/guessing they shard the data.