3 ms·

Your main resource contention there would be network throughput, then broker CPU time in iowait if there's a lot of concurrent large disk reads happening on the

Your main resource contention there would be network throughput, then broker CPU time in iowait if there's a lot of concurrent large disk reads happening on the same broker (although Kafka's optimised for sequential reads). Partition count only limits concurrent consumption within a single consumer group. One consumer group won't impact another unless its consumers are doing sufficiently bad things to bottleneck the network or disk.

morelisp 4y ago
> Your main resource contention there would be network throughput, then broker CPU time The default consumer read sizes are so small you will hit broker CPU and worker thread limits long before network throughput. (Both consumer batch sizes and broker threads can be increased trivially but there's not much documentation around when to do this.)
- EdwardDiego 4y ago
  The default fetch.min.bytes is 1, that's true, but the default fetch.max.bytes is 50 MiB. And fetching small amounts of data repeatedly doesn't impose much overhead unless you're deliberately disconnecting and reconnecting between polls. And I assure you, you can definitely bottleneck network before CPU and/or network threads, Kafka was literally designed for very large numbers of consumers. And as for tuning, Kafka The Definitive Guide is pretty much as the name suggests. I've been recommending it very strongly, especially the chapters on monitoring and cluster replication, for years. You can download a free draft copy of the 2nd edition from Confluent, check it out :)
  - morelisp 4y ago
    > but the default fetch.max.bytes is 50 MiB. But max.partition.fetch.bytes is only 1MB.
    - EdwardDiego 4y ago
      Yep, it's aligned with the default max.message.bytes, largest batch size the broker will accept. 1 MiB is allegedly the ideal batch size for throughput.

3 ms·

Your main resource contention there would be network throughput, then broker CPU time in iowait if there's a lot of concurrent large disk reads happening on the

by EdwardDiego 4y ago

morelisp 4y ago
> Your main resource contention there would be network throughput, then broker CPU time The default consumer read sizes are so small you will hit broker CPU and worker thread limits long before network throughput. (Both consumer batch sizes and broker threads can be increased trivially but there's not much documentation around when to do this.)
- EdwardDiego 4y ago
  The default fetch.min.bytes is 1, that's true, but the default fetch.max.bytes is 50 MiB. And fetching small amounts of data repeatedly doesn't impose much overhead unless you're deliberately disconnecting and reconnecting between polls. And I assure you, you can definitely bottleneck network before CPU and/or network threads, Kafka was literally designed for very large numbers of consumers. And as for tuning, Kafka The Definitive Guide is pretty much as the name suggests. I've been recommending it very strongly, especially the chapters on monitoring and cluster replication, for years. You can download a free draft copy of the 2nd edition from Confluent, check it out :)
  - morelisp 4y ago
    > but the default fetch.max.bytes is 50 MiB. But max.partition.fetch.bytes is only 1MB.
    - EdwardDiego 4y ago
      Yep, it's aligned with the default max.message.bytes, largest batch size the broker will accept. 1 MiB is allegedly the ideal batch size for throughput.