5 ms·

Starting a new consumer group won't cause rebalances (except in the new group). It sounds like you're asking if you can double the number of readers in a syste

Starting a new consumer group won't cause rebalances (except in the new group). It sounds like you're asking if you can double the number of readers in a system with no performance impact. If you're at capacity, the answer is obviously no. Yes, every consumer takes some i/o and CPU on the brokers serving the data. I have never used Pulsar but I'm sure that's also the case there.

bacheaul 4y ago
There's also the file system cache to consider, which Kafka famously leans on heavily for IO performance. If the majority of your consumers are reading the latest messages which were just written, these will likely come from the cache which is in memory. A consumer reading from the earliest messages on a large topic could conceivably cause changes to what's available from file system cache for other consumers reading from the latest messages, so they're not necessarily totally isolated. I've not taken measurements of this though to say it's an actual issue, just saying that I wouldn't dismiss it.
- EdwardDiego 4y ago
  Kafka special cases the "tail" of a partition - the open log segment (i.e., the very end of the partition that's still being written too) is never evicted, and log segments closer to the tail are evicted last, IIRC. It definitely prioritises tail consumption over read from 0.
- morelisp 4y ago
  FS cache is, at least for me, included in "i/o resources". But this utilization will occur for any consumer reading any partition from any topic from anywhere other than the tail segment which isn't currently being read (or even the tail segment of a partition not being produced); it's not specific to partition 0. And I don't believe you'd gain anything by turning on a mirroring cluster rather than increasing the number of brokers in the same cluster; in both cases you're solving it by spreading the i/o load out more.
EdwardDiego 4y ago
I think they were discussing rebalancing replicas when you increase partitions. Cruise Control is great here.

bacheaul 4y ago

There's also the file system cache to consider, which Kafka famously leans on heavily for IO performance. If the majority of your consumers are reading the latest messages which were just written, these will likely come from the cache which is in memory. A consumer reading from the earliest messages on a large topic could conceivably cause changes to what's available from file system cache for other consumers reading from the latest messages, so they're not necessarily totally isolated. I've not taken measurements of this though to say it's an actual issue, just saying that I wouldn't dismiss it.

EdwardDiego 4y ago
Kafka special cases the "tail" of a partition - the open log segment (i.e., the very end of the partition that's still being written too) is never evicted, and log segments closer to the tail are evicted last, IIRC. It definitely prioritises tail consumption over read from 0.
morelisp 4y ago
FS cache is, at least for me, included in "i/o resources". But this utilization will occur for any consumer reading any partition from any topic from anywhere other than the tail segment which isn't currently being read (or even the tail segment of a partition not being produced); it's not specific to partition 0. And I don't believe you'd gain anything by turning on a mirroring cluster rather than increasing the number of brokers in the same cluster; in both cases you're solving it by spreading the i/o load out more.

EdwardDiego 4y ago

I think they were discussing rebalancing replicas when you increase partitions. Cruise Control is great here.