5 ms·
On efficiently partitioning a topic in Apache Kafka
- belter 5y agoPDF: https://arxiv.org/pdf/2205.09415.pdf https://arxiv.org/pdf/2205.09415.pdf "...Even though Apache Kafka provides some out of the box optimizations, it does not strictly define how each topic shall be efficiently distributed into partitions. The well-formulated fine-tuning that is needed in order to improve an Apache Kafka cluster performance is still an open research problem. In this paper, we first model the Apache Kafka topic partitioning process for a given topic. Then, given the set of brokers, constraints and application requirements on throughput, OS load, replication latency and unavailability, we formulate the optimization problem of finding how many partitions are needed and show that it is computationally intractable, being an integer program. Furthermore, we propose two simple, yet efficient heuristics to solve the problem: the first tries to minimize and the second to maximize the number of brokers used in the cluster. Finally, we evaluate its performance via large-scale simulations, considering as benchmarks some Apache Kafka cluster configuration recommendations provided by Microsoft and Confluent. We demonstrate that, unlike the recommendations, the proposed heuristics respect the hard constraints on replication latency and perform better w.r.t. unavailability time and OS load, using the system resources in a more prudent way..."
- claytonjy 5y agoTo what extent do these concerns change or disappear if using alternatives, like Pulsar or Redpanda? It's been awhile since I dug into this but a couple jobs ago I was very concerned with what happens if replaying a topic from 0 for a new consumer: existing, up-to-date consumers are negatively impacted! As I recall this was due to fundamental architecture around partitions, and a notable advantage of Pulsar was not having such issues. Is that correct? Is that still the case?
- bobgt 5y agoAssuming i) you've set topic retention so that old messages still exist and your new consumer can "replay from 0"; and ii) your new consumer is using their own consumer group, then existing consumers won't be impacted.
- claytonjy 5y agoIs there not resource contention when multiple consumer (groups) are reading from the same Kafka partition? You can of course over-provision your partitions to better allow for this, but rebalancing is not cheap and also tends to affect consumers. I may be describing the problem incorrectly, but I know vendors we talked to were aware of this issue and had workarounds; IIRC Aiven had tooling to easily spin up a temporary new "mirror" cluster for the new consumer to catch up.
- morelisp 5y agoStarting a new consumer group won't cause rebalances (except in the new group). It sounds like you're asking if you can double the number of readers in a system with no performance impact. If you're at capacity, the answer is obviously no. Yes, every consumer takes some i/o and CPU on the brokers serving the data. I have never used Pulsar but I'm sure that's also the case there.
- bacheaul 5y agoThere's also the file system cache to consider, which Kafka famously leans on heavily for IO performance. If the majority of your consumers are reading the latest messages which were just written, these will likely come from the cache which is in memory. A consumer reading from the earliest messages on a large topic could conceivably cause changes to what's available from file system cache for other consumers reading from the latest messages, so they're not necessarily totally isolated. I've not taken measurements of this though to say it's an actual issue, just saying that I wouldn't dismiss it.
- EdwardDiego 5y agoKafka special cases the "tail" of a partition - the open log segment (i.e., the very end of the partition that's still being written too) is never evicted, and log segments closer to the tail are evicted last, IIRC. It definitely prioritises tail consumption over read from 0.
- 5y ago
- EdwardDiego 5y agoNo, Pulsar has partitions too. And I'm surprised you saw a massive effect unless the brokers were under huge load.
- blagie 5y agoIs there a good, simple alternative to Kafka? My experience so far with Kafka is that error messages are nearly useless. Even an invalid command line parameter raises an exception, rather than printing a usage. In many cases, if I do something wrong, it just stalls and times out, with no real way to debug. The documentation caps out at a very nominal level. I tried to implement a simple KVS with compact logs. Documentation ran out and I was wading in deep waters. I'm trying to build a platform that's developer-friendly, and I'm deep enough in to know Kafka shouldn't be part of it. That's not to mention the growing uncertainty around the JDK/JVM. I want open-source, archival, and to be able to reply event streams from the beginning. For my application, I'll take lower performance in return for better usability.
- EdwardDiego 5y agoNATS maybe? https://github.com/nats-io https://github.com/nats-io Yeah, Kafka's complex, and it's really a tool that you need to think long and hard about whether or not your problems are complicated enough that introducing Kafka to solve them will ultimately reduce complexity. And it's definitely something that developers have to learn the semantics of to use well. > The documentation caps out at a very nominal level. I tried to implement a simple KVS with compact logs. Documentation ran out and I was wading in deep waters I occasionally contribute to Apache Kafka, any pointers on what documentation was lacking, and where you could have used more? I will see if I can contribute something around that :) > That's not to mention the growing uncertainty around the JDK/JVM. What do you mean by this? > I want open-source, archival, and to be able to reply event streams from the beginning. Yeah, maybe check out Liftbridge, it's built on NATS. https://liftbridge.io/ https://liftbridge.io/ Apache Pulsar might be worth a look, but it's actually more complex under the hood than Kafka, but has a lot of features built-in that either aren't in FOSS Kafka yet, like tiered storage, or won't be until Confluent doesn't dominate the PMC (like an integrated schema registry), or just can't be done very nicely, if at all, like decent multi-tenancy. That said, it's a fast moving target, the code quality last I looked was patchy in places, ditto the documentation for both it and Bookkeeper, and the admin overhead is higher (managing bookies and brokers and Zookeepers vs. just brokers and ZK with Kafka, or when KRaft is production ready, just brokers).