4 ms·
Congrats to the kafka/confluent team. Side note: https://pulsar.apache.org https://pulsar.apache.org also seems to be gaining traction and has a much better st
by manigandham 9y ago
Congrats to the kafka/confluent team.
Side note: https://pulsar.apache.org https://pulsar.apache.org also seems to be gaining traction and has a much better story around performance, pub/sub, multi-tenancy, and cross-dc replication. Will be interesting to see the evolution of both going forward.
- delaSoul 9y agoHas somebody compared pulsar with pravega(http://pravega.io http://pravega.io)?
- slap_shot 9y agoInteresting. Can someone give a quick/high level list what makes it better than Kafka and in what use cases?
- manigandham 9y agoI recommend reading the Pulsar architecture page: https://pulsar.apache.org/docs/latest/getting-started/ConceptsAndArchitecture/ https://pulsar.apache.org/docs/latest/getting-started/Concep... Biggest difference is that Pulsar splits compute from storage. Pulsar brokers are stateless services that manage topics and handle clients (publishers and consumers) while reading/writing data from a lower tier of bookies, which is the Apache BookKeeper project for low-latency real-time storage workloads. Better performance and easier scaling compared to Kafka's partition rebalancing. It also natively supports data center replication and multiple clusters working together globally. Pulsar addressing is property/cluster/namespace/topic so multi-tenant isolation is built-in. Anywhere you use Kafka can use Pulsar and also consolidate pub/sub only systems that need lower latency than Kafka.
- lmsp 9y agoI think it is beyond partition rebalancing. There is a fact that people didn't realize of making message broker `stateless`. It is actually much better on reacting to failures or shifting traffic, which is critical when running a messaging bus for online services. because it doesn't have to wait for copying the data of a whole partition when error occurs.
- manigandham 9y agoAgreed, the architecture page and other blog posts do a more thorough job of explaining the details. Having a stateless broker layer on top of a focused data layer makes all the operations much easier. I expect BookKeeper to further integrate with the various cloud storage APIs so it can also start to become stateless cache.
- majidazimi 9y agoThe biggest differences: - rebalancing: Bookkeeper (which is the storage backend of Pulsar) manages rebalancing way better than kafka. - Read scaling: Bookkeeper uses consensus to store log records, which means you can read from whatever shard you want. Whereas in kafka, reader has to connect to partition master.
- petermelias 9y agoTo clarify the use of the term "rebalancing" here for people who are familiar with Kafka's terminology: this is (I believe) referring to partition _storage_ rebalancing (redistribution of topics replicated across brokers). It is _not_ referring to Kafka's notion of rebalancing across a consumer group subscribed to a set of topic partitions. The specific concern here is the possibility that Kafka's ISR strategy can potentially result in a corrupt leader partition and truncate messages to recover from a broker machine failure. The unclean leader election configuration setting for Kafka brokers is relevant here. Also "better" is subjective depending on your configuration, requirements, and storage backends.
- lmsp 9y agoanother point added to 'rebalancing' -- when kafka rebalances the partitions, it has to copy all the data for the partitions that are moved around. it might not be a big problem when retention is small. however it is pretty worse when retention period is longer, rebalancing is going to exhaust all the bandwidth (both network and I/O) in the cluster. people don't realize the fact until they want to grow the cluster (adding more brokers) to support increased traffic.
- lmsp 9y agoanother side note, pulsar supports kafka api since 1.20.0 - https://github.com/apache/incubator-pulsar/releases/tag/v1.20.0-incubating https://github.com/apache/incubator-pulsar/releases/tag/v1.2...
- colordrops 9y agoWould you say that pulsar is strictly better than kafka, or that there are cases where kafka is preferred?
- manigandham 9y agoYes, right now Pulsar does everything Kafka does but easier, faster, and with more features. It's especially great in combining message queuing with low-latency pub/sub. It's also been around for years but recently open-sourced so community is smaller than Kafka, however it's growing quickly along with the usual ecosystem of drivers, extensions, and services. Unless you're already running Kafka, I would look at Pulsar first for new projects. A great overview of Pulsar history with plenty of links is here: https://streaml.io/blog/messaging-storage-or-both/ https://streaml.io/blog/messaging-storage-or-both/
- tkanos 9y agoUnfortunately pulsar does not have a system such kafka stream. In pulsar we need to use spark to do the same (https://pulsar.incubator.apache.org/docs/latest/adaptors/PulsarSpark/ https://pulsar.incubator.apache.org/docs/latest/adaptors/Pul...). And pulsar does not provide exactly once neither, that is something very important.