10 ms·

If designing a new system is there any reason to choose Kafka over Pulsar at this point? Apart from Confluent wanting you to use Kafka so they can keep leechin

If designing a new system is there any reason to choose Kafka over Pulsar at this point? Apart from Confluent wanting you to use Kafka so they can keep leeching money off you by hijacking de facto ownership of an open source project, of course.

skyde 6y ago
I think Pulsar is a much better design and further investment in Kafka is a mistake at this point. Kafka have a lot of downside 1- size for single topic limited to the size of one machine 2- complex stateful client library that need to know which machine is currently the master for each partition. ....
- ovis 6y ago
  1. That's untrue. Partitions are limited to what a machine may handle, but topics may be scaled across many ordered partitions. 2. This is generally handled by the client library transparently. Have you ever needed to manage this state manually?
  - rad_gruchalski 6y ago
    Re 2, have you ever tried using Kafka with a non-JVM client?
    - EdwardDiego 6y ago
      Yep, I have, no issues with the consumer/producer knowing who the partition leader is. That said, curious to hear your experiences :)
      - rad_gruchalski 6y ago
        I don’t have issues per se as long as I stick to librdkafka but even that is constantly playing catch up. Outside of librdkafka and jvm client, it’s gloves off.
        EdwardDiego 6y ago
        Yeah, that's true, using librdkafka from C# I hit a few issues where librdkafka was somewhat behind Java in terms of features, I think the one I hit was multi-topic subscriptions. IIRC Confluent has started putting resources into it - I would hope so, given how .NET Core is going. That said, the state of Pulsar clients outside of the official Java ones was far worse, I was looking into .NET Core ones and the "official" one (Pulsar-DotPulsar) lacked some key features, whereas a third party one, pulsar-client-dotnet, had far more features, but was still somewhat behind the Java clients. Caveat is that I looked into all of this when Pulsar was at version 2.6, it's not at 2.7.1, so my comments may well be out of date.
    - mrkeen 6y ago
      kafkacat is excellent. https://github.com/edenhill/kafkacat https://github.com/edenhill/kafkacat
      - rad_gruchalski 6y ago
        Definitely, if all one needs is command line.
  - skyde 6y ago
    for #1 if you need message order to be maintained you cannot use more than 1 partition. for #2 99% of the production issue we had where caused by bugs in librdkafka not talking to the right brokers. Also it mean that the IP for the broker need to all be public IP and the writing throughput and latency is limited by the capacity of the machine that is the master for the partition you are trying to write. If the machine hosting you partition become overloaded you have to switch the master for that partition to another machine unlike pulsar this is not done automatically and also if you replication factor is 3 your choice are limited to 2 other machine if you don't want to copy the whole partition to a new machine which would take hours.
tyingq 6y ago
Hijacking seems like a strong word since Jay Kreps is the CEO.
- taywrobel 6y ago
  Fair enough, not hijacking. Just presenting the software as open source then requiring you to pay substantially for features that it’d be irresponsible to use the software in production without (e.g. geo replication via mirror maker). Add to that their insistence in claiming “exactly once delivery semantics” from Kafka despite that being provably impossible and I don’t see any reason to trust them as a company or pay for their software. I’ve been sticking to pulsar for all new projects and have yet to hit a single drawback. It scales better, has less fiddly knobs needing adjusting, has cluster management already built in, and supports traditional pub/suv as well as worker queue semantics. It even has Kafka compatible adapters so it’s relatively easy to migrate existing systems. Kafka played an important role in the history of distributed system design but it’s time to move to something better built and better managed IMO.
  - staticassertion 6y ago
    > Add to that their insistence in claiming “exactly once delivery semantics” from Kafka despite that being provably impossible Exactly once delivery is impossible. Exactly once processing is possible. TBH, semantically there's very little difference between those two from an end user perspective.
  - EdwardDiego 6y ago
    > requiring you to pay substantially for features that it’d be irresponsible to use the software in production without (e.g. geo replication via mirror maker). MM1 and MM2 are free. You might be getting confused with Confluent Replicator. > Kafka compatible adapters For very limited subsets of the Kafka APIs.
simplecto 6y ago
This is a very uncharitable take on Confluent. They contribute heavily to the project and offer enterprise support, professional services, and proprietary tech like Cluster linking. Source: I am an enterprise customer of Confluent.
- deleted 6y ago
  [deleted]
dan_quixote 6y ago
Quoting myself here [1], but Kafka is still very architecturally un-sound for containerized deployments. ------------------------------------------------------- The problem I see with kafka is that it was built before cloud architectures were commonly adopted (with distributed systems everywhere). Confluent has put a lot of effort dragging kafka's architecture to the present, but some major features are missing: - Auto-scaling: Confluent finally introduced "elastic scaling" a few months ago but it only allows you to scale up and must be triggered by the admin (no threshold-based auto-scaling). - Multi-tenancy: Planning for a multi-tenant kafka cluster is not for the faint of heart. Achieving isolation tends toward liberal usage of topics of which starts to become unmanageable in the low thousands. This isn't crazy when you've got a few hundred microservices and several tenants to keep isolated. - Decoupled brokers and storage: Any broker scaling or failure can lead to downtime while event storage is redistributed. Confluent's Cloud service reduces operational overhead but isn't always feasible due to cost, resource limits (like service accounts or schemas for instance), data controls, etc. ------------------------------------------------------- [1] https://news.ycombinator.com/item?id=25984560 https://news.ycombinator.com/item?id=25984560
plank_time 6y ago
I’ve been following Kafka for 5+ years now. Confluent has done their level best to stack the Kafka PMC with their own employees. Only recently have they added other members. They should not be an Apache project due to how much gaming of ownership they are doing with the project.
ovis 6y ago
One advantage of Kafka is the ecosystem effect. There are many systems (Flink, Kafka Streams, Pinot, Druid, Presto, etc) that connect to Kafka. I'm not sure about the extent of Pulsar support here, although I'd love to learn more!
colin_mccabe 6y ago
I personally think Kafka has the edge in many ways. It will soon be possible to run a single-process Kafka cluster, which will unlock a lot of applications that previously people used an older systems for, simply because it was easier than standing up a full ZK cluster + Kafka cluster. The broader Kafka ecosystem has features like exactly-once support, KSQL, Kafka Connect, Cluster Linking, and excellent client support that are very valuable. The Kafka community is huge and the velocity of development is very high. It's easy to forget now, but in the beginning, Kafka didn't even have replication. That's a good reminder that things that seem like permanent advantages of system X over Kafka (for various values of X) may very well prove to be temporary. For example, in this very thread, I see people talking about how various system X'es have the advantage over Kafka because they can run without ZK. Those discussions are almost out of date. Finally, I work at Confluent and I think the company has always been a positive force in the open source community. I respect the Pulsar people as well, but I think they have a difficult challenge to overcome.
- Hallucinaut 6y ago
  What challenges do you see Pulsar having (and potentially not overcoming)?
  - tamale 6y ago
    Pulsar is fighting a massive up-hill battle. Kafka is everywhere (and for good reason).
- taywrobel 6y ago
  > The broader Kafka ecosystem has features like exactly-once support No. No it doesn’t. It has at-least-once delivery with client-side deduplication. That’s not new, it’s what TCP does FFS. Why would you lie to people about supporting something long established at best and demonstrably impossible at worst? > Finally, I work at Confluent.... Oh, that’s why. Never mind then. Continue selling digital snake oil.
  - tiew9Vii 6y ago
tveita 6y ago
Pulsar also requires you to run ZooKeeper and BookKeeper, so TFA has at least one reason you might choose Kafka. (That said, unlike many I consider depending on ZooKeeper to be a positive sign. "We wrote our own consensus protocol" belongs in roughly the same bucket as "we wrote our own crypto." Using ZooKeeper doesn't automatically mean your distributed system will work but at least you'll have a fighting chance.)
- rad_gruchalski 6y ago
  > Pulsar also requires you to run ZooKeeper and BookKeeper, so TFA has at least one reason you might choose Kafka. BookKeeper is a feature though. Allows to scale the partition beyond the capacity of a storage unit. Effectively unlimited retention for a partition. The problem with Kafka is that the broker is tied to storage.
  - EdwardDiego 6y ago
    It can be a feature, yep, although Pulsar's tiered storage (offload old data into S3 and then retrieve it if required) is far more of a feature IMO. And BK isn't a feature without cost, it's documentation is... somewhat sparse, and there is significant complexity to maintaining it.
    - rad_gruchalski 6y ago
      > offload old data into S3 and then retrieve it if required I remember asking for it 5 years ago: https://radek-gruchalski.medium.com/the-case-for-kafka-cold-storage-32929d0a57b2 https://radek-gruchalski.medium.com/the-case-for-kafka-cold-.... Confluent turned it into a paid feature.
      - EdwardDiego 6y ago
        Yep, it's available from Confluent if you pay for it, but like how Mirror Maker 2 is awfully similar to Confluent Replicator, I believe that Kafka will (eventually) get tiered storage under the Apache licence (I know there's a KIP for it[1]). It's a hard issue to solve, and not sure how much effort in the community is being directed towards it. But bear in mind that it's not just Confluent who have a stake in Kafka - there's a bunch of big corps selling managed/supported Kafka and all of them would probably quite like tiered storage in core Kafka as a feature to help them sell their support/management, so I have some faith in their enlightened self-interest. The fact that MM2 happened, and Confluent didn't try to stop it, despite it being awfully similar to Replicator, makes me think that Confluent are acting in good faith. Incidentally, I quite like how Pulsar solved tiered storage, and it's a definite tick in the Pulsar box - it's transparent from a consumer's POV, although there somewhat of a delay in rehydrating the offloaded block, I don't think anyone's expecting near-realtime performance when loading historical data. [1]: https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A...
EdwardDiego 6y ago
As someone who evaluated Pulsar to replace Kafka, my thoughts... More moving parts. Brokers, and Bookies, and ZK, plus proxies etc. Plus an additional ZK for inter-cluster replication. Immaturity - it's still early days for Pulsar, and there's still a lot of bugs being found - and then rapidly fixed, full credit to them, but yeah, not yet as stable. Documentation is often obsoleted, and I found myself having read the code to figure out what was actually going on. More complex workflows - there's only really one model for a developer consuming or producing against Kafka. With Pulsar, there's multiple different subscription modes, and choosing the wrong one could produce problems. Also, the need to explicit ack the messages is something you'd have to always watch for to avoid duplicated reads. Also, if using batch receive, when I was looking at Pulsar, you either had to acknowledge the entire batch, or none of it, so a failure during batch processing would lead to the batch being reprocessed, but I think acking within a batch is in development. No Pulsar IO S3 sink yet. That said, there's a lot of cool things it's doing, like the built-in schema registry and far easier multitenancy, and offloading older data into S3 etc. transparently to the consumers, so I'm definitely I'm keeping an eye on it. Lastly, you're taking aim at Confluent, you realise Pulsar is largely controlled by people employed by StreamNative, yeah?
- taywrobel 6y ago
  StreamNative doesn’t lock critical functionality behind enterprise agreements while still advertising the software as open source, and doesn’t openly lie about what the system capabilities are. Not saying that they won’t turn evil at some point, but so far they’re leagues ahead of Confluent in terms of earning developer trust. At a minimum this developer, but also others that I’ve worked with. Maybe I’m the minority opinion here and that’s fine, but confluent has been far too shady for me to ever consider contracting with them.
  - EdwardDiego 6y ago
    What critical functionality are you thinking of? We used FOSS Kafka for yonks without hitting any limitations - At one point we were looking at Confluent Replicator, but decided it was just easier to go with Mirror Maker 1 (and you know, no massive licensing fees) - and Mirror Maker 2 largely emulates Replicator in terms of functionality. I'm aware of a few other things like the MQTT KC Connector, but that was never part of Kafka in the first instance, it's something that Confluent built for paying customers. And I could argue that the StreamNative "critical functionality" that they lock away behind enterprise agreements is "quick bug fixes for the many bugs we're still finding", if I was feeling mean spirited. But anyway, it seems your preference for Pulsar is due to Confluent, but they're not the only ones offering managed Kafkas - AWS, IBM, RedHat, etc. etc.
atombender 6y ago
Apache Pulsar is hardly an ideal comparison in this context, considering that Pulsar requires ZooKeeper and Apache BookKeeper (which also requires ZooKeeper). One of the benefits of the Kafka rearchitecture effort is to allow Kafka to "scale down" to run without external dependencies. Using Pulsar would add more dependencies.
- tuhaihe 6y ago
  As I know, Pulsar community is doing the work of removing ZooKeeper, this is a helpful link: https://github.com/apache/pulsar/projects/10 https://github.com/apache/pulsar/projects/10
  - atombender 6y ago
    They are, but I think they have some ways to go still. BookKeeper is still on ZooKeeper, though they've abstracted the API and have added support for using Etcd instead (not sure if this is production-ready).
manigandham 6y ago
There's actually no reason to choose Pulsar anymore. Pulsar has even more layers with Zookeeper + Bookkeeper that requires something like Kubernetes to run well. It was great 5 years ago for heavy users who need better scalability and features than Kafka, however the development has become a mess. With the removal of zookeeper and tiered storage (separated from compute), Kafka has caught up on scalability while being simpler to deploy. It also has a far bigger ecosystem with more polished features like ksqldb.

skyde 6y ago

I think Pulsar is a much better design and further investment in Kafka is a mistake at this point. Kafka have a lot of downside 1- size for single topic limited to the size of one machine 2- complex stateful client library that need to know which machine is currently the master for each partition. ....

ovis 6y ago
1. That's untrue. Partitions are limited to what a machine may handle, but topics may be scaled across many ordered partitions. 2. This is generally handled by the client library transparently. Have you ever needed to manage this state manually?
- rad_gruchalski 6y ago
  Re 2, have you ever tried using Kafka with a non-JVM client?
  - EdwardDiego 6y ago
    Yep, I have, no issues with the consumer/producer knowing who the partition leader is. That said, curious to hear your experiences :)
    - rad_gruchalski 6y ago
      I don’t have issues per se as long as I stick to librdkafka but even that is constantly playing catch up. Outside of librdkafka and jvm client, it’s gloves off.
      - EdwardDiego 6y ago
        Yeah, that's true, using librdkafka from C# I hit a few issues where librdkafka was somewhat behind Java in terms of features, I think the one I hit was multi-topic subscriptions. IIRC Confluent has started putting resources into it - I would hope so, given how .NET Core is going. That said, the state of Pulsar clients outside of the official Java ones was far worse, I was looking into .NET Core ones and the "official" one (Pulsar-DotPulsar) lacked some key features, whereas a third party one, pulsar-client-dotnet, had far more features, but was still somewhat behind the Java clients. Caveat is that I looked into all of this when Pulsar was at version 2.6, it's not at 2.7.1, so my comments may well be out of date.
  - mrkeen 6y ago
    kafkacat is excellent. https://github.com/edenhill/kafkacat https://github.com/edenhill/kafkacat
    - rad_gruchalski 6y ago
      Definitely, if all one needs is command line.
- skyde 6y ago
  for #1 if you need message order to be maintained you cannot use more than 1 partition. for #2 99% of the production issue we had where caused by bugs in librdkafka not talking to the right brokers. Also it mean that the IP for the broker need to all be public IP and the writing throughput and latency is limited by the capacity of the machine that is the master for the partition you are trying to write. If the machine hosting you partition become overloaded you have to switch the master for that partition to another machine unlike pulsar this is not done automatically and also if you replication factor is 3 your choice are limited to 2 other machine if you don't want to copy the whole partition to a new machine which would take hours.

tyingq 6y ago

Hijacking seems like a strong word since Jay Kreps is the CEO.

taywrobel 6y ago
Fair enough, not hijacking. Just presenting the software as open source then requiring you to pay substantially for features that it’d be irresponsible to use the software in production without (e.g. geo replication via mirror maker). Add to that their insistence in claiming “exactly once delivery semantics” from Kafka despite that being provably impossible and I don’t see any reason to trust them as a company or pay for their software. I’ve been sticking to pulsar for all new projects and have yet to hit a single drawback. It scales better, has less fiddly knobs needing adjusting, has cluster management already built in, and supports traditional pub/suv as well as worker queue semantics. It even has Kafka compatible adapters so it’s relatively easy to migrate existing systems. Kafka played an important role in the history of distributed system design but it’s time to move to something better built and better managed IMO.
- staticassertion 6y ago
  > Add to that their insistence in claiming “exactly once delivery semantics” from Kafka despite that being provably impossible Exactly once delivery is impossible. Exactly once processing is possible. TBH, semantically there's very little difference between those two from an end user perspective.
- EdwardDiego 6y ago
  > requiring you to pay substantially for features that it’d be irresponsible to use the software in production without (e.g. geo replication via mirror maker). MM1 and MM2 are free. You might be getting confused with Confluent Replicator. > Kafka compatible adapters For very limited subsets of the Kafka APIs.

simplecto 6y ago

This is a very uncharitable take on Confluent. They contribute heavily to the project and offer enterprise support, professional services, and proprietary tech like Cluster linking. Source: I am an enterprise customer of Confluent.

deleted 6y ago
[deleted]

dan_quixote 6y ago

Quoting myself here [1], but Kafka is still very architecturally un-sound for containerized deployments. ------------------------------------------------------- The problem I see with kafka is that it was built before cloud architectures were commonly adopted (with distributed systems everywhere). Confluent has put a lot of effort dragging kafka's architecture to the present, but some major features are missing: - Auto-scaling: Confluent finally introduced "elastic scaling" a few months ago but it only allows you to scale up and must be triggered by the admin (no threshold-based auto-scaling). - Multi-tenancy: Planning for a multi-tenant kafka cluster is not for the faint of heart. Achieving isolation tends toward liberal usage of topics of which starts to become unmanageable in the low thousands. This isn't crazy when you've got a few hundred microservices and several tenants to keep isolated. - Decoupled brokers and storage: Any broker scaling or failure can lead to downtime while event storage is redistributed. Confluent's Cloud service reduces operational overhead but isn't always feasible due to cost, resource limits (like service accounts or schemas for instance), data controls, etc. ------------------------------------------------------- [1] https://news.ycombinator.com/item?id=25984560 https://news.ycombinator.com/item?id=25984560

plank_time 6y ago

I’ve been following Kafka for 5+ years now. Confluent has done their level best to stack the Kafka PMC with their own employees. Only recently have they added other members. They should not be an Apache project due to how much gaming of ownership they are doing with the project.

ovis 6y ago

One advantage of Kafka is the ecosystem effect. There are many systems (Flink, Kafka Streams, Pinot, Druid, Presto, etc) that connect to Kafka. I'm not sure about the extent of Pulsar support here, although I'd love to learn more!

colin_mccabe 6y ago

I personally think Kafka has the edge in many ways. It will soon be possible to run a single-process Kafka cluster, which will unlock a lot of applications that previously people used an older systems for, simply because it was easier than standing up a full ZK cluster + Kafka cluster. The broader Kafka ecosystem has features like exactly-once support, KSQL, Kafka Connect, Cluster Linking, and excellent client support that are very valuable. The Kafka community is huge and the velocity of development is very high. It's easy to forget now, but in the beginning, Kafka didn't even have replication. That's a good reminder that things that seem like permanent advantages of system X over Kafka (for various values of X) may very well prove to be temporary. For example, in this very thread, I see people talking about how various system X'es have the advantage over Kafka because they can run without ZK. Those discussions are almost out of date. Finally, I work at Confluent and I think the company has always been a positive force in the open source community. I respect the Pulsar people as well, but I think they have a difficult challenge to overcome.

Hallucinaut 6y ago
What challenges do you see Pulsar having (and potentially not overcoming)?
- tamale 6y ago
  Pulsar is fighting a massive up-hill battle. Kafka is everywhere (and for good reason).
taywrobel 6y ago
> The broader Kafka ecosystem has features like exactly-once support No. No it doesn’t. It has at-least-once delivery with client-side deduplication. That’s not new, it’s what TCP does FFS. Why would you lie to people about supporting something long established at best and demonstrably impossible at worst? > Finally, I work at Confluent.... Oh, that’s why. Never mind then. Continue selling digital snake oil.
- tiew9Vii 6y ago

tveita 6y ago

Pulsar also requires you to run ZooKeeper and BookKeeper, so TFA has at least one reason you might choose Kafka. (That said, unlike many I consider depending on ZooKeeper to be a positive sign. "We wrote our own consensus protocol" belongs in roughly the same bucket as "we wrote our own crypto." Using ZooKeeper doesn't automatically mean your distributed system will work but at least you'll have a fighting chance.)

rad_gruchalski 6y ago
> Pulsar also requires you to run ZooKeeper and BookKeeper, so TFA has at least one reason you might choose Kafka. BookKeeper is a feature though. Allows to scale the partition beyond the capacity of a storage unit. Effectively unlimited retention for a partition. The problem with Kafka is that the broker is tied to storage.
- EdwardDiego 6y ago
  It can be a feature, yep, although Pulsar's tiered storage (offload old data into S3 and then retrieve it if required) is far more of a feature IMO. And BK isn't a feature without cost, it's documentation is... somewhat sparse, and there is significant complexity to maintaining it.
  - rad_gruchalski 6y ago
    > offload old data into S3 and then retrieve it if required I remember asking for it 5 years ago: https://radek-gruchalski.medium.com/the-case-for-kafka-cold-storage-32929d0a57b2 https://radek-gruchalski.medium.com/the-case-for-kafka-cold-.... Confluent turned it into a paid feature.
    - EdwardDiego 6y ago
      Yep, it's available from Confluent if you pay for it, but like how Mirror Maker 2 is awfully similar to Confluent Replicator, I believe that Kafka will (eventually) get tiered storage under the Apache licence (I know there's a KIP for it[1]). It's a hard issue to solve, and not sure how much effort in the community is being directed towards it. But bear in mind that it's not just Confluent who have a stake in Kafka - there's a bunch of big corps selling managed/supported Kafka and all of them would probably quite like tiered storage in core Kafka as a feature to help them sell their support/management, so I have some faith in their enlightened self-interest. The fact that MM2 happened, and Confluent didn't try to stop it, despite it being awfully similar to Replicator, makes me think that Confluent are acting in good faith. Incidentally, I quite like how Pulsar solved tiered storage, and it's a definite tick in the Pulsar box - it's transparent from a consumer's POV, although there somewhat of a delay in rehydrating the offloaded block, I don't think anyone's expecting near-realtime performance when loading historical data. [1]: https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A...

EdwardDiego 6y ago

As someone who evaluated Pulsar to replace Kafka, my thoughts... More moving parts. Brokers, and Bookies, and ZK, plus proxies etc. Plus an additional ZK for inter-cluster replication. Immaturity - it's still early days for Pulsar, and there's still a lot of bugs being found - and then rapidly fixed, full credit to them, but yeah, not yet as stable. Documentation is often obsoleted, and I found myself having read the code to figure out what was actually going on. More complex workflows - there's only really one model for a developer consuming or producing against Kafka. With Pulsar, there's multiple different subscription modes, and choosing the wrong one could produce problems. Also, the need to explicit ack the messages is something you'd have to always watch for to avoid duplicated reads. Also, if using batch receive, when I was looking at Pulsar, you either had to acknowledge the entire batch, or none of it, so a failure during batch processing would lead to the batch being reprocessed, but I think acking within a batch is in development. No Pulsar IO S3 sink yet. That said, there's a lot of cool things it's doing, like the built-in schema registry and far easier multitenancy, and offloading older data into S3 etc. transparently to the consumers, so I'm definitely I'm keeping an eye on it. Lastly, you're taking aim at Confluent, you realise Pulsar is largely controlled by people employed by StreamNative, yeah?

taywrobel 6y ago
StreamNative doesn’t lock critical functionality behind enterprise agreements while still advertising the software as open source, and doesn’t openly lie about what the system capabilities are. Not saying that they won’t turn evil at some point, but so far they’re leagues ahead of Confluent in terms of earning developer trust. At a minimum this developer, but also others that I’ve worked with. Maybe I’m the minority opinion here and that’s fine, but confluent has been far too shady for me to ever consider contracting with them.
- EdwardDiego 6y ago
  What critical functionality are you thinking of? We used FOSS Kafka for yonks without hitting any limitations - At one point we were looking at Confluent Replicator, but decided it was just easier to go with Mirror Maker 1 (and you know, no massive licensing fees) - and Mirror Maker 2 largely emulates Replicator in terms of functionality. I'm aware of a few other things like the MQTT KC Connector, but that was never part of Kafka in the first instance, it's something that Confluent built for paying customers. And I could argue that the StreamNative "critical functionality" that they lock away behind enterprise agreements is "quick bug fixes for the many bugs we're still finding", if I was feeling mean spirited. But anyway, it seems your preference for Pulsar is due to Confluent, but they're not the only ones offering managed Kafkas - AWS, IBM, RedHat, etc. etc.

atombender 6y ago

Apache Pulsar is hardly an ideal comparison in this context, considering that Pulsar requires ZooKeeper and Apache BookKeeper (which also requires ZooKeeper). One of the benefits of the Kafka rearchitecture effort is to allow Kafka to "scale down" to run without external dependencies. Using Pulsar would add more dependencies.

tuhaihe 6y ago
As I know, Pulsar community is doing the work of removing ZooKeeper, this is a helpful link: https://github.com/apache/pulsar/projects/10 https://github.com/apache/pulsar/projects/10
- atombender 6y ago
  They are, but I think they have some ways to go still. BookKeeper is still on ZooKeeper, though they've abstracted the API and have added support for using Etcd instead (not sure if this is production-ready).

manigandham 6y ago

There's actually no reason to choose Pulsar anymore. Pulsar has even more layers with Zookeeper + Bookkeeper that requires something like Kubernetes to run well. It was great 5 years ago for heavy users who need better scalability and features than Kafka, however the development has become a mess. With the removal of zookeeper and tiered storage (separated from compute), Kafka has caught up on scalability while being simpler to deploy. It also has a far bigger ecosystem with more polished features like ksqldb.