4 ms·

Use pulsar - so much better than kafka

• lytedev 6y ago
  Why is that?
  • dominotw 6y ago
    millions of topics, no zookeeper ect. Kafka is addressing these shortcomings on the roadmap.
    • nitwit005 6y ago
      The Pulsar documentation says it requires Zookeeper: https://pulsar.apache.org/docs/en/administration-zk-bk/ https://pulsar.apache.org/docs/en/administration-zk-bk/
      • dominotw 6y ago
        oh sorry i meant storing topic info in zookeper that limits kafka to a certain number of topics.
        • EdwardDiego 6y ago
          Nope, it also stores topic metadata in ZK - it's not exactly going to store that in the (near) stateless brokers, or in Bookkeeper - and BK also relies on ZK, but it's common to reuse the ZK quorum between the brokers and the bookies. It also needs an additional ZK for cluster replication.
    • oweiler 6y ago
      For a lot a projects this is hardly a problem. On the other hand Kafka is more mature and has a huge ecosystem (Kafka Connect, Kafka Streams, KSQL, ...).
      • math 6y ago
        Kafka also has less moving parts even today before zookeeper removal is complete (2 vs pulsar 3).
        • biggestlou 6y ago
          But one of those moving parts of Pulsar, BookKeeper, means that you're no longer storing data on message brokers. Worth the extra puzzle piece for a lot of use cases.
      • biggestlou 6y ago
        Pulsar is less mature but does provide functional equivalents to all of the above. Pulsar IO (Kafka Connect), Pulsar SQL (KSQL), Pulsar Functions (Kafka Streams).
        • EdwardDiego 6y ago
          Nah, Pulsar functions is nowhere near Kafka Streams - it's more like AWS lambda. Example off the top of my head, is that you can't, in Pulsar Functions write the equivalent of "aggregate this stream across a 10 minute window and emit the results on window close". But that's fine, it doesn't need to be like Kafka Streams, you can use Flink or Spark or Storm etc. to fill the same niche. In fact one of the founders of StreamNative (Pulsar's equivalent of Confluent) is a core committer on Flink.
      • skyde 6y ago
        Kafka is not more mature just more hyped. I just wish Aphyr jepsen test would also cover more scenario like - what happen to your data if x+1 server permanently fail in the cluster with a replication factor of X. - what happen if a single partition data size or request rate become 90% of the cluster capacity - what happen in multi-tenancy scenario to other user throughput and latency when one user try to use all the capacity of the cluster - ...
        • EdwardDiego 6y ago
  • wpietri 6y ago
    If anybody has seen a good detailed comparison, I'd love to read one. The first dozen hits were pretty weak.
• mrunkel 6y ago
  I am all for self-reliance, but if you really want to influence someone else, you might want to include a link to the project, especially when the only word you share has a much more prevalent meaning.
  • jooz 6y ago
    There was a Pulsar post on HN a few months ago, with interesting comments, some of the related to kafka. I keep Pulsar on the radar since that post: https://news.ycombinator.com/item?id=21936252 https://news.ycombinator.com/item?id=21936252