3 ms·

Why should I consider Pulsar instead of Kafka?

by arnon 7y ago

manigandham 7y ago
Separates storage from brokers for better scaling and performance. Millions of topics without a problem and built-in tenant/namespace/topic hierarchy. Kubernetes-native. Per-message acknowledgement instead of just an offset. Ephemeral pub/sub or persistent data. Built-in functions/lambda platform. Long-term/tiered storage into S3/object storage. Geo-replication across clusters.
- philote 7y ago
  I'd looked into Kafka before and really liked the partitioned topics and specifying partition keys. This makes sure only one consumer handles all messages for a specific key. Is there a way to do this in Pulsar? I see they have partitioned topics but it seems to only for performance reasons, but I don't see how you'd set up a partition key or make sure a specific key would always go to the same consumer. EDIT: Also wanted to note that, for Python, there is some support for async clients in Kafka (aiokafka, asynckafka) but I don't see any async libs for Pulsar yet. Maybe I'm missing something.
  - manigandham 7y ago
    Yes, you can tag messages with a 'key' for compaction (maintaining the latest message for a given key) and there's now a beta feature for key-ordered consumers on the same topic: https://pulsar.apache.org/docs/en/concepts-messaging/#key_shared https://pulsar.apache.org/docs/en/concepts-messaging/#key_sh... You can also just use separate topics since they're very lightweight and can be nested under the tenant/namespace hierarchy.
- staticassertion 7y ago
  Wow, yeah that does sound way better than Kafka, and very useful for what I'm doing. Thanks.
tiew9Vii 7y ago
It has all the bits missing from Kafka you want and they are trying to bolt on to Kafka. Tiered storage Geo replication Multi-tenancy Massive partition counts Benchmarks have lower latency Really the question should be why Kafka over Pulsar and the only real answer is it’s been around longer and Confluent are really aggressive with their marketing, as an example you might often hear “why Kafka is more ACID than your database” from Confluent, or “Kafka does exactly once” vs effectively once with idempotent operations.

Why should I consider Pulsar instead of Kafka?

by arnon 7y ago

Why should I consider Pulsar instead of Kafka?

Separates storage from brokers for better scaling and performance. Millions of topics without a problem and built-in tenant/namespace/topic hierarchy. Kubernetes-native. Per-message acknowledgement instead of just an offset. Ephemeral pub/sub or persistent data. Built-in functions/lambda platform. Long-term/tiered storage into S3/object storage. Geo-replication across clusters.

philote 7y ago
I'd looked into Kafka before and really liked the partitioned topics and specifying partition keys. This makes sure only one consumer handles all messages for a specific key. Is there a way to do this in Pulsar? I see they have partitioned topics but it seems to only for performance reasons, but I don't see how you'd set up a partition key or make sure a specific key would always go to the same consumer. EDIT: Also wanted to note that, for Python, there is some support for async clients in Kafka (aiokafka, asynckafka) but I don't see any async libs for Pulsar yet. Maybe I'm missing something.
- manigandham 7y ago
  Yes, you can tag messages with a 'key' for compaction (maintaining the latest message for a given key) and there's now a beta feature for key-ordered consumers on the same topic: https://pulsar.apache.org/docs/en/concepts-messaging/#key_shared https://pulsar.apache.org/docs/en/concepts-messaging/#key_sh... You can also just use separate topics since they're very lightweight and can be nested under the tenant/namespace hierarchy.
staticassertion 7y ago
Wow, yeah that does sound way better than Kafka, and very useful for what I'm doing. Thanks.

tiew9Vii 7y ago

It has all the bits missing from Kafka you want and they are trying to bolt on to Kafka. Tiered storage Geo replication Multi-tenancy Massive partition counts Benchmarks have lower latency Really the question should be why Kafka over Pulsar and the only real answer is it’s been around longer and Confluent are really aggressive with their marketing, as an example you might often hear “why Kafka is more ACID than your database” from Confluent, or “Kafka does exactly once” vs effectively once with idempotent operations.