3 ms·
I think saying it's analogous to Kafka 0.8 is reasonably apt. It's very stable and performant in the verticals it has been used in at Yahoo but not as widely in
by jpgvm 6y ago
I think saying it's analogous to Kafka 0.8 is reasonably apt. It's very stable and performant in the verticals it has been used in at Yahoo but not as widely integrated, understood and optimised as Kafka across a broader set of use cases.
That said if what you are trying to do fits squarely in the box of things that do work well it's considerably better than Kafka in a few keys ways.
The first is the obvious separation of storage and compute/broker responsibilities, the benefits of which also led to the same design being used in LogDevice - a similar system designed and built completely independently and at roughly the same time at Facebook.
The second is selective acknowledgement, i.e the ability to acknowledge messages as processed by a consumer out of order rather than merely the offset of the latest message processed. This allows Pulsar to be more easily used as a workqueue without the multitude of hacks and layered infrastructure required to get the same out of Kafka.
Shared subscriptions/partitioning model. Compared to Kafka its more flexible, less punishing and less beholden to the architecture of consumers.
Finally I would say tiering, for some it means nothing but depending on your use case it can be defining. Pulsar can offload historical segements to long term stable storage but still present a unified offset like API to consume historical data.
So I think for some Pulsar provides enough benefit to make up for any shortcomings in integration as long as you have sufficient engineers able to debug/patch issues.
- EdwardDiego 6y agoI agree wholeheartedly, especially with the last paragraph :)