4 ms·
Kafka promises strictly ordered delivery, PubSub promises mostly ordered. The differences between those promises are what drive PubSubs ability to scale throug
by kasey_junk 11y ago
Kafka promises strictly ordered delivery, PubSub promises mostly ordered. The differences between those promises are what drive PubSubs ability to scale throughput and global availability.
From an availability standpoint, I don't disagree with anything you mention, but the difference between the consistency models means that PubSub is solving a different set of problems than Kafka, thus my opinion that comparing them is problematic.
- vgt 11y agoThat's a fair point. But remember, Kafka promises this as long as the underlying VM infrastructure is alive and well. PubSub completely removes this worry, or even the concept of VMs. There are several ways to look at it, but I'd opine that a "mostly ordered" fully-managed truly-global service that's easy to unscramble on the receiving end is more "guaranteed" than something that is single-zone and relies on the health of underlying VMs that YOU have to manage. edit: Kafka and PubSub have a lot of overlap, but they each have qualities the other one doesn't. I suppose you gotta choose which qualities are more important for you.
- kasey_junk 11y agoIf you can design your protocol such that it can work in a mostly ordered fashion, I'd highly recommend that you do. It opens up your choices for technology stack tremendously. But, if you require ordered delivery, your choices start shrinking dramatically. Also, just so we are on the same page. Kafka is a software product that can be run on hardware or VMs, not a managed service. Possibly, you are thinking of the Amazon Kinesis product which does offer a managed service with strict ordering.
- vgt 11y agoAgree on first point. No confusion on second point. My argument was that Kafka adds significant complexity and delivery risk because it's software that you must run on hardware/VMs, rather than a fully-managed service. You have to pay a whole lot of eng time to make Kafka truly "guaranteed delivery" because there's always risk of underlying hardware/VM/LB dying. Pubsub guarantees delivery regardless of what happens with underlying infrastructure. In a sense, the bar has been raised dramatically.