4 ms·

My comment is mostly about part 2 of this post, but wrt message ordering being a kafka "win" I'd raise the point that in the actual use case of "a consumer fail

My comment is mostly about part 2 of this post, but wrt message ordering being a kafka "win" I'd raise the point that in the actual use case of "a consumer fails in some way to process the message" you can still end up with out of order processing of the consumer's input since you might want to dump them into a DLQ or something. The fact that the message isn't reappended to the topic by default for processing is kind of an academic point no? Unrelatedly, I've been looking at Pulsar lately. Anyone have experience with Pulsar and either RMQ/Kafka want to throw out some opinions from having tried both?

EdwardDiego 3y ago
Pulsar can have both MQ semantics and pub/sub semantics. In pub/sub it's sorta like "Kafka with all bits people found it necessary to build later already built in", e.g., a proxy, schema registry, connectors, replication, tiered storage, all out of the box. It also has lightweight streaming functions built-in, but they operate per record, so good for lightweight transforms/routing, not for stream aggregations etc. It has more moving parts also, brokers are decoupled from storage, which is handled by BookKeeper, and replication between two clusters requires a ZooKeeper that both clusters can access, in addition to the ZK used by the brokers and bookies. And it's a reasonably new project, so last time I looked into it, some of the documentation was incorrect, especially around managing bookies.
- relay23 3y ago
  Massive and complex platform.. at a certain point why not just run 2 different platforms that are best of breed for each.
  - olavgg 3y ago
    You can just run Pulsar standalone which is super-simple, it is a great start and scales just fine for one machine. Once you outscale one machine, you can refactor Pulsar into a distributed setup. The advantage is that everything is still familiar.

EdwardDiego 3y ago

Pulsar can have both MQ semantics and pub/sub semantics. In pub/sub it's sorta like "Kafka with all bits people found it necessary to build later already built in", e.g., a proxy, schema registry, connectors, replication, tiered storage, all out of the box. It also has lightweight streaming functions built-in, but they operate per record, so good for lightweight transforms/routing, not for stream aggregations etc. It has more moving parts also, brokers are decoupled from storage, which is handled by BookKeeper, and replication between two clusters requires a ZooKeeper that both clusters can access, in addition to the ZK used by the brokers and bookies. And it's a reasonably new project, so last time I looked into it, some of the documentation was incorrect, especially around managing bookies.

relay23 3y ago
Massive and complex platform.. at a certain point why not just run 2 different platforms that are best of breed for each.
- olavgg 3y ago
  You can just run Pulsar standalone which is super-simple, it is a great start and scales just fine for one machine. Once you outscale one machine, you can refactor Pulsar into a distributed setup. The advantage is that everything is still familiar.