3 ms·
Pulsar doesn't support transactions e.g. producing to multiple topics/partitions atomically. So, you cannot have exactly-once delivery consuming from a topic a
by tgrez 8y ago
Pulsar doesn't support transactions e.g. producing to multiple topics/partitions atomically.
So, you cannot have exactly-once delivery consuming from a topic and producing to another topic. Kafka starting from 0.11 supports this.
- manigandham 8y agoThat's true. KSQL is another thing that Kafka wins at, which makes a pretty smooth experience for straightforward stream processing and general management over the command line tools. Pulsar is working on Presto integration for querying which should be interesting.
- refset 8y agoIsn't that what Pulsar Functions are designed for? https://streaml.io/blog/pulsar-functions https://streaml.io/blog/pulsar-functions This presentation compares the Pulsar Function approach against Kafka's 2PC: https://www.slideshare.net/merlimat/effectivelyonce-semantics-in-apache-pulsar https://www.slideshare.net/merlimat/effectivelyonce-semantic...
- tgrez 8y agoPulsar Functions appears to be a stream processing engine similar to Flink or Kafka Streams. In other words, it is more like a runtime for streaming apps. The relation Pulsar ~ Pulsar Functions appears similar to Kafka ~ Kafka Streams. One is a message broker, the other a streaming engine. Exactly-once (or effectively-once) in Pulsar Functions is depended on assigning unique sequence ids on the producer side. When consuming from Pulsar topic those can be taken from the consumed message, but AFAIK Pulsar doesn't assign those unique ids on its own, so it needs to be done externally.