I've spent a lot of time poking it and running it through scenarios to get an understanding of it - haven't used it in production though. Might look at triallin

I've spent a lot of time poking it and running it through scenarios to get an understanding of it - haven't used it in production though. Might look at trialling it there in a year or so, once some of the early bugs are shaken out. It has a lot of promise, but adopting it right now is going to require you spending a lot of time reading source code when you, like I did, find the docs are out of date, or have giant holes. Things I liked: Tiered storage - the ability to offload old data to S3 / Azure / HDFS, and then reload it upon transparently by a consumer, is awesome. (albeit with some delay, but their reasoning is that historical reads don't need millisecond latency) Inter-cluster replication is a lot easier than running up a Kafka Connect cluster to run Mirror Maker 2, but there's a slight caveat that it doesn't work for chained clusters (A -> B -> C). A message from A won't be replicated to C by B. But Pulsar assumes A is going to replicate to both of them if they need it. Schema registry baked in is nice. Pulsar's replication automatically replicating the schema to other clusters is verrrrry nice for consumers on a different cluster. It's doable with Confluent's Schema Registry, but it's another thing to manage. Load balancing brokers - haven't seen it any action, but they attempt to redistribute partition ownership based on load. Only downsides were the docs were sparse or out-of-date in places, especially the BookKeeper ones. There were some configuration fun and odd errors with the BK command line tools (and they could never tell me which bookie was the auditor) and there's two scripts that do very similar things, but not quite. That said, having an auditor process that checks for under-replicated segments automatically is nice, once it's working. There's several tools for Kafka that do similar, but nice to have it out of the box. Can't really comment on its capacities to act as a message queue, but it has that too. But yeah, I reckon I'll look again in a year. On an interesting note, Pulsar is very similar design to the PubSub system - near-stateless easily scalable brokers with (another layer) then BK as the storage layer. They moved to Kafka, but then, they have Twitter sized problems so YMMV. https://blog.twitter.com/engineering/en_us/topics/insights/2018/twitters-kafka-adoption-story.html https://blog.twitter.com/engineering/en_us/topics/insights/2...