7 ms·

Pulsar – an open-source distributed pub-sub messaging platform

155 points by dragonsh 6y ago

hestefisk 6y ago
Anyone using this? Looks interesting.
- samdung 6y ago
  Not yet, but reading the docs. Has 'once only delivery' which is good. Still trying to find if it does FIFO (i think its hard to achieve this in a distributed system).
  - oweiler 6y ago
    What do you mean by FIFO? Messages delivered in insertion order? With Kafka, this is easy: messages with the same key land in the same partition. I guess Pulsar offers something similar.
  - jackvanlightly 6y ago
    Yes it does FIFO. Just like RabbitMQ, Apache Kafka and many other distributed systems, writes go through an elected leader, who is able to ensure ordering guarantees. Specifically with Apache Pulsar, each topic has an owner broker (leader) who accepts writes and serves readers. It should be noted that Apache Pulsar supports shared subscriptions which allow two or more consumers to compete for the same messages, like having two consumers on a RabbitMQ queue. Here FIFO order cannot be guaranteed for all kinds of reasons.
    - SuddsMcDuff 6y ago
      See also: competing consumers pattern https://www.enterpriseintegrationpatterns.com/patterns/messaging/CompetingConsumers.html https://www.enterpriseintegrationpatterns.com/patterns/messa...
  - dikei 6y ago
    https://pulsar.apache.org/docs/en/concepts-messaging/#ordering-guarantee https://pulsar.apache.org/docs/en/concepts-messaging/#orderi... Basically, they have the same ordering guarantee as Kafka: FIFO is guaranteed for messages from the same producer, within the same partition. If you need producer-level FIFO, then you can only use 1 partition. There's no ordering guarantee for messages coming from different producer.
    - eis 6y ago
      To add: In a distributed system, an ordering guarantee by producer (and optionally partition "key" which in the end is like an extension of the topic key) is pretty much all you need and all you can get. If you have two producers then how would one decide which one sent the message first? Go by some timestamp? Clocks are unreliable for these purposes so it comes down to consesus. And letting the queue decide which one came first is equivalent. Once a message got acknowledged by the queue, ordering cannot change anymore.
- jgaa 6y ago
  Yes. I use it in setup where some streams are Geo-replicated to other continents. Most of the time, it just works. Here is a performance test program I wrote to see what I could squeeze out of it: https://github.com/jgaa/pulsartest
- EdwardDiego 6y ago
  I've spent a lot of time poking it and running it through scenarios to get an understanding of it - haven't used it in production though. Might look at trialling it there in a year or so, once some of the early bugs are shaken out. It has a lot of promise, but adopting it right now is going to require you spending a lot of time reading source code when you, like I did, find the docs are out of date, or have giant holes. Things I liked: Tiered storage - the ability to offload old data to S3 / Azure / HDFS, and then reload it upon transparently by a consumer, is awesome. (albeit with some delay, but their reasoning is that historical reads don't need millisecond latency) Inter-cluster replication is a lot easier than running up a Kafka Connect cluster to run Mirror Maker 2, but there's a slight caveat that it doesn't work for chained clusters (A -> B -> C). A message from A won't be replicated to C by B. But Pulsar assumes A is going to replicate to both of them if they need it. Schema registry baked in is nice. Pulsar's replication automatically replicating the schema to other clusters is verrrrry nice for consumers on a different cluster. It's doable with Confluent's Schema Registry, but it's another thing to manage. Load balancing brokers - haven't seen it any action, but they attempt to redistribute partition ownership based on load. Only downsides were the docs were sparse or out-of-date in places, especially the BookKeeper ones. There were some configuration fun and odd errors with the BK command line tools (and they could never tell me which bookie was the auditor) and there's two scripts that do very similar things, but not quite. That said, having an auditor process that checks for under-replicated segments automatically is nice, once it's working. There's several tools for Kafka that do similar, but nice to have it out of the box. Can't really comment on its capacities to act as a message queue, but it has that too. But yeah, I reckon I'll look again in a year. On an interesting note, Pulsar is very similar design to the PubSub system - near-stateless easily scalable brokers with (another layer) then BK as the storage layer. They moved to Kafka, but then, they have Twitter sized problems so YMMV. https://blog.twitter.com/engineering/en_us/topics/insights/2018/twitters-kafka-adoption-story.html https://blog.twitter.com/engineering/en_us/topics/insights/2...
- ckdarby 6y ago
  Yes, I'm using it in production and we're transitioning from our existing over entirely to it. Running with official helm charts in AWS managed kubernetes and using EBS. Currently passing about 20k/s in and 20k/s out. Got any specific questions just let me know.
fralix 6y ago
Indeed, Apache Pulsar is a very good project, but as far as I am concerned, I prefer RabbitMQ in Production env and for little projects, Redis is enough to take this role.
- sradman 6y ago
  As I understand it, Pulsar combines Kafka-like event processing with RabbitMQ-like message queueing in a single system. It came out of Yahoo, is JVM based, and is often combined with Flink or Spark for the streaming interface. At its heart is the Bookkeeper engine which is a wrapper on RocksDB. The comparisons I’ve seen are with Kafka and its advantage is a cleaner and more flexible cluster architecture.
- nostrebored 6y ago
  Pulsar and Rabbit MQ are fundamentally different products; viewing pulsar like a push based message queue is missing a lot of the features that make it so suitable for event driven architectures.
  - eternalban 6y ago
    RabbitMQ simply does not like holding on to data. Performance craters with long queues. RabbitMQ is accurately a “message broker” and quite an excellent one at that. Tooling is fantastic, and the software (built on Erlang/OTP) is very reliable. But it is not a “distributed log”. Pulsar (as Kafka) is built on a distributed ledger/log. People confuse ‘semantics’ of messaging with “message broker” so equate various products supporting ‘messaging semantics’. To your point, and to doubters, simply try building an event sourcing system (complete with replays to recover) on RabbitMQ and see how that works out!
    - jackvanlightly 6y ago
      Not "currently" a distributed log, but we're working on adding that (in a Rabbity way). We're not trying to compete with the likes of Pulsar or Kafka though, we're just trying to round out RabbitMQ's functionality to ensure it remains the best swiss army knife of messaging - and log semantics (and performance) is now a dominant paradigm.
tannhaeuser 6y ago
What's up with github?
- donut 6y ago
  https://www.githubstatus.com/ https://www.githubstatus.com/ > Update - We have identified the source of elevated errors and are working on recovery. > Jul 13, 05:53 UTC
  - bArray 6y ago
    A recent version can be accessed here: https://web.archive.org/web/20200713060005/https://github.com/apache/pulsar https://web.archive.org/web/20200713060005/https://github.co...
gas9S9zw3P9c 6y ago
I've used Kafka a lot in the past and I've been looking at Pulsar. The broker being stateless is great. I've had issues with moving partitions around in Kafka. That being said, I'm getting tired of managing these clunky memory-hungry JVM-based systems that rely on external dependencies like Apache Zookeeper and BookKeeper. They may be necessary if you're Yahoo, but I would argue that for the vast majority of companies, these complex systems create so many administration headaches that the net productivity impact is negative. I've spend days or weeks debugging Kafka issues. I almost feel like "big clunky heavy JVM enterprise software" has become synonymous with Apache projects. If you don't absolutely need the messaging guarantees I would recommend looking into NATS (https://nats.io/ https://nats.io/) - it's a brokered systems that's significantly more lightweight and super easy to deploy and play around with. You can get persistence and delivery guarantees with NATS streaming, but that part is optional and a bit more early stage.
- KptMarchewa 6y ago
  >That being said, I'm getting tired of managing these clunky memory-hungry JVM-based systems that rely on external dependencies like Apache Zookeeper and BookKeeper. One of my pet peeves with Kafka were CLI tools written in JVM languages. Slow JVM startup time was killing my focus.
  - gas9S9zw3P9c 6y ago
    Yep. I'm using containers and Kubernetes in production so I'm trying to stay far away from anything JVM-based. On top of being slow to start, memory hungry, and plagued by GC issues, JVM docker images are always huge. I don't want two layers of virtualization. If Docker and k8s become more mainstream that's probably bad news for a lot of JVM projects. Of course, Pulsar itself is quite old (2013?) so it was built well before any of the containerization stuff took off.
    - ckdarby 6y ago
      I don't understand the relationship between containers, kubernetes and JVM. I run a prod Pulsar cluster using helm charts. All containers & kubernetes, zero issue.
sciurus 6y ago
Recent discussion of Pulsar vs Kafka: https://news.ycombinator.com/item?id=23787958 https://news.ycombinator.com/item?id=23787958
KoftaBob 6y ago
Any other Publix fans here that thought this was referring to their sub sandwiches for a second?
- taywrobel 6y ago
  Given the technical audience of hacker news, I can all but assure you that no one else here who reads “pub-sub” defaults to thinking “Publix Sub Sandwiches”
  - ryaan_anthony 6y ago
    Publix makes great subs but i definitely wasn't thinking about them at first.

155 points by dragonsh 6y ago

hestefisk 6y ago

Anyone using this? Looks interesting.

samdung 6y ago
Not yet, but reading the docs. Has 'once only delivery' which is good. Still trying to find if it does FIFO (i think its hard to achieve this in a distributed system).
- oweiler 6y ago
  What do you mean by FIFO? Messages delivered in insertion order? With Kafka, this is easy: messages with the same key land in the same partition. I guess Pulsar offers something similar.
- jackvanlightly 6y ago
  Yes it does FIFO. Just like RabbitMQ, Apache Kafka and many other distributed systems, writes go through an elected leader, who is able to ensure ordering guarantees. Specifically with Apache Pulsar, each topic has an owner broker (leader) who accepts writes and serves readers. It should be noted that Apache Pulsar supports shared subscriptions which allow two or more consumers to compete for the same messages, like having two consumers on a RabbitMQ queue. Here FIFO order cannot be guaranteed for all kinds of reasons.
  - SuddsMcDuff 6y ago
    See also: competing consumers pattern https://www.enterpriseintegrationpatterns.com/patterns/messaging/CompetingConsumers.html https://www.enterpriseintegrationpatterns.com/patterns/messa...
- dikei 6y ago
  https://pulsar.apache.org/docs/en/concepts-messaging/#ordering-guarantee https://pulsar.apache.org/docs/en/concepts-messaging/#orderi... Basically, they have the same ordering guarantee as Kafka: FIFO is guaranteed for messages from the same producer, within the same partition. If you need producer-level FIFO, then you can only use 1 partition. There's no ordering guarantee for messages coming from different producer.
  - eis 6y ago
    To add: In a distributed system, an ordering guarantee by producer (and optionally partition "key" which in the end is like an extension of the topic key) is pretty much all you need and all you can get. If you have two producers then how would one decide which one sent the message first? Go by some timestamp? Clocks are unreliable for these purposes so it comes down to consesus. And letting the queue decide which one came first is equivalent. Once a message got acknowledged by the queue, ordering cannot change anymore.
jgaa 6y ago
Yes. I use it in setup where some streams are Geo-replicated to other continents. Most of the time, it just works. Here is a performance test program I wrote to see what I could squeeze out of it: https://github.com/jgaa/pulsartest
EdwardDiego 6y ago
I've spent a lot of time poking it and running it through scenarios to get an understanding of it - haven't used it in production though. Might look at trialling it there in a year or so, once some of the early bugs are shaken out. It has a lot of promise, but adopting it right now is going to require you spending a lot of time reading source code when you, like I did, find the docs are out of date, or have giant holes. Things I liked: Tiered storage - the ability to offload old data to S3 / Azure / HDFS, and then reload it upon transparently by a consumer, is awesome. (albeit with some delay, but their reasoning is that historical reads don't need millisecond latency) Inter-cluster replication is a lot easier than running up a Kafka Connect cluster to run Mirror Maker 2, but there's a slight caveat that it doesn't work for chained clusters (A -> B -> C). A message from A won't be replicated to C by B. But Pulsar assumes A is going to replicate to both of them if they need it. Schema registry baked in is nice. Pulsar's replication automatically replicating the schema to other clusters is verrrrry nice for consumers on a different cluster. It's doable with Confluent's Schema Registry, but it's another thing to manage. Load balancing brokers - haven't seen it any action, but they attempt to redistribute partition ownership based on load. Only downsides were the docs were sparse or out-of-date in places, especially the BookKeeper ones. There were some configuration fun and odd errors with the BK command line tools (and they could never tell me which bookie was the auditor) and there's two scripts that do very similar things, but not quite. That said, having an auditor process that checks for under-replicated segments automatically is nice, once it's working. There's several tools for Kafka that do similar, but nice to have it out of the box. Can't really comment on its capacities to act as a message queue, but it has that too. But yeah, I reckon I'll look again in a year. On an interesting note, Pulsar is very similar design to the PubSub system - near-stateless easily scalable brokers with (another layer) then BK as the storage layer. They moved to Kafka, but then, they have Twitter sized problems so YMMV. https://blog.twitter.com/engineering/en_us/topics/insights/2018/twitters-kafka-adoption-story.html https://blog.twitter.com/engineering/en_us/topics/insights/2...
ckdarby 6y ago
Yes, I'm using it in production and we're transitioning from our existing over entirely to it. Running with official helm charts in AWS managed kubernetes and using EBS. Currently passing about 20k/s in and 20k/s out. Got any specific questions just let me know.

fralix 6y ago

Indeed, Apache Pulsar is a very good project, but as far as I am concerned, I prefer RabbitMQ in Production env and for little projects, Redis is enough to take this role.

sradman 6y ago
As I understand it, Pulsar combines Kafka-like event processing with RabbitMQ-like message queueing in a single system. It came out of Yahoo, is JVM based, and is often combined with Flink or Spark for the streaming interface. At its heart is the Bookkeeper engine which is a wrapper on RocksDB. The comparisons I’ve seen are with Kafka and its advantage is a cleaner and more flexible cluster architecture.
nostrebored 6y ago
Pulsar and Rabbit MQ are fundamentally different products; viewing pulsar like a push based message queue is missing a lot of the features that make it so suitable for event driven architectures.
- eternalban 6y ago
  RabbitMQ simply does not like holding on to data. Performance craters with long queues. RabbitMQ is accurately a “message broker” and quite an excellent one at that. Tooling is fantastic, and the software (built on Erlang/OTP) is very reliable. But it is not a “distributed log”. Pulsar (as Kafka) is built on a distributed ledger/log. People confuse ‘semantics’ of messaging with “message broker” so equate various products supporting ‘messaging semantics’. To your point, and to doubters, simply try building an event sourcing system (complete with replays to recover) on RabbitMQ and see how that works out!
  - jackvanlightly 6y ago
    Not "currently" a distributed log, but we're working on adding that (in a Rabbity way). We're not trying to compete with the likes of Pulsar or Kafka though, we're just trying to round out RabbitMQ's functionality to ensure it remains the best swiss army knife of messaging - and log semantics (and performance) is now a dominant paradigm.

tannhaeuser 6y ago

What's up with github?

donut 6y ago
https://www.githubstatus.com/ https://www.githubstatus.com/ > Update - We have identified the source of elevated errors and are working on recovery. > Jul 13, 05:53 UTC
- bArray 6y ago
  A recent version can be accessed here: https://web.archive.org/web/20200713060005/https://github.com/apache/pulsar https://web.archive.org/web/20200713060005/https://github.co...

gas9S9zw3P9c 6y ago

I've used Kafka a lot in the past and I've been looking at Pulsar. The broker being stateless is great. I've had issues with moving partitions around in Kafka. That being said, I'm getting tired of managing these clunky memory-hungry JVM-based systems that rely on external dependencies like Apache Zookeeper and BookKeeper. They may be necessary if you're Yahoo, but I would argue that for the vast majority of companies, these complex systems create so many administration headaches that the net productivity impact is negative. I've spend days or weeks debugging Kafka issues. I almost feel like "big clunky heavy JVM enterprise software" has become synonymous with Apache projects. If you don't absolutely need the messaging guarantees I would recommend looking into NATS (https://nats.io/ https://nats.io/) - it's a brokered systems that's significantly more lightweight and super easy to deploy and play around with. You can get persistence and delivery guarantees with NATS streaming, but that part is optional and a bit more early stage.

KptMarchewa 6y ago
>That being said, I'm getting tired of managing these clunky memory-hungry JVM-based systems that rely on external dependencies like Apache Zookeeper and BookKeeper. One of my pet peeves with Kafka were CLI tools written in JVM languages. Slow JVM startup time was killing my focus.
- gas9S9zw3P9c 6y ago
  Yep. I'm using containers and Kubernetes in production so I'm trying to stay far away from anything JVM-based. On top of being slow to start, memory hungry, and plagued by GC issues, JVM docker images are always huge. I don't want two layers of virtualization. If Docker and k8s become more mainstream that's probably bad news for a lot of JVM projects. Of course, Pulsar itself is quite old (2013?) so it was built well before any of the containerization stuff took off.
  - ckdarby 6y ago
    I don't understand the relationship between containers, kubernetes and JVM. I run a prod Pulsar cluster using helm charts. All containers & kubernetes, zero issue.

sciurus 6y ago

Recent discussion of Pulsar vs Kafka: https://news.ycombinator.com/item?id=23787958 https://news.ycombinator.com/item?id=23787958

KoftaBob 6y ago

Any other Publix fans here that thought this was referring to their sub sandwiches for a second?

taywrobel 6y ago
Given the technical audience of hacker news, I can all but assure you that no one else here who reads “pub-sub” defaults to thinking “Publix Sub Sandwiches”
- ryaan_anthony 6y ago
  Publix makes great subs but i definitely wasn't thinking about them at first.