20 ms·

Apache Pulsar is an open-source distributed pub-sub messaging system

587 points by LinuxBender 7y ago

eerrt 7y ago
How does this compare to Redis Pub-Sub or RabbitMQ?
- xrd 7y ago
  Or Firebase?
  - Eikon 7y ago
    Or Kafka ?
    - sz4kerto 7y ago
      Kafka is primarily designed for streaming, pulsar is both for streaming and queueing. Firebase is a completely different animal.
    - qaq 7y ago
      Scales much better the storage layer is separate from brokers so you can scale things independently.
    - manigandham 7y ago
      Read the other comment thread: https://news.ycombinator.com/item?id=21936523 https://news.ycombinator.com/item?id=21936523
- sz4kerto 7y ago
  Very different. Pulsar is primarily a Kafka competitor. - it is much more performant than RabbitMQ - it's a commit log as well, not just a pub-sub system, ie. it is a good candidate as the storage backend for event sourcing - it supports geodistributed and tiered storage (eg. some data on NVMe drives, some on a coldline storage) - it's persistent, not in-memory (primarily) .. and so on.
  - EGreg 7y ago
    What about ZeroMQ? Why use RabbitMQ and Kafka if you can use ZeroMQ? Meaning, isn’t it far more performant and distributed? Maybe I am missing something here.
    - Joeri 7y ago
      Message queues and message logs do different things. The idea of the log is that subscribers can show up after the log is written and read or reread it from the beginning. In an event sourced architecture you use the log as the source of truth and all consumers can replay the log against a local store to reconstruct a view of the system’s state. You also can use a log for pubsub, but if that’s all you need one of the MQ solutions is usually a better fit.
    - sz4kerto 7y ago
      > Why use RabbitMQ and Kafka if you can use ZeroMQ? They are totally different, you're comparing apples with oranges. ZeroMQ gives you basic, very fast tooling to communicate between distributed processes. ZeroMQ does not provide tooling for e.g. maintaining a strictly ordered, multi-terabyte event log. And so on.
      - EGreg 7y ago
        Yes but isn’t this a bit like comparing git / bitkeeper vs subversion / perforce? Basically, one is decentralized and you can set up a massively parallel architecture, with eg each topic or subthread having its own pubsub. The other is a monolithic centralized pubsub architecture. You could argue that git in large institutional projects converges to a monolithic repo so at that point it’s less efficient even than svn. But for most use cases, ZeroMQ would allow far more flexible distributed systems topologies and solutions. No? Edit: HN and Google are both awesome: https://news.ycombinator.com/item?id=9634925 https://news.ycombinator.com/item?id=9634925
        frogger23123 7y ago
        > You could argue that git in large institutional projects converges to a monolithic repo so at that point it’s less efficient even than svn. Not true. Facebook and Google do not use Git. Microsoft does not use vanilla Git for their monorepo. They created this extension to make it scalable https://en.wikipedia.org/wiki/Virtual_File_System_for_Git https://en.wikipedia.org/wiki/Virtual_File_System_for_Git
- qaq 7y ago
  This scales to multi-datacenter deployments well. Has strong multi-tenancy support if memory serves Yahoo is running a single cluster for all of their properties.
- tyingq 7y ago
  It's a persistent store, so that would be different from Redis Pub-Sub. Compared to RabbitMQ, Pulsar seems to favor strong ordering and protection from message loss. This blog post offers some more info and leaders to other posts comparing Pulsar to RabbitMQ and Kafka: https://jack-vanlightly.com/blog/2018/10/2/understanding-how-apache-pulsar-works https://jack-vanlightly.com/blog/2018/10/2/understanding-how...
- Joeri 7y ago
  It’s closer to redis streams, except like kafka you can scale topics beyond the limits of a single server because they can be distributed. You couldn’t run the twitter firehose over redis streams, but you can run it over pulsar or kafka, given enough hardware.
- deleted 7y ago
  [deleted]
samzer 7y ago
For a moment I thought Bajaj and TVS came together.
- deleted 7y ago
  [deleted]
- lonesword 7y ago
  Captain here. TVS and Bajaj are major motorbike manufacturers in India, and TVS had a model named "Apache" and Bajaj had a model named "Pulsar". Flies away
  - opendomain 7y ago
    Thank you for the explanation.
  - takeda 7y ago
    It's not obvious if you're not from India, so thank you for the explanation :)
js4ever 7y ago
"high-level APIs for Java, C++, Python and GO", no love for Node.js? :(
- tyingq 7y ago
  It exists: https://pulsar.apache.org/docs/en/next/client-libraries-node/ https://pulsar.apache.org/docs/en/next/client-libraries-node...
  - c0brac0bra 7y ago
    However it currently lacks the ability to listen for messages and run an event handler when one comes in: https://github.com/apache/pulsar-client-node/pull/56 https://github.com/apache/pulsar-client-node/pull/56 You have to manually call ".receive()" to attempt to receive a message.
    - gperinazzo 7y ago
      Using `.receive()` will occupy a worker thread from node until it returns. Having multiple consumers waiting on receive will clog up the worker threadpool, preventing anything that uses it from running. If you want to use the consumer right now, I would suggest always using a timeout on the receive call, and waiting between timed-out calls to receive. This is extremely important if you have multiple consumers.
- maxtollenaar 7y ago
  You can also use the websocket proxy: https://pulsar.apache.org/docs/en/next/client-libraries-websocket/ https://pulsar.apache.org/docs/en/next/client-libraries-webs...
bovermyer 7y ago
How does this compare with NATS?
- sqreept 7y ago
  NATS is a simpler PUB/SUB system that delivers in the UNIX spirit of small composable parts. Apache Pulsar or Apache Kafka deliver the banana, the ape holding it and the rest of the jungle.
  - tylertreat 7y ago
    Check out Liftbridge (https://liftbridge.io https://liftbridge.io) as a way to add these capabilities to NATS.
    - omegabravo 7y ago
      your FAQ still says it's not production ready? Is this still the case, I've been keeping my eye on this project
      - tylertreat 7y ago
        It's getting very close. I had wanted to make a production-ready 1.0 release before the end of the year, but we're in the process of switching from protobuf to flatbuffers. Once that is complete, a stable release will be made.
- manigandham 7y ago
  NATS is ephemeral pub/sub only. There is no persistence or replay, but focuses on high performance and messaging patterns like request/reply. Kafka and Pulsar persist every message and different consumers can replay the stream from their own positions. Pulsar also supports ephemeral pub/sub like NATS with a lot more advanced features. NATS does have the NATS Streaming project for persistence and replay but it has scalability issues. They're working on a new project called Jetstream to replace this in the future.
barbarbar 7y ago
How is it compared to kafka?
- SkyRocknRoll 7y ago
  Most of the flaws of Kafka are carefully studied and fixed in Apache pulsar. I have written a blog about why we went ahead with pulsar https://medium.com/@yuvarajl/why-nutanix-beam-went-ahead-with-apache-pulsar-instead-of-apache-kafka-1415f592dbbb https://medium.com/@yuvarajl/why-nutanix-beam-went-ahead-wit...
  - progval 7y ago
    > when consumers are lagging behind, producer throughput falls off a cliff because lagging consumers introduce random reads I am confused by this. The format of Kafka's log files is designed to allow reading and sending to clients directly using sendfile, in sequential reads of batches of messages. http://kafka.apache.org/documentation/#maximizingefficiency http://kafka.apache.org/documentation/#maximizingefficiency
    - geeio 7y ago
      Kafka works best when the data it is returning to consumers is in the page cache. When consumers fall behind, they start to request data that might not be in the page cache, causing things to slow down.
    - manigandham 7y ago
      Kafka brokers handle connections to consumers and data storage. This creates contention as the primaries for each partition have to service the traffic and handle IO. Consumers that aren't tailing the stream will cause slowdowns because Kafka has to seek to that offset from files which aren't cached in RAM. Pulsar separates storage into a different layer (powered by Apache Bookkeeper) which allows consumers to read directly from multiple nodes. There's much more IO throughput available to handle consumers picking up anywhere in the stream.
  - EdwardDiego 7y ago
bsaul 7y ago
Sidenote question : Are we heading toward a split between apache/java/zookeeper stacks and go/etcd on the other ? I've seen an issue related to that question on pulsar, and this got me investigating the distributed KV part of the stack. It seems by looking at some benchmark that etcd is much more performant than zookeeper, and that to some people, operating two stacks seems like an operation maintenance cost a bit too high. Is that a valid concern ? Also, i've seen that kafka is working on removing the dependency to zookeeper, is pulsar going to take the same road ?
- matteomerli 7y ago
  > is pulsar going to take the same road ? Yes, it's in the works
  - FBISurveillance 7y ago
    Related Kafka KIP: https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A... It doesn't look like it's going to be ready anytime soon though.
  - whycombin8 7y ago
    Any plans for bookkeeper?
  - wilhow 7y ago
    Just to confirm, Pulsar has on it's roadmap to remove it's dependency on Zookeeper. Is that correct?
    - matteomerli 7y ago
      That's correct, we're moving to have a pluggable metadata store and coordination service.
      - senderista 7y ago
JackRabbitSlim 7y ago
Another over engineered Lego block for quicker dev and even less thought on design, upkeep or overhead. Now if you excuse me I need to go take my quad-core, petaflop processing power and multiple gigabytes of RAM to read email from a javascript infested, multi-byte to single byte encoded webpage hosted across half a dozen server instances scattered across the planet. CS is damned, and this is hell.
- neeleshs 7y ago
  Please do elaborate
- _frkl 7y ago
  Care to elaborate? I just started using the standalone version of pulsar for a project, it looked better designed than Kafka, resources usage looks quite acceptable so far, but i dont have much experience with any solutions in this space, so im not sure which problems im going to run into. Any suggestions for good tech/strategies for a streaming-type setup like this? Or what to do alternatively? Should i look into rolling my own?
- manigandham 7y ago
  This is one of the best designed pub/sub messaging systems available, but you don't have to use it if you don't want to.
- maximente 7y ago
  not really, people legitimately need this for truly big data. you can't reliably processing 7 trillion events per day with a completely C/unix CLI stack.
  - ivalm 7y ago
    I would bet that vast, vast majority of production kafka deployments do not see 7 trillion events per day. I bet many/most do not even see 7 billion events per day.
zackmorris 7y ago
This looks promising. Is there such thing as a generalized SQL query engine that runs over any key-value store that provides certain minimal core operations? For example, say you have a KV Store with basic mathematical Set operations like GET, SET, UNION, INTERSECT, EXCEPT, etc. The Engine would parse the SQL and then call the low-level KV Store Set operations, returning the result or updating KV pairs. This explains how Join relates to Set operations: https://blog.jooq.org/2015/10/06/you-probably-dont-use-sql-intersect-or-except-often-enough/ https://blog.jooq.org/2015/10/06/you-probably-dont-use-sql-i... Another thing I'd like is if KV stores exposed a general purpose functional programming language (maybe a LISP or a minimal stack-based language like PostScript) for running the same SQL Set operations without ugly syntax. I don't know the exact name for this. But if we had that, then we could build our own distributed databases, similar to Firebase but with a SQL interface as well, from KV stores like Pulsar. I'm thinking something similar to RethinkDB but with a more distinct/open separation of layers. The hard part would be around transactions and row locking. A slightly related question is if anyone has ever made a lock-free KV store with Set operations using something like atomic compare-and-swap (CAS) operations. There might be a way to leave requests "open" until the CAS has been fully committed. Not sure if this applies to ledger/log based databases since the transaction might already be deterministic as long as the servers have exact copies of the same query log. Edit: I wrote this thinking of something like Redis, but maybe Pulsar is only the message component and not a store. So the layering might look like: [Pulsar][KV Store (like Redis)][minimal Set operations][SQL query engine].
- PaulHoule 7y ago
  e.g. how about a complex event processing engine? Something like that will do a lot of the above, but the inference database stays managable since old data will fall out of the windows.
  - TuringNYC 7y ago
    I usually dont associated CEP with this, but is makes sense. Are they meant to operate at this level (rather than at a higher level of abstraction)? Which ones would you recommend looking at?
deleted 7y ago
[deleted]
deleted 7y ago
[deleted]
cbnotfromthere 7y ago
Can it be used to build something like a Whatsapp-like chat system? If yes, why? If no, why?
buboard 7y ago
Why is apache developing all those servers that are only useful to a handful of companies that are rich enough to build them themselves? How about building something that individuals can use, like, i dunno, apache server itself?
- Eikon 7y ago
  If you did bother to read the linked page, you would have understood that it’s a yahoo project handed over to Apache for management like many of Apache’s projects.
  - buboard 7y ago
    yeah i m talking more generally about their full list here: https://www.apache.org/ https://www.apache.org/
    - mindw0rk 7y ago
      There are a lot of projects that were handed to Apache to manage. Kafka for example was initially created by LinkedIn. So yeah, you are right, big corps are actually creating those tools, and in addition to this, giving it away as open source to public.
      - eronwright 7y ago
        Best part is that the tools are put into production before being open-sourced. In other words, they actually work.
mattboyle 7y ago
We tried to adopt this but found the documentation very lacking and a severe lack of quality client libraries for our language of choice (go).the "official" one had race conditions in the code as well as "todo" for key pieces littered throughout. There is another from comcast which is abandoned. We had a serious discussion about picking up ownership of the library or writing our own but as a small start up we didnt feel we could do it and still develop the product. I'll continue to keep an eye on pulsar but for now Kafka is the clear go to imo. It's well documented, great SAS offerings (confluent) and tons of books and training courses for it.
- ckdarby 7y ago
  > found the documentation very lacking Really? It is one of the few open source projects that we've felt has had modern documentation. How long ago was this? > As a small startup You'll spend more time & money on the OpEx cost with Kafka than picking up the client library for Pulsar.
  - cpx86 7y ago
    > You'll spend more time & money on the OpEx cost with Kafka than picking up the client library for Pulsar. Could you elaborate why this would be the case?
    - gen220 7y ago
      Not the OP, but I think they were exaggerating a bit. In practice, operating kafka is a major PITA, because it means you have to (1) choose a "flavor" wrapper (confluent seems to be a popular one), because the base project isn't easy to develop against (2) write your own wrappers of those wrappers, to keep your developers from shooting themselves in the foot with wacky defaults (3) suffer the immense pain that is authenticating topic write/reads, if that's even possible??? (4) stand up zookeeper... and probably lose some data along the way. (5) suffer zookeeper outages due to buggy code in kafka/zk (I've experienced lost production data due to unpredictable bugs in kafka/zk, but obviously YMMV). Based on my naive assessment, the kafka/zookeeper ecosystem is maybe 10x as complicated as the problem it's solving, and that shows up in the OpEx. I personally doubt that Pulsar is that much better, but it might be.
srameshc 7y ago
This is great !! What would be the easiest way to run a 3 node cluster ?
- ckdarby 7y ago
  The standalone mode will let you get started as a developer. You grab tar.gz, uncompress, run standalone.sh. There are helm charts for running an actual cluster.
addisonj 7y ago
I just finished rolling out Pulsar to 8 AWS regions with geo-replication. Messages rates are currently at about 50k msgs/sec but still in the process of migrating many more applications. We run on top of kubernetes (EKS). It took about 5 months for our implementation with a chunk of that work mostly about figuring out how to integrate our internal auth as well as a using hashicorp vault as a clean automated way to get auth tokens for an AWS IAM role. Overall, we are very pleased and the rest of the engineering org is very excited about it and planning to migrate most of our SQS and Kinesis apps. Ask me anything in thread and will try and answer questions. At some point we will do a blog post on our experience.
- rocky1138 7y ago
  Why did you choose Pulsar?
  - addisonj 7y ago
    The main driver for Pulsar is that we have a number of different messaging use cases, some more "pub/sub" like and some that are more "log" like. Pulsar really does unify those two worlds while also being a ton more flexible than any hosted options. For example, Kinesis is really limiting with the limited retention and making it very difficult to do any real ordering at scale due to the really tiny size of each shard. Similarly, SQS does pub/sub well, but we keep finding that we do need to use the data more than the first initial delivery. Instead of having multiple systems where we store that data we have one. As for why we didn't go with Kafka, the biggest single reason is that Pulsar is easier operationally with no needing to re-balance and also with the awesome feature that is tiered storage via offloading that allows us to actually do topics that have unlimited retention. Perhaps more importantly for the adoption though is pub/sub is much easier with Pulsar and the API is just much easier to reason about for developers than all the complexity of consumer groups, etc. There are a ton of other nice things like being able to have topics be so cheap such that we can have hundred of thousands and all of the built-in multi-tenancy features, geo-replication, flexible ACL system, pulsar functions and pulsar IO and many other things that really have us excited about all the capabilities
reggieband 7y ago
I'm still on the fence with these distributed log/queue hybrids. From a theoretical perspective it seems these are excellent. I just have this nagging suspicion that there is some even-worse problem architectures based on these systems will harbor. This kind of ambivalence is something I find myself having to battle more and more in my career as I age. Most of the time the hype around new design/development patterns leads to a worse situation. Very rarely it leads to a significant improvement. I dislike that my first impression looking at a system like this is risk aversion.
- mpmpmpmp 7y ago
  I totally get where you're coming from as I feel like that alot too. But the fact that you are thinking about risk and business value over working on cool new tech should be a positive for the ventures you are a part of.
- zomglings 7y ago
  Your risk aversion seems justified. It seems reasonable to estimate that very few teams are in the position of needing the kind of scale/scalability that something like Apache Pulsar offers. They are much more likely to be in either a state where they will not put Pulsar through its paces or where they already have a solution in place that serves their scale/scalability needs. When a team you are on starts discussing switching over to a technology like Pulsar because of its amazing benefits, unless your pants are on fire, it is much more likely than not that you do not stand to gain much from the benefits that such software brings but you are accepting the maintenance burden that it represents.
mbostleman 7y ago
This might be entirely off topic, but I'm having issues using RabbitMQ whereby durability suffers because messages are sent to remote hosts thus exposing them to both the network and remote host availability. On a previous platform I used an MSMQ based system which didn't have this problem since it uses a local store and forward service. So all sends are to localhost and are not affected by the network or the receiver availability. The MSMQ system was my first and only experience with messaging up to now, so I was surprised that any system would not work that way. How is this dealt with in other systems? Is it just a feature that exists or not and you just decide if it's important? And maybe just to shoe horn it to be on topic, does Pulsar use a local service?
- bauerd 7y ago
  You're free to have queue and workers run on the same machine, just bind to loopback. As soon as you deal with more than one machine, which is required in HA scenarios, you deal with a networked (distributed) system. I might not have understood your question correctly though ... Edit: Maybe you're looking for acks/confirms? https://www.rabbitmq.com/confirms.html https://www.rabbitmq.com/confirms.html
  - mbostleman 7y ago
    I have many machines, each of which have one or many applications that send messages. And I have one machine with an instance of Rabbit to which all messages are sent. If the network is down or the Rabbit machine is down, the messages are gone along with their data. Clustering the Rabbit machine helps one particular failure scenario, but it's not a solution to the problem.
    - shantly 7y ago
      I haven't worked with RabbitMQ in years, but IIRC I solved this with federation. My sends were all to localhost or a VM on the same machine. That solves the problem of lost messages if you have e.g. a network problem between servers. I don't know what to do about RabbitMQ being down, all you can really do about that is move the exact same problem somewhere else by making things much more complicated, such that it's probably not worth it. If the place you're storing outbound messages is broken, you (obviously) can't store them, whether that's PostgreSQL, Redis, a remote RabbitMQ server, a local RabbitMQ, whatever.
eatonphil 7y ago
Most of the comments are just pro-Pulsar but what's the architectural trade-off? (Non-architectural trade-off is that Pulsar is a new system to learn for folks familiar with maintaining and using Kafka.)
- qeternity 7y ago
  Someone linked some benchmarks here (on mobile and can’t find) that showed a single node Kafka outperforms but as soon as you start scaling Pulsar pulls ahead. I’m not familiar enough with the nitty gritty to comment beyond that.
- manigandham 7y ago
  Pulsar is better designed than Kafka in every way with the main trade-off being more moving pieces. That's why the recommended deployment is Kubernetes which can manage all that complexity for you. Pulsar also lacks in the size of the community and ecosystem where Kafka has much more available.
breckcs 7y ago
Being able to scale the durable-storage layer independently has a lot of advantages. More thoughts here: https://twitter.com/breckcs/status/1203736751681896449 https://twitter.com/breckcs/status/1203736751681896449.
mickster99 7y ago
Is there a reason you went with Pulsar over Kafka? How is the pulsar community? Where are you turning when you have support issues?
throwawaysea 7y ago
A lot is said or referenced in this conversation about why people chose Pulsar over Kafka. I'm not an expert in this area but are there use cases where Kafka is still better?
- EdwardDiego 7y ago
  As someone with a few years of Kafka and the ecosystem under my belt, but no experience of using Pulsar in anger, the areas I can see where Pulsar is behind are mainly ancillary, and will likely be caught up by the community given a year or two. Kafka Streaming - Pulsar functions don't intend to (by the looks of it) provide all of the functionality available in Kafka Streaming. E.g., joining streams, treating a stream as a table (a log with a key can be treated as a table), aggregating over a window. They seem to be more focused on do X or Y to a given record. That said, you don't need Kafka Streaming for that, other streaming products like Spark Streaming can do it also (although last I checked, Spark Structured Streaming still had some limitations compared to Kafka Streaming - can't do multiple aggregations on a stream etc.) A use case I have and love Kafka Streaming's KTables for is enriching/denormalising transaction records on their way through the pipeline. Kafka Connect - Pulsar IO will get there with time, but currently KC has a lot more connectors - for example, Pulsar IO's Debezium source is nice, (Debezium was built to use Kafka Connect, but can run embedded), but you may not want to publish an entire change event stream onto a topic, you might just want a view of a given database table available - so KC's JDBC connector is a lot more flexible in that regard, and Pulsar IO currently doesn't have a JDBC source. It also looks like its JDBC sink only supports MySQL and SQLite (according to the docs) - KC's JDBC connector as a sink has a wider range of support for DBs, and can take advantage of functionality like Postgres 9.5+s UPSERT. Likewise, there's no S3 sources or sinks - the tiered storage Pulsar offers is really nice, but you may only want to persist a particular topic to S3. KSQL - KSQL lets your BAs etc. write SQL queries for streams. That said, I do like Pulsar SQL's ability to query your stored log data. When I've needed to do this with Kafka, I've had to consume the data with a Spark job, which adds overhead for troubleshooting. So yeah, that's the main areas I can see, but it's really a function of time until Pulsar or community code develops similar features. The only other major difference I can see is that at the current time, it's comprised of three distributed systems (Pulsar brokers, Zookeeper, Bookkeeper) which is one more distributed system to maintain with all the fun that entails. That said, I'll be keeping my eye on this, and trialling it when I get some spare time, as I've found that people will inevitably use Kafka like a messaging queue, and that is a bit clunky. Plus I'm a little over having people ask me how many partitions they need :D
alfalfasprout 7y ago
I keep seeing new message queue solutions pop up over the years and it's just been my impression at least that this is one area where silicon valley really is way behind the trading industry. Reliable pub/sub that supports message rates over 100k/sec (even up to the millions) has been available for a while now and with a great deal of efficiency (eg; the Aeron project). The incredible amount of effort to support complex partitions, extreme fault tolerance (instead of more clever recovery logic), etc. add a lot of overhead. To the point of talking about "low latency" overhead in the order of 5ms instead of microseconds or even nanoseconds as is expected in trading. Worse, many startups try to adopt these technologies where their message rates are miniscule. To give you some context, even two beefy machines with an older message queue solution like ZeroMQ can tolerate throughput in excess of what most companies produce. This is not to discredit the authors of Pulsar or Kafka at all... but it's just a concerning trend where easy to use horizontally scalable message queues are being deployed everywhere. Similar to how everyone was running hadoop a few years back even when the data fit in memory.
- gfodor 7y ago
  Worth noting that Kafka is not a queue, but an append-only log.
- tylertreat 7y ago
  ZeroMQ is not a message queue, it's a networking library.
  - injinj 7y ago
    I'll bet he is aware. The problem with the trading industry is that they have hundreds of users with bespoke solutions catering to extreme performance criteria rather than hundreds of thousands like NATS. They will keep reinventing the wheel every time a nanosecond can be saved by the latest hardware stack.
- platform 7y ago
  Aeron[1] is something I have been look at as well. I like that it is adaptable to run without an external broker (using embedded media driver option), yet can add that for scalability/redundancy. Our use cases do not require a query on top of streams. Instead all data would go into timescaleDB [2]. [1] https://github.com/real-logic/aeron/wiki/Java-Programming-Guide https://github.com/real-logic/aeron/wiki/Java-Programming-Gu... [2] https://github.com/timescale/timescaledb https://github.com/timescale/timescaledb
firstposterone 7y ago
Splunk just acquired streamlio and most of the core devs got sucked up. While pulsar is a great product - are you not concerned that these guys are getting paid $$ bank to do something else now?
- matteomerli 7y ago
  spoiler: we're still working on Pulsar
  - firstposterone 7y ago
    That’s good news! I guess it would be very helpful to formally address this concern. Is there something that has been written / published to that effect?
- zhaijia 7y ago
  There is another company StreamNative, founded by core Pulsar/BookKeeper devs 1 year ago. :)
microcolonel 7y ago
I'm sure Pulsar is worth it if you use most of what they're offering, but the Java client library is crusty, throws exceptions for control flow. I'm looking at a persistence mechanism built on top of NATS to replace it. The NATS layer would make it simpler to decouple the gateways from the persistence layer, and support our bulk computing needs.
dikei 7y ago
From reading their documents, I really like the design of Pulsar. However, Kafka has been working so well for us and has much better integration with other components of our stack (Flink, Spark, NiFi, etc) that there's no compelling reason to switch. I think Pulse should really focus on the integration with the rest of the Apache stack if they want to gain traction.
adkafka 7y ago
I really enjoyed the talk given by Quentin Adam & Steven Le Roux at Devoxx 2019. It gave a great overview of Apache Pulsar. Hope someone else can find it useful too! [0] https://www.youtube.com/watch?v=De6avNyQUMw https://www.youtube.com/watch?v=De6avNyQUMw
jsilence 7y ago
Would Pulsar be suitable for IoT messaging? An alternative to mqtt?
manigandham 7y ago
This is one of the best overviews on Pulsar with comparisons to Kafka: https://jack-vanlightly.com/blog/2018/10/2/understanding-how-apache-pulsar-works https://jack-vanlightly.com/blog/2018/10/2/understanding-how...

587 points by LinuxBender 7y ago

eerrt 7y ago

How does this compare to Redis Pub-Sub or RabbitMQ?

xrd 7y ago
Or Firebase?
- Eikon 7y ago
  Or Kafka ?
  - sz4kerto 7y ago
    Kafka is primarily designed for streaming, pulsar is both for streaming and queueing. Firebase is a completely different animal.
  - qaq 7y ago
    Scales much better the storage layer is separate from brokers so you can scale things independently.
  - manigandham 7y ago
    Read the other comment thread: https://news.ycombinator.com/item?id=21936523 https://news.ycombinator.com/item?id=21936523
sz4kerto 7y ago
Very different. Pulsar is primarily a Kafka competitor. - it is much more performant than RabbitMQ - it's a commit log as well, not just a pub-sub system, ie. it is a good candidate as the storage backend for event sourcing - it supports geodistributed and tiered storage (eg. some data on NVMe drives, some on a coldline storage) - it's persistent, not in-memory (primarily) .. and so on.
- EGreg 7y ago
  What about ZeroMQ? Why use RabbitMQ and Kafka if you can use ZeroMQ? Meaning, isn’t it far more performant and distributed? Maybe I am missing something here.
  - Joeri 7y ago
    Message queues and message logs do different things. The idea of the log is that subscribers can show up after the log is written and read or reread it from the beginning. In an event sourced architecture you use the log as the source of truth and all consumers can replay the log against a local store to reconstruct a view of the system’s state. You also can use a log for pubsub, but if that’s all you need one of the MQ solutions is usually a better fit.
  - sz4kerto 7y ago
    > Why use RabbitMQ and Kafka if you can use ZeroMQ? They are totally different, you're comparing apples with oranges. ZeroMQ gives you basic, very fast tooling to communicate between distributed processes. ZeroMQ does not provide tooling for e.g. maintaining a strictly ordered, multi-terabyte event log. And so on.
    - EGreg 7y ago
      Yes but isn’t this a bit like comparing git / bitkeeper vs subversion / perforce? Basically, one is decentralized and you can set up a massively parallel architecture, with eg each topic or subthread having its own pubsub. The other is a monolithic centralized pubsub architecture. You could argue that git in large institutional projects converges to a monolithic repo so at that point it’s less efficient even than svn. But for most use cases, ZeroMQ would allow far more flexible distributed systems topologies and solutions. No? Edit: HN and Google are both awesome: https://news.ycombinator.com/item?id=9634925 https://news.ycombinator.com/item?id=9634925
      - frogger23123 7y ago
        > You could argue that git in large institutional projects converges to a monolithic repo so at that point it’s less efficient even than svn. Not true. Facebook and Google do not use Git. Microsoft does not use vanilla Git for their monorepo. They created this extension to make it scalable https://en.wikipedia.org/wiki/Virtual_File_System_for_Git https://en.wikipedia.org/wiki/Virtual_File_System_for_Git
qaq 7y ago
This scales to multi-datacenter deployments well. Has strong multi-tenancy support if memory serves Yahoo is running a single cluster for all of their properties.
tyingq 7y ago
It's a persistent store, so that would be different from Redis Pub-Sub. Compared to RabbitMQ, Pulsar seems to favor strong ordering and protection from message loss. This blog post offers some more info and leaders to other posts comparing Pulsar to RabbitMQ and Kafka: https://jack-vanlightly.com/blog/2018/10/2/understanding-how-apache-pulsar-works https://jack-vanlightly.com/blog/2018/10/2/understanding-how...
Joeri 7y ago
It’s closer to redis streams, except like kafka you can scale topics beyond the limits of a single server because they can be distributed. You couldn’t run the twitter firehose over redis streams, but you can run it over pulsar or kafka, given enough hardware.
deleted 7y ago
[deleted]

samzer 7y ago

For a moment I thought Bajaj and TVS came together.

deleted 7y ago
[deleted]
lonesword 7y ago
Captain here. TVS and Bajaj are major motorbike manufacturers in India, and TVS had a model named "Apache" and Bajaj had a model named "Pulsar". Flies away
- opendomain 7y ago
  Thank you for the explanation.
- takeda 7y ago
  It's not obvious if you're not from India, so thank you for the explanation :)

js4ever 7y ago

"high-level APIs for Java, C++, Python and GO", no love for Node.js? :(

tyingq 7y ago
It exists: https://pulsar.apache.org/docs/en/next/client-libraries-node/ https://pulsar.apache.org/docs/en/next/client-libraries-node...
- c0brac0bra 7y ago
  However it currently lacks the ability to listen for messages and run an event handler when one comes in: https://github.com/apache/pulsar-client-node/pull/56 https://github.com/apache/pulsar-client-node/pull/56 You have to manually call ".receive()" to attempt to receive a message.
  - gperinazzo 7y ago
    Using `.receive()` will occupy a worker thread from node until it returns. Having multiple consumers waiting on receive will clog up the worker threadpool, preventing anything that uses it from running. If you want to use the consumer right now, I would suggest always using a timeout on the receive call, and waiting between timed-out calls to receive. This is extremely important if you have multiple consumers.
maxtollenaar 7y ago
You can also use the websocket proxy: https://pulsar.apache.org/docs/en/next/client-libraries-websocket/ https://pulsar.apache.org/docs/en/next/client-libraries-webs...

bovermyer 7y ago

How does this compare with NATS?

sqreept 7y ago
NATS is a simpler PUB/SUB system that delivers in the UNIX spirit of small composable parts. Apache Pulsar or Apache Kafka deliver the banana, the ape holding it and the rest of the jungle.
- tylertreat 7y ago
  Check out Liftbridge (https://liftbridge.io https://liftbridge.io) as a way to add these capabilities to NATS.
  - omegabravo 7y ago
    your FAQ still says it's not production ready? Is this still the case, I've been keeping my eye on this project
    - tylertreat 7y ago
      It's getting very close. I had wanted to make a production-ready 1.0 release before the end of the year, but we're in the process of switching from protobuf to flatbuffers. Once that is complete, a stable release will be made.
manigandham 7y ago
NATS is ephemeral pub/sub only. There is no persistence or replay, but focuses on high performance and messaging patterns like request/reply. Kafka and Pulsar persist every message and different consumers can replay the stream from their own positions. Pulsar also supports ephemeral pub/sub like NATS with a lot more advanced features. NATS does have the NATS Streaming project for persistence and replay but it has scalability issues. They're working on a new project called Jetstream to replace this in the future.

barbarbar 7y ago

How is it compared to kafka?

SkyRocknRoll 7y ago
Most of the flaws of Kafka are carefully studied and fixed in Apache pulsar. I have written a blog about why we went ahead with pulsar https://medium.com/@yuvarajl/why-nutanix-beam-went-ahead-with-apache-pulsar-instead-of-apache-kafka-1415f592dbbb https://medium.com/@yuvarajl/why-nutanix-beam-went-ahead-wit...
- progval 7y ago
  > when consumers are lagging behind, producer throughput falls off a cliff because lagging consumers introduce random reads I am confused by this. The format of Kafka's log files is designed to allow reading and sending to clients directly using sendfile, in sequential reads of batches of messages. http://kafka.apache.org/documentation/#maximizingefficiency http://kafka.apache.org/documentation/#maximizingefficiency
  - geeio 7y ago
    Kafka works best when the data it is returning to consumers is in the page cache. When consumers fall behind, they start to request data that might not be in the page cache, causing things to slow down.
  - manigandham 7y ago
    Kafka brokers handle connections to consumers and data storage. This creates contention as the primaries for each partition have to service the traffic and handle IO. Consumers that aren't tailing the stream will cause slowdowns because Kafka has to seek to that offset from files which aren't cached in RAM. Pulsar separates storage into a different layer (powered by Apache Bookkeeper) which allows consumers to read directly from multiple nodes. There's much more IO throughput available to handle consumers picking up anywhere in the stream.
- EdwardDiego 7y ago

bsaul 7y ago

Sidenote question : Are we heading toward a split between apache/java/zookeeper stacks and go/etcd on the other ? I've seen an issue related to that question on pulsar, and this got me investigating the distributed KV part of the stack. It seems by looking at some benchmark that etcd is much more performant than zookeeper, and that to some people, operating two stacks seems like an operation maintenance cost a bit too high. Is that a valid concern ? Also, i've seen that kafka is working on removing the dependency to zookeeper, is pulsar going to take the same road ?

matteomerli 7y ago
> is pulsar going to take the same road ? Yes, it's in the works
- FBISurveillance 7y ago
  Related Kafka KIP: https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A... It doesn't look like it's going to be ready anytime soon though.
- whycombin8 7y ago
  Any plans for bookkeeper?
- wilhow 7y ago
  Just to confirm, Pulsar has on it's roadmap to remove it's dependency on Zookeeper. Is that correct?
  - matteomerli 7y ago
    That's correct, we're moving to have a pluggable metadata store and coordination service.
    - senderista 7y ago

JackRabbitSlim 7y ago

Another over engineered Lego block for quicker dev and even less thought on design, upkeep or overhead. Now if you excuse me I need to go take my quad-core, petaflop processing power and multiple gigabytes of RAM to read email from a javascript infested, multi-byte to single byte encoded webpage hosted across half a dozen server instances scattered across the planet. CS is damned, and this is hell.

neeleshs 7y ago
Please do elaborate
_frkl 7y ago
Care to elaborate? I just started using the standalone version of pulsar for a project, it looked better designed than Kafka, resources usage looks quite acceptable so far, but i dont have much experience with any solutions in this space, so im not sure which problems im going to run into. Any suggestions for good tech/strategies for a streaming-type setup like this? Or what to do alternatively? Should i look into rolling my own?
manigandham 7y ago
This is one of the best designed pub/sub messaging systems available, but you don't have to use it if you don't want to.
maximente 7y ago
not really, people legitimately need this for truly big data. you can't reliably processing 7 trillion events per day with a completely C/unix CLI stack.
- ivalm 7y ago
  I would bet that vast, vast majority of production kafka deployments do not see 7 trillion events per day. I bet many/most do not even see 7 billion events per day.

zackmorris 7y ago

This looks promising. Is there such thing as a generalized SQL query engine that runs over any key-value store that provides certain minimal core operations? For example, say you have a KV Store with basic mathematical Set operations like GET, SET, UNION, INTERSECT, EXCEPT, etc. The Engine would parse the SQL and then call the low-level KV Store Set operations, returning the result or updating KV pairs. This explains how Join relates to Set operations: https://blog.jooq.org/2015/10/06/you-probably-dont-use-sql-intersect-or-except-often-enough/ https://blog.jooq.org/2015/10/06/you-probably-dont-use-sql-i... Another thing I'd like is if KV stores exposed a general purpose functional programming language (maybe a LISP or a minimal stack-based language like PostScript) for running the same SQL Set operations without ugly syntax. I don't know the exact name for this. But if we had that, then we could build our own distributed databases, similar to Firebase but with a SQL interface as well, from KV stores like Pulsar. I'm thinking something similar to RethinkDB but with a more distinct/open separation of layers. The hard part would be around transactions and row locking. A slightly related question is if anyone has ever made a lock-free KV store with Set operations using something like atomic compare-and-swap (CAS) operations. There might be a way to leave requests "open" until the CAS has been fully committed. Not sure if this applies to ledger/log based databases since the transaction might already be deterministic as long as the servers have exact copies of the same query log. Edit: I wrote this thinking of something like Redis, but maybe Pulsar is only the message component and not a store. So the layering might look like: [Pulsar][KV Store (like Redis)][minimal Set operations][SQL query engine].

PaulHoule 7y ago
e.g. how about a complex event processing engine? Something like that will do a lot of the above, but the inference database stays managable since old data will fall out of the windows.
- TuringNYC 7y ago
  I usually dont associated CEP with this, but is makes sense. Are they meant to operate at this level (rather than at a higher level of abstraction)? Which ones would you recommend looking at?

deleted 7y ago

[deleted]

deleted 7y ago

[deleted]

cbnotfromthere 7y ago

Can it be used to build something like a Whatsapp-like chat system? If yes, why? If no, why?

buboard 7y ago

Why is apache developing all those servers that are only useful to a handful of companies that are rich enough to build them themselves? How about building something that individuals can use, like, i dunno, apache server itself?

Eikon 7y ago
If you did bother to read the linked page, you would have understood that it’s a yahoo project handed over to Apache for management like many of Apache’s projects.
- buboard 7y ago
  yeah i m talking more generally about their full list here: https://www.apache.org/ https://www.apache.org/
  - mindw0rk 7y ago
    There are a lot of projects that were handed to Apache to manage. Kafka for example was initially created by LinkedIn. So yeah, you are right, big corps are actually creating those tools, and in addition to this, giving it away as open source to public.
    - eronwright 7y ago
      Best part is that the tools are put into production before being open-sourced. In other words, they actually work.

mattboyle 7y ago

We tried to adopt this but found the documentation very lacking and a severe lack of quality client libraries for our language of choice (go).the "official" one had race conditions in the code as well as "todo" for key pieces littered throughout. There is another from comcast which is abandoned. We had a serious discussion about picking up ownership of the library or writing our own but as a small start up we didnt feel we could do it and still develop the product. I'll continue to keep an eye on pulsar but for now Kafka is the clear go to imo. It's well documented, great SAS offerings (confluent) and tons of books and training courses for it.

ckdarby 7y ago
> found the documentation very lacking Really? It is one of the few open source projects that we've felt has had modern documentation. How long ago was this? > As a small startup You'll spend more time & money on the OpEx cost with Kafka than picking up the client library for Pulsar.
- cpx86 7y ago
  > You'll spend more time & money on the OpEx cost with Kafka than picking up the client library for Pulsar. Could you elaborate why this would be the case?
  - gen220 7y ago
    Not the OP, but I think they were exaggerating a bit. In practice, operating kafka is a major PITA, because it means you have to (1) choose a "flavor" wrapper (confluent seems to be a popular one), because the base project isn't easy to develop against (2) write your own wrappers of those wrappers, to keep your developers from shooting themselves in the foot with wacky defaults (3) suffer the immense pain that is authenticating topic write/reads, if that's even possible??? (4) stand up zookeeper... and probably lose some data along the way. (5) suffer zookeeper outages due to buggy code in kafka/zk (I've experienced lost production data due to unpredictable bugs in kafka/zk, but obviously YMMV). Based on my naive assessment, the kafka/zookeeper ecosystem is maybe 10x as complicated as the problem it's solving, and that shows up in the OpEx. I personally doubt that Pulsar is that much better, but it might be.

srameshc 7y ago

This is great !! What would be the easiest way to run a 3 node cluster ?

ckdarby 7y ago
The standalone mode will let you get started as a developer. You grab tar.gz, uncompress, run standalone.sh. There are helm charts for running an actual cluster.

addisonj 7y ago

I just finished rolling out Pulsar to 8 AWS regions with geo-replication. Messages rates are currently at about 50k msgs/sec but still in the process of migrating many more applications. We run on top of kubernetes (EKS). It took about 5 months for our implementation with a chunk of that work mostly about figuring out how to integrate our internal auth as well as a using hashicorp vault as a clean automated way to get auth tokens for an AWS IAM role. Overall, we are very pleased and the rest of the engineering org is very excited about it and planning to migrate most of our SQS and Kinesis apps. Ask me anything in thread and will try and answer questions. At some point we will do a blog post on our experience.

rocky1138 7y ago
Why did you choose Pulsar?
- addisonj 7y ago
  The main driver for Pulsar is that we have a number of different messaging use cases, some more "pub/sub" like and some that are more "log" like. Pulsar really does unify those two worlds while also being a ton more flexible than any hosted options. For example, Kinesis is really limiting with the limited retention and making it very difficult to do any real ordering at scale due to the really tiny size of each shard. Similarly, SQS does pub/sub well, but we keep finding that we do need to use the data more than the first initial delivery. Instead of having multiple systems where we store that data we have one. As for why we didn't go with Kafka, the biggest single reason is that Pulsar is easier operationally with no needing to re-balance and also with the awesome feature that is tiered storage via offloading that allows us to actually do topics that have unlimited retention. Perhaps more importantly for the adoption though is pub/sub is much easier with Pulsar and the API is just much easier to reason about for developers than all the complexity of consumer groups, etc. There are a ton of other nice things like being able to have topics be so cheap such that we can have hundred of thousands and all of the built-in multi-tenancy features, geo-replication, flexible ACL system, pulsar functions and pulsar IO and many other things that really have us excited about all the capabilities

reggieband 7y ago

I'm still on the fence with these distributed log/queue hybrids. From a theoretical perspective it seems these are excellent. I just have this nagging suspicion that there is some even-worse problem architectures based on these systems will harbor. This kind of ambivalence is something I find myself having to battle more and more in my career as I age. Most of the time the hype around new design/development patterns leads to a worse situation. Very rarely it leads to a significant improvement. I dislike that my first impression looking at a system like this is risk aversion.

mpmpmpmp 7y ago
I totally get where you're coming from as I feel like that alot too. But the fact that you are thinking about risk and business value over working on cool new tech should be a positive for the ventures you are a part of.
zomglings 7y ago
Your risk aversion seems justified. It seems reasonable to estimate that very few teams are in the position of needing the kind of scale/scalability that something like Apache Pulsar offers. They are much more likely to be in either a state where they will not put Pulsar through its paces or where they already have a solution in place that serves their scale/scalability needs. When a team you are on starts discussing switching over to a technology like Pulsar because of its amazing benefits, unless your pants are on fire, it is much more likely than not that you do not stand to gain much from the benefits that such software brings but you are accepting the maintenance burden that it represents.

mbostleman 7y ago

This might be entirely off topic, but I'm having issues using RabbitMQ whereby durability suffers because messages are sent to remote hosts thus exposing them to both the network and remote host availability. On a previous platform I used an MSMQ based system which didn't have this problem since it uses a local store and forward service. So all sends are to localhost and are not affected by the network or the receiver availability. The MSMQ system was my first and only experience with messaging up to now, so I was surprised that any system would not work that way. How is this dealt with in other systems? Is it just a feature that exists or not and you just decide if it's important? And maybe just to shoe horn it to be on topic, does Pulsar use a local service?

bauerd 7y ago
You're free to have queue and workers run on the same machine, just bind to loopback. As soon as you deal with more than one machine, which is required in HA scenarios, you deal with a networked (distributed) system. I might not have understood your question correctly though ... Edit: Maybe you're looking for acks/confirms? https://www.rabbitmq.com/confirms.html https://www.rabbitmq.com/confirms.html
- mbostleman 7y ago
  I have many machines, each of which have one or many applications that send messages. And I have one machine with an instance of Rabbit to which all messages are sent. If the network is down or the Rabbit machine is down, the messages are gone along with their data. Clustering the Rabbit machine helps one particular failure scenario, but it's not a solution to the problem.
  - shantly 7y ago
    I haven't worked with RabbitMQ in years, but IIRC I solved this with federation. My sends were all to localhost or a VM on the same machine. That solves the problem of lost messages if you have e.g. a network problem between servers. I don't know what to do about RabbitMQ being down, all you can really do about that is move the exact same problem somewhere else by making things much more complicated, such that it's probably not worth it. If the place you're storing outbound messages is broken, you (obviously) can't store them, whether that's PostgreSQL, Redis, a remote RabbitMQ server, a local RabbitMQ, whatever.

eatonphil 7y ago

Most of the comments are just pro-Pulsar but what's the architectural trade-off? (Non-architectural trade-off is that Pulsar is a new system to learn for folks familiar with maintaining and using Kafka.)

qeternity 7y ago
Someone linked some benchmarks here (on mobile and can’t find) that showed a single node Kafka outperforms but as soon as you start scaling Pulsar pulls ahead. I’m not familiar enough with the nitty gritty to comment beyond that.
manigandham 7y ago
Pulsar is better designed than Kafka in every way with the main trade-off being more moving pieces. That's why the recommended deployment is Kubernetes which can manage all that complexity for you. Pulsar also lacks in the size of the community and ecosystem where Kafka has much more available.

breckcs 7y ago

Being able to scale the durable-storage layer independently has a lot of advantages. More thoughts here: https://twitter.com/breckcs/status/1203736751681896449 https://twitter.com/breckcs/status/1203736751681896449.

mickster99 7y ago

Is there a reason you went with Pulsar over Kafka? How is the pulsar community? Where are you turning when you have support issues?

throwawaysea 7y ago

A lot is said or referenced in this conversation about why people chose Pulsar over Kafka. I'm not an expert in this area but are there use cases where Kafka is still better?

EdwardDiego 7y ago
As someone with a few years of Kafka and the ecosystem under my belt, but no experience of using Pulsar in anger, the areas I can see where Pulsar is behind are mainly ancillary, and will likely be caught up by the community given a year or two. Kafka Streaming - Pulsar functions don't intend to (by the looks of it) provide all of the functionality available in Kafka Streaming. E.g., joining streams, treating a stream as a table (a log with a key can be treated as a table), aggregating over a window. They seem to be more focused on do X or Y to a given record. That said, you don't need Kafka Streaming for that, other streaming products like Spark Streaming can do it also (although last I checked, Spark Structured Streaming still had some limitations compared to Kafka Streaming - can't do multiple aggregations on a stream etc.) A use case I have and love Kafka Streaming's KTables for is enriching/denormalising transaction records on their way through the pipeline. Kafka Connect - Pulsar IO will get there with time, but currently KC has a lot more connectors - for example, Pulsar IO's Debezium source is nice, (Debezium was built to use Kafka Connect, but can run embedded), but you may not want to publish an entire change event stream onto a topic, you might just want a view of a given database table available - so KC's JDBC connector is a lot more flexible in that regard, and Pulsar IO currently doesn't have a JDBC source. It also looks like its JDBC sink only supports MySQL and SQLite (according to the docs) - KC's JDBC connector as a sink has a wider range of support for DBs, and can take advantage of functionality like Postgres 9.5+s UPSERT. Likewise, there's no S3 sources or sinks - the tiered storage Pulsar offers is really nice, but you may only want to persist a particular topic to S3. KSQL - KSQL lets your BAs etc. write SQL queries for streams. That said, I do like Pulsar SQL's ability to query your stored log data. When I've needed to do this with Kafka, I've had to consume the data with a Spark job, which adds overhead for troubleshooting. So yeah, that's the main areas I can see, but it's really a function of time until Pulsar or community code develops similar features. The only other major difference I can see is that at the current time, it's comprised of three distributed systems (Pulsar brokers, Zookeeper, Bookkeeper) which is one more distributed system to maintain with all the fun that entails. That said, I'll be keeping my eye on this, and trialling it when I get some spare time, as I've found that people will inevitably use Kafka like a messaging queue, and that is a bit clunky. Plus I'm a little over having people ask me how many partitions they need :D

alfalfasprout 7y ago

I keep seeing new message queue solutions pop up over the years and it's just been my impression at least that this is one area where silicon valley really is way behind the trading industry. Reliable pub/sub that supports message rates over 100k/sec (even up to the millions) has been available for a while now and with a great deal of efficiency (eg; the Aeron project). The incredible amount of effort to support complex partitions, extreme fault tolerance (instead of more clever recovery logic), etc. add a lot of overhead. To the point of talking about "low latency" overhead in the order of 5ms instead of microseconds or even nanoseconds as is expected in trading. Worse, many startups try to adopt these technologies where their message rates are miniscule. To give you some context, even two beefy machines with an older message queue solution like ZeroMQ can tolerate throughput in excess of what most companies produce. This is not to discredit the authors of Pulsar or Kafka at all... but it's just a concerning trend where easy to use horizontally scalable message queues are being deployed everywhere. Similar to how everyone was running hadoop a few years back even when the data fit in memory.

gfodor 7y ago
Worth noting that Kafka is not a queue, but an append-only log.
tylertreat 7y ago
ZeroMQ is not a message queue, it's a networking library.
- injinj 7y ago
  I'll bet he is aware. The problem with the trading industry is that they have hundreds of users with bespoke solutions catering to extreme performance criteria rather than hundreds of thousands like NATS. They will keep reinventing the wheel every time a nanosecond can be saved by the latest hardware stack.
platform 7y ago
Aeron[1] is something I have been look at as well. I like that it is adaptable to run without an external broker (using embedded media driver option), yet can add that for scalability/redundancy. Our use cases do not require a query on top of streams. Instead all data would go into timescaleDB [2]. [1] https://github.com/real-logic/aeron/wiki/Java-Programming-Guide https://github.com/real-logic/aeron/wiki/Java-Programming-Gu... [2] https://github.com/timescale/timescaledb https://github.com/timescale/timescaledb

firstposterone 7y ago

Splunk just acquired streamlio and most of the core devs got sucked up. While pulsar is a great product - are you not concerned that these guys are getting paid $$ bank to do something else now?

matteomerli 7y ago
spoiler: we're still working on Pulsar
- firstposterone 7y ago
  That’s good news! I guess it would be very helpful to formally address this concern. Is there something that has been written / published to that effect?
zhaijia 7y ago
There is another company StreamNative, founded by core Pulsar/BookKeeper devs 1 year ago. :)

microcolonel 7y ago

I'm sure Pulsar is worth it if you use most of what they're offering, but the Java client library is crusty, throws exceptions for control flow. I'm looking at a persistence mechanism built on top of NATS to replace it. The NATS layer would make it simpler to decouple the gateways from the persistence layer, and support our bulk computing needs.

dikei 7y ago

From reading their documents, I really like the design of Pulsar. However, Kafka has been working so well for us and has much better integration with other components of our stack (Flink, Spark, NiFi, etc) that there's no compelling reason to switch. I think Pulse should really focus on the integration with the rest of the Apache stack if they want to gain traction.

adkafka 7y ago

I really enjoyed the talk given by Quentin Adam & Steven Le Roux at Devoxx 2019. It gave a great overview of Apache Pulsar. Hope someone else can find it useful too! [0] https://www.youtube.com/watch?v=De6avNyQUMw https://www.youtube.com/watch?v=De6avNyQUMw

jsilence 7y ago

Would Pulsar be suitable for IoT messaging? An alternative to mqtt?

manigandham 7y ago

This is one of the best overviews on Pulsar with comparisons to Kafka: https://jack-vanlightly.com/blog/2018/10/2/understanding-how-apache-pulsar-works https://jack-vanlightly.com/blog/2018/10/2/understanding-how...