6 ms·

> two of the most favored messaging systems on the market Give me a break, I'd literally never even heard of Pulsar until this article popped up. Of all messa

> two of the most favored messaging systems on the market Give me a break, I'd literally never even heard of Pulsar until this article popped up. Of all messaging systems I would have thought Kafka vs SQS, or even RabbitMQ at the very least

bsaul 6y ago
For people currently interested in building an event based architecture, pulsar is definitely a very well known option.
- ramraj07 6y ago
  I am squarely in the "people interested in building an event based architecture". Not a CS background, but know tech decently. I typically know the names of more of these apache projects than most people I've talked to IRL (though ostensibly I'm not part of the tech elite). Yet pulsar I only came across on HN a week back. And nothing changes due to the knowledge. It's yet another apache product which has great design but needs either a genius or an army of devops to deploy. Like even if I figure out what the hell this thing is, I'm then tasked with figuring out what the hell zookeeper is (like for real, I'll buy a beer for someone who can successfully ELI5 wth zookeeper is. And also pig. Or impala).
  - whorleater 6y ago
    pulsar was seeing quite a bit of hype back in ApacheCon 2019 already
  - maps7 6y ago
    > Yet pulsar I only came across on HN a week back. Same - was it the comment saying nobody chooses Kafka anymore? I was surprised by it.
    - polote 6y ago
      If you search you will find that there are always the same 4-5 HN users who comment on pulsar/kafka topics arguing that Kafka is not a viable solution anymore. Even though almost everyone is still using Kafka
      - z9e 6y ago
        Meanwhile the Confluent Kafka conference seems to double in size year after year. And AWS chooses to offer Kafka as a service, rather than Pulsar. These "<Solution> is dead / not viable anymore!" comments are pretty rampant in the Big Data space. Some are true though but it still makes me roll my eyes. These people seem to be invested (quite literally) in Pulsar disrupting Kafka's position in the pub/sub & event streaming space. This isn't to say Pulsar can't someday be superior, though. Time will tell. But Kafka has such a massive lead and it fits the needs for most people who need to do data integration that I don't see the need for everyone to uproot their solutions in favor of Pulsar.
      - Jedd 6y ago
        Almost everyone still uses Microsoft Windows. But almost everyone has heard of GNU/Linux or Apple OSX.
  - bmcfeeley 6y ago
    I'm no expert, but my understanding is that pig is a combination of - a language for specifying data transformations, and - an engine to compile programs written in that language into mapreduce jobs to execute on a hadoop cluster it was designed to easily map some common functional and SQL idioms (e.g. filter, group by w/ aggregation functions) to parallel execution for processing huge amounts of data. Impala is another big data project that is an engine for planning and executing SQL on data stored in a hadoop cluster. Zookeeper is... black magic??
  - sciurus 6y ago
    If you can't understand what Zookeeper is, I'd recommend reading Martin Kleppmann's book Designing Data-Intensive Applications (https://dataintensive.net/ https://dataintensive.net/). You don't need a CS degree to work in this field (I don't have one either!) but there are fundamental concepts you need to understand in order to make informed decisions when designing distributed systems.
    - toomanybits 6y ago
      You don't need a CS degree or Martin Kleppmann's book to work out it's a GPITA.
      - papaf 6y ago
        I am also not a fan of Zookeeper but I have come to respect it. In defense of Zookeeper, distributed systems are a GPITA. If you need to select some components why not go for ones that are solid [1]. [1] https://aphyr.com/posts/291-jepsen-zookeeper https://aphyr.com/posts/291-jepsen-zookeeper
  - prostodata 6y ago
    Apache Pulsar should not be confused with another quite popular Pulsar: Event driven concurrent framework for Python https://github.com/quantmind/pulsar https://github.com/quantmind/pulsar
edem 6y ago
My thoughts exactly.
tomkwong 6y ago
I love SQS but it isn't quite comparable to Kafka because it is poll-based. It's a solid queue but not suitable for streaming.

bsaul 6y ago

For people currently interested in building an event based architecture, pulsar is definitely a very well known option.

ramraj07 6y ago
I am squarely in the "people interested in building an event based architecture". Not a CS background, but know tech decently. I typically know the names of more of these apache projects than most people I've talked to IRL (though ostensibly I'm not part of the tech elite). Yet pulsar I only came across on HN a week back. And nothing changes due to the knowledge. It's yet another apache product which has great design but needs either a genius or an army of devops to deploy. Like even if I figure out what the hell this thing is, I'm then tasked with figuring out what the hell zookeeper is (like for real, I'll buy a beer for someone who can successfully ELI5 wth zookeeper is. And also pig. Or impala).
- whorleater 6y ago
  pulsar was seeing quite a bit of hype back in ApacheCon 2019 already
- maps7 6y ago
  > Yet pulsar I only came across on HN a week back. Same - was it the comment saying nobody chooses Kafka anymore? I was surprised by it.
  - polote 6y ago
    If you search you will find that there are always the same 4-5 HN users who comment on pulsar/kafka topics arguing that Kafka is not a viable solution anymore. Even though almost everyone is still using Kafka
    - z9e 6y ago
      Meanwhile the Confluent Kafka conference seems to double in size year after year. And AWS chooses to offer Kafka as a service, rather than Pulsar. These "<Solution> is dead / not viable anymore!" comments are pretty rampant in the Big Data space. Some are true though but it still makes me roll my eyes. These people seem to be invested (quite literally) in Pulsar disrupting Kafka's position in the pub/sub & event streaming space. This isn't to say Pulsar can't someday be superior, though. Time will tell. But Kafka has such a massive lead and it fits the needs for most people who need to do data integration that I don't see the need for everyone to uproot their solutions in favor of Pulsar.
    - Jedd 6y ago
      Almost everyone still uses Microsoft Windows. But almost everyone has heard of GNU/Linux or Apple OSX.
- bmcfeeley 6y ago
  I'm no expert, but my understanding is that pig is a combination of - a language for specifying data transformations, and - an engine to compile programs written in that language into mapreduce jobs to execute on a hadoop cluster it was designed to easily map some common functional and SQL idioms (e.g. filter, group by w/ aggregation functions) to parallel execution for processing huge amounts of data. Impala is another big data project that is an engine for planning and executing SQL on data stored in a hadoop cluster. Zookeeper is... black magic??
- sciurus 6y ago
  If you can't understand what Zookeeper is, I'd recommend reading Martin Kleppmann's book Designing Data-Intensive Applications (https://dataintensive.net/ https://dataintensive.net/). You don't need a CS degree to work in this field (I don't have one either!) but there are fundamental concepts you need to understand in order to make informed decisions when designing distributed systems.
  - toomanybits 6y ago
    You don't need a CS degree or Martin Kleppmann's book to work out it's a GPITA.
    - papaf 6y ago
      I am also not a fan of Zookeeper but I have come to respect it. In defense of Zookeeper, distributed systems are a GPITA. If you need to select some components why not go for ones that are solid [1]. [1] https://aphyr.com/posts/291-jepsen-zookeeper https://aphyr.com/posts/291-jepsen-zookeeper
- prostodata 6y ago
  Apache Pulsar should not be confused with another quite popular Pulsar: Event driven concurrent framework for Python https://github.com/quantmind/pulsar https://github.com/quantmind/pulsar

edem 6y ago

My thoughts exactly.

tomkwong 6y ago

I love SQS but it isn't quite comparable to Kafka because it is poll-based. It's a solid queue but not suitable for streaming.