11 ms·

RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

292 points by gslin 3y ago

deleted 3y ago
[deleted]
supermatt 3y ago
The same MQ patterns as mentioned in the article (exactly once, consumer groups) can also be done in kafka, contrary to what the article suggests.
robertlagrant 3y ago
> one is a message broker, and the other is a distributed streaming platform I think this is an odd way of putting it. One is smart messaging; dumb clients. The other is dumb messaging; smart clients. It turns out the latter (i.e. Kafka) scales wonderfully so you can send more data, but you add complexity to your clients, who can't just now pluck messages off a queue to process, or have messages retry upon the first 3 failures, as they could with RabbitMQ. Having said that, Kafka lets you keep all your data, so you don't have to worry about losing messages to unexpected interactions between RabbitMQ rules. But having said that, now you have to store all your data.
- debadyutirc 3y ago
  Every decision has a consequence. There are a lot more options depending on the use case.
- supermatt 3y ago
  > One is smart messaging; dumb clients. The other is dumb messaging; smart clients. All the smartness of the messaging can be implemented in the smart clients. Then you can expose that as a smart messaging api to dumb clients. The most obvious example is kafka streams which exposes a "simple" api rather than dealing directly with kafka, but obviously you could create a less featurefull wrapper than that.
  - debadyutirc 3y ago
    That's a neat way to put it! - RabbitMQ: Smart messaging, dumb client - Kafka: Dumb messaging, smart client Have you heard of Fluvio? Fluvio: Smart messaging, Smart Client, Stateful Streaming Kafka + Flink in Rust + WASM Git Repo - https://github.com/infinyon/fluvio https://github.com/infinyon/fluvio
  - waynesonfire 3y ago
    And reimplement rabbitmq? Great idea. Let's do it in rust too.
    - deleted 3y ago
      [deleted]
  - eddythompson80 3y ago
    I can't help but think that this just gives you the worst of both worlds. You are now on the hook managing that non-standard "smart" wrapper which will quickly just become the status quo for the project. Anyone wanting to change how it works needs to understand exactly how "smart" you made it and all the side effects that will come with making a change there. I pushed against knative in our company particularly for that reason. Like we wanna use kafka because [Insert kafka sales pitch], but we don't want our developers to utilize any of the kafka features. We're just gonna define the kafka client in some yaml format and have our clients handle an http request per message. It didn't make sense to me.
    - robertlagrant 3y ago
      This would be my instinct too.
    - supermatt 3y ago
      Thats kind of like saying dont use any software libraries because they all use the standard lib indirectly so you may as well just use that? Its just an abstraction layer to make things less effort.
- raducu 3y ago
  > who can't just now pluck messages off a queue to process The problem is you cannot mark individual messages as read, for a given consumer&partition you can only update the offset for a partition. If a certain message processing takes very long, all other messages in that partition will have to wait. Also, with kafka, the max read concurrency is equal to the number of partitions, for something like rabbitMq it is much higher; but you do get nice message ordering for any given partition in kafka which you do not get in RabbitMq (afik); you are also get some really nice data locality with kafka because unless the consumers get the partitions re-assigned, all messages for the same key are served on the same physical consumer.
  - math 3y ago
    Worth noting that Kafka is getting queues: https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A+Queues+for+Kafka https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A...
    - facorreia 3y ago
      And also Rabbit has streams[1]. There's a lot of overlap. [1] https://www.rabbitmq.com/streams.html https://www.rabbitmq.com/streams.html
  - ryanjshaw 3y ago
    > The problem is you cannot mark individual messages as read, for a given consumer&partition you can only update the offset for a partition. Hence "smart clients". If you MUST process every message at least once, you will anyway be tracking messages individually on the client (e.g. a DB or file system plus logic for idempotent message processing) and thus disable auto-offset commits back to the cluster for your consumer. RabbitMQ says "let me track this for you", Kafka says "you already need to track this so why duplicate the data in the cluster and complicate the protocol". If you don't have reliable persistent storage available and insist on using the Kafka cluster to track offsets, you can track processed offsets in memory and whenever your lowest processed offset moves forward, you have your consumer commit that offset manually as part of its message loop. If your service restarts your downstream commands need to be idempotent of course because you will reconsume messages you may have previously processed, but this would be the case regardless of Kafka or RabbitMQ unless you're using distributed transactions (yuck). > If a certain message processing takes very long, all other messages in that partition will have to wait. You can stream messages into a buffer and process them in parallel, and commit the low watermark offset whenever it changes, as described above. I've implemented this in .NET with Channels and saturate the CPUs with no problem.
- pnpnp 3y ago
  Just my 2c but for anyone unaware, you should check out NATS. It combines the best of both Kafka and RabbitMQ IMO.
  - doctor_eval 3y ago
    I thought NATS didn't actually store messages, am I mistaken? Looking at Wikipedia (https://en.wikipedia.org/wiki/NATS_Messaging https://en.wikipedia.org/wiki/NATS_Messaging) I see that I'm technically right, it's JetStream that does the storage layer - but it's part of the NATS server. From memory, I really liked the philosophy of NATS but found the nomenclature confusing.
    - pnpnp 3y ago
      I think they call that part of NATS “Jetstream” if I’m not mistaken. I haven’t used it, but I believe it has some form of message persistence. I have used it mostly for message-first services, and found subject-based messaging a breath of fresh air to decouple services. You can do the same thing with RabbitMQ topic exchanges, but it requires quite a bit more hand-waving.
      - zerbinxx 3y ago
        Jetstream does indeed have message persistence: I can issue queries like “get messages on topic since 5 minutes ago” - I do this a ton. However, that seems to be the extent of the storage/query API that it exposes for historical messages. I’m quite a big fan, and would recommend it with the caveat that Jetstream is considerably more complex than simple nats and I get the feeling I’m barely scratching the surface with it.
        doctor_eval 3y ago
        I’m really interested in a Kafka like message broker in the Go ecosystem and look forward to checking it out for whatever my next project ends up being.
- cbsmith 3y ago
  You're getting to the key thing: You don't want to classify them by what they do. You want to classify them by what the clients must do/experience.
- continuitylimit 3y ago
  It is not odd, it is basically accurate. You are making a fetish of the S-C interaction but the essential matter is that Kafka is designed to store & distribute logs, whereas Rabbit is designed to route & send messages. The ‘store’ bit is very much a part of Kafka’s mission statement but not for Rabbit.
BWStearns 3y ago
My comment is mostly about part 2 of this post, but wrt message ordering being a kafka "win" I'd raise the point that in the actual use case of "a consumer fails in some way to process the message" you can still end up with out of order processing of the consumer's input since you might want to dump them into a DLQ or something. The fact that the message isn't reappended to the topic by default for processing is kind of an academic point no? Unrelatedly, I've been looking at Pulsar lately. Anyone have experience with Pulsar and either RMQ/Kafka want to throw out some opinions from having tried both?
- EdwardDiego 3y ago
  Pulsar can have both MQ semantics and pub/sub semantics. In pub/sub it's sorta like "Kafka with all bits people found it necessary to build later already built in", e.g., a proxy, schema registry, connectors, replication, tiered storage, all out of the box. It also has lightweight streaming functions built-in, but they operate per record, so good for lightweight transforms/routing, not for stream aggregations etc. It has more moving parts also, brokers are decoupled from storage, which is handled by BookKeeper, and replication between two clusters requires a ZooKeeper that both clusters can access, in addition to the ZK used by the brokers and bookies. And it's a reasonably new project, so last time I looked into it, some of the documentation was incorrect, especially around managing bookies.
  - relay23 3y ago
    Massive and complex platform.. at a certain point why not just run 2 different platforms that are best of breed for each.
    - olavgg 3y ago
      You can just run Pulsar standalone which is super-simple, it is a great start and scales just fine for one machine. Once you outscale one machine, you can refactor Pulsar into a distributed setup. The advantage is that everything is still familiar.
bhouston 3y ago
If you use Confluent Kafka, the billing is pretty high. About 4 years ago it was much cheaper, but then they completely revamped the pricing to something ridiculous. I found that switching to Google Pub/Sub, at least if it meets your needs, is cheaper.
- AndyPa32 3y ago
  Yes, I can confirm that. Confluent is the most expensive part of our current infrastructure.
  - bhouston 3y ago
    Can you switch away from it? Or do you need its advanced features?
  - relay23 3y ago
    Sounds about right.
- esafak 3y ago
  I see they offer Kafka's exactly-once delivery: https://cloud.google.com/blog/products/data-analytics/cloud-pub-sub-exactly-once-delivery-feature-is-now-ga https://cloud.google.com/blog/products/data-analytics/cloud-...
  - relay23 3y ago
    Kafka doesn't guarantee exactly once delivery at all, unless you're using Kafka Streams and even then your final output topic still won't get exactly once, the consumer group protocol doesn't allow for it.
- relay23 3y ago
  It's cheaper until you get to a sizeable workload, and the P90+ latency is ridiculous.. the kafka api is weak and when you're not using kafka api you're limited on integration tools unless you want to be super locked in to GCP.
dividedbyzero 3y ago
NATS (https://nats.io/ https://nats.io/) is another option, though I'm not sure if it's still considered a viable Kafka replacement.
- vorpalhex 3y ago
  Different set of promises. NATS is great but has a different tradeoff bargain from Rabbit or Kafka.
  - kumarvvr 3y ago
    Could you expand on this a bit more? I am curious.
    - vorpalhex 3y ago
      NATs has a decent-ish guide here: https://docs.nats.io/nats-concepts/overview/compare-nats https://docs.nats.io/nats-concepts/overview/compare-nats A few things they get wrong mostly about Rabbit: + RabbitMQ does support replay, and also has a memory only mode which will support persistance in a cluster + RabbitMQ doesn't have that sensitive of a latency between cluster members (no more sensitive than NATS in some setups). + RabbitMQ also supports Prometheus A good (but incomplete) rule of thumb: + Kafka is a distributed Append-Only-Log that looks like a message bus. It allows time travel but has very simple server semantics. + RabbitMQ is a true message broker with all the abilities and patterns therein. It comes with more complexity. + NATs is primarily a straight forward streaming messaging platform. Also consider Redis' message queue mode, zeromq, mqtt brokers (Eclipse Mosquitto) and the option of just not using a message broker/queue system. Even as someone who really likes the pubsub pattern, there's a good chance you don't need it and you may be heading to a distributed monolith antipattern.
      - gunapologist99 3y ago
        > distributed monolith antipattern Beauty is in the eye of the beholder, I guess. (I would definitely not consider a distributed monolith an antipattern) Also wanted to add NSQ to your very good list.
bicijay 3y ago
Message ordering is an illusion. Unless you track/store the messages on the client and are willing to deal with stuck queues due to failures in one "poisoned" message.
- klabb3 3y ago
  There are different kinds of order. Yes, there’s no total order in a distributed system, but you can have certain partial order guarantees. It’s nice if something is added before it’s updated, for instance.
  - bicijay 3y ago
    Could you expand on that? How would you achieve "partial order" guarantees?
    - klabb3 3y ago
      One type of partial order would be that a producer puts all the messages that are related in the same queue, so that A always precedes B. Basically the invariant becomes: If a consumer sees an event B, it will have certainly have seen the event A before that. Assuming business logic is correctly written, that saves you from having to write certain retry logic on the B handlers. This requires the message queue to be always available. If it goes down, the system would not make progress (like a db - in fact the MQ is a db). Once you add more actors/nodes to the same related events, maintaining a “causal order” can be very tricky and subtle, especially if you have an MQ and a DB as multiple sources of truth. So I’m not exactly endorsing it, even though MQ-as-a-DB (aka event sourcing) is a very interesting idea.
    - CogitoCogito 3y ago
      Let's say you have events coming in that result in inserts, updates, and deletes on a table with a certain primary key. Assuming no dependencies external to this table, you only need events involving a specific key to be ordered. I.e. it doesn't really matter if row_a gets updated before or after row_b. In either case, you end up with the same thing. So if you do something like kafka partitions and you send events to certain partitions based on their primary key, then those partitions will be ordered which will be enough. That doesn't fix your example of dealing with individual errors, but in many cases that's enough.
tannhaeuser 3y ago
I think this is an excellent article. The only thing I'd add is that RabbitMQ is an implementation of AQMP (optionally v1.0) as a standardized broker service protocol so is designed to be interchangeable with other extant implementations such as Apache Active MQ and Cupid whereas Kafka is one-of-a-kind software. Beyond that RabbitMQ has standardized client libs and frameworks in Java land if that matters to you - it did matter in the original context of message queue middlewares and SOA from where AMQP originated and where enterprise messaging sees major use. OTOH Kafka, with caveats, is in principle more "web scale" - though that is far from a free ride.
whalesalad 3y ago
One is a tomato, the other is an orange. From a distance they might look alike but they really are two completely different tools. This is a pretty solid explanation of the differences with good illustrations. Rabbit can do everything Kafka does - and much more - in a more configurable manner. Kafka is highly optimized for essentially one use case and does that well. Nothing in life is free, there are trade-offs everywhere. I am not privy to which one is theoretically faster - but once you reach that question methinks the particular workload is the deciding factor.
- JohnMakin 3y ago
  I am not an expert in either and have only worked with Kafka. At a past job I had to write a connector job to parse and sanitize some extremely dirty, unstructured data and pass it along somewhere else. RabbitMQ supports this? What is the one use case of kafka? I think you have it backwards.
  - whalesalad 3y ago
    > parse and sanitize some extremely dirty, unstructured data and pass it along somewhere else can you be more specific? that to me sounds like hello world for either of these tools. "santize data" is an application level concern that neither rabbit or kafka would handle. as far as "pass along somewhere else" again both tools can do.
    - JohnMakin 3y ago
      It was a Sink Connector. I don’t know what it was or wasn’t supposed to do but I was asked to do it, as is often the case in tech. I could have done any number of transformations in that process though, which I’m not sure rabbitmq supports
      - whalesalad 3y ago
        It sounds to me like you aren't really even sure what you built. I have operated both rabbit and kafka at scale I definitely do not have it backwards :)
FridgeSeal 3y ago
I’m personally a fan of Kafka. I think the design of persisting the messages, and tracking offsets for progress instead of message acknowledgments is a much cleaner and more versatile design. You can get all the same advantages of message acknowledgments, but now you can also replay queues, let different applications use the messages (handy for cross cutting event/notification systems) and you get better scaling properties-which doesn’t hurt at the small scale, and provides further scaling when you need it.
- whalesalad 3y ago
  > You can get all the same advantages of message acknowledgments, but now you can also replay queues with rmq you can reject/nack a message and have it put back on the queue. rmq is not well suited for long term historical retention inside queues a-la kafka's logs but it is possible to do. > let different applications use the messages (handy for cross cutting event/notification systems) rmq also does a publish once and fanout to multiple queues to support this. data is replicated so that could be a deal breaker, but it is possible. how often have you had to diagnose a stuck consumer or some other kind of offset glitch where a consumer is unable to resume where it left off? not knocking kafka here but I do think it is a tool you should reach for when you need to solve a very hyper focused problem, while rabbit is a tool more suited to most cases where queuing is required. kafka is a code smell in a lot of organizations from my experience - most do not need it.
  - raducu 3y ago
    > afka is a code smell in a lot of organizations from my experience - most do not need it. Kafka is really nice if you don't care that much about latency during peak load and you don't have absurd processing times for messages.
    - ceencee 3y ago
      Kafka can sustain sub 20ms at millions or even billions per second scale. Processing time delays is bad consumer code and partition design smell. Aka , your consumer shouldnt depend on a slower resource within an ordering domain. This can also be mitigated with an async consumer
purpleblue 3y ago
If someone is asking if they should decide between RabbitMQ vs Kafka, they should 100% use RabbitMQ. It means they have no idea what they're dealing with, the architectural differences, and the investment that the company needs in order to use Kafka. So use RabbitMQ.
- patrec 3y ago
  How do you create anything with RabbitMQ that a) has performance characteristics under load you can reason about and b) can handle individual node or networking failures without data loss? Kafka is overkill in most scenarios and you should probably just see if postgres isn't enough for your needs first (especially since you will almost certainly already need a database anyway). Kafka is more pain to setup and run than it ought to be. But underlying it is a useful and sensible abstraction for building robust distributed systems.
postalrat 3y ago
Redis isn't an option?
- rafaelturk 3y ago
  Yes, Don't why is not mentioned in the article, maybe because is more barebones.
- noitpmeder 3y ago
  I'd also like someone with experience to contrast redis for these use cases.
- blindriver 3y ago
  Redis is a cache, it's not a queue. Can you do something similar? I guess, just like you can use a highlighter to paint your house. But the semantics are not correct.
  - relay23 3y ago
    There is Redis Streams, but certainly not without it's problems. Super obscure, not a lot of client support.
- akvadrako 3y ago
  Or postgres. If you are under 1000 messages a second it functions well as a transactional queue.
  - sgarland 3y ago
    Until table bloat happens.
rafaelturk 3y ago
Nice post! RabbitMq is battle tested, exceptionally fast and low resources app. Capable of handling millions of transactions/second. RabbitMQ will handle vast majority of usecases. I'm puzzled why often startups, or even banks use Kafka, soley because is hype. Kafka on the order hand requires massive CPUs, Memory, often requiring its own K8S cluster just to be alive.
- rafaelturk 3y ago
  If your have a clean event-driven architecture, ie messages are completely agnostic and decoupled from one-another you don't need Kafka.
  - ceencee 3y ago
    This is a ridiculous statement if you really build an EDA. Kafka is what enables the decoupling.
  - inkyoto 3y ago
    Event-driven architecture is an architectural principle, and Kafka, RabbitMQ/ActiveMQ, Pulsar, NATS and so forth are implementations that support the event-driven architectural principle. Yet, all of them range in a variety, complexity and extent of features they provide which may or may not be a good fit for a particular use case. Traditional message brokers (RabbitMQ and similar) do support the event-driven architecture, yet the data they handle is ephemeral. Once a message has been processed, it is gone forever. Connecting a new raw data source is not an established practice and requires a technical «adapter» of sorts to be built. High concurrency levels is problematic for scenarios where the strict message processing ordering is required: the traditional message brokers do not handle it well in highly parallel scenarios out of the box. Kafka and similar also support event-driven architectures, yet they allow the data to be processed multiple times – by existing (i.e. a data replay) and, most importantly, new or unknown at the time consumers (note: this is distinct from the data replay!). This is allows to plug existing data source(s) into a data streaming platform (Kafka) and incrementally add new data consumers and processors over the time with the datasets being available intact. This is an important distinction. Kafka and similar also improve on the strict processing order guarantee by allowing a message source (a Kafka topic) to be explicitly partitioned out and guaranteeing that the message order will be retained and enforced for a consumer group receiving messages from that partition. To recap, traditional message brokers are a good fit for handling the ephemeral data, and data streaming platforms are a good fit for connecting data sources and allowing the data to be ingested multiple times. Both implement and support event-driven architectures in a variety of scenarios.
dventimi 3y ago
> Apache Kafka is not an implementation of a message broker. Instead, it is a distributed streaming platform. Unlike RabbitMQ, which is based on queues and exchanges, Kafka’s storage layer is implemented using a partitioned transaction log. Kafka also... This seems like an important passage, drawing the crucial and long-awaited distinction between RabbitMQ and Kafka, and yet without having defined a "partitioned transaction log" the author strands the reader without any help in absorbing the distinction.
- zerbinxx 3y ago
  This is sadly commonplace in tech blogs: rather than taking a great opportunity to hook the reader, the writer will drop a vocab term in bold and move on.
- estiller 3y ago
  Hi, this is the article author here. Thanks for the feedback! I've written this article 3.5 years ago, and it could definitely deserve a shake-up. I agree this should be cleared up a bit.
bob1029 3y ago
I've seen Tibco Rendezvous used in manufacturing. ~300 megabytes per hour of raw log generated 24/7/365 by tools and control systems in a factory setting. Probably on the order of 10k+ participants in the pub/sub network. If you are running something like a factory where thousands of independent systems need to communicate in some way, this kind of tech starts to look like the only option. If you are orchestrating the concerns of 5-10 services, I think you are making your life harder than it needs to be with ESB-style abstractions. Direct method invocation is much more reliable than whatever any one of these vendors could ever sell you. Put all your services into one exe. If you can't be bothered to use one language/repo, there are still ways to achieve this. The real architect's dilemma is shoving one's edifice-constructing ego into a box long enough to produce a useful shack for the business.
- strangemonad 3y ago
  Or NATS
  - relay23 3y ago
    or kafka, or any other publish/subscribe system
- kuchenbecker 3y ago
  Our prod cluster generates that about every minute at O(1M) qps. We JUST turned on remote Logs because until now Kafka didn't have capacity.
  - inkyoto 3y ago
    TIBCO Rendezvous is tech from 1998/99, millions of messages per second didn't exist at the time. Only NYSE and NASDAQ were capable of producing millions of events back then (still not by minute nor by second). TIBCO Rendezvous was one of the first successful large scale, low latency and near real-time pub/sub implementations, and it had a very efficient, Avro like, wire level serialisation format that made messages very compact and efficient to deliver. It was very popular in finance, banking and manufacturing, and is all but legacy now.
cma256 3y ago
The great thing about Kafka is the ability to batch operations. Collect a set of messages in memory and, when its time to commit, submit a bulk operation. If something fails you just rebuild the buffer from the last committed offset. Pretty neat piece of technology.
jpgvm 3y ago
If you want features of RabbitMQ (specifically queue like behavior) but the scalability of Kafka then you probably want Apache Pulsar. To elaborate on that a bit the main things Pulsar gives you are: 1. Still underlying distributed stream based architecture, this is what makes it able to do Kafka like things. 2. Broker side management of subscription state which allow out of order acknowledgement, this means you can use it like a queue. (Subscriptions sort of act like AMQP mailboxes but without the exchange routing semantics). Vs Kafka which can only do cumulative acknowledgement, i.e head of line blocking. 3. Separated "compute" and storage. By storing data in Bookkeeper you can scale your needs to support a lot of consumers separately from how you stash the data those consumers need to read vs Kafka where these 2 are coupled and an imbalance between the two becomes awkward. 4. In built offload with transparent pass-through read. When your data falls off the retention cliff for your standard broker cluster the data can be archived to object storage. The broker can transparently handle read request for these earlier messages though, just with higher startup latency to pull the archived ledgers. 5. Way more plugability than Kafka, in-fact similar plugability as RabbitMQ. You can implement your own authz/authn, a different listener to support a different protocol (there is a Kafka one, MQTT, AMQP etc). 6. Much greater metadata scalability. Before the new KRaft implementation the layout of metadata in ZK meant that you couldn't feasibly have more than about 10k topics. Especially because of how long the downtime would be on controller failover. Pulsar can easily support much larger numbers of topics which prevents needing to use a firehose design when you would prefer individual topics per tenant/customer/whatever.
- mianos 3y ago
  While pulsar on paper seems a superior solution, in my experience it is very still very immature and very buggy. I really want to use it over kafka but I would not bet my business on it. I am not a fan of Kafka, it's kinda old, and the code is a bit messy, a lot of the once only semantic problems 100% solved by Pulser are sorta kinda in Kfaka these days. All the newer stuff like built in RAFT makes it competitive with anything. But, anyone who has used Kafka at scale has to say it 100% does what it says on the box. Many people are used to it's idiosyncrasies at scale and can get it to scale. Now RabbitMQ. I have been burnt so badly by it breaking at scale I'll never be touching it again. Maybe my fault, maybe not, but replacing it with Kafka solved all my flakey issues and I never looked at it again.
deleted 3y ago
[deleted]
jonstewart 3y ago
I'm sure there remain good use cases for message buses that you have to run yourself, where there really are millions of messages that can't be batched up and need real-time and whatnot. But you can get pretty far with: 1. Write a bunch of records to an S3 object. 2. Trigger a lambda to process — infinite scale out! and if a queue really is needed due to constrained consumers, then: 1. Write a bunch of records to an S3 object. 2. The lambda trigger puts a message into an SQS queue with the S3 object's URI. 3. An auto-scaling group gets the message off the queue and processes the data in the object. If ordering's important, then the SQS queue can be made FIFO. That has pretty low numbers in messages/second, but since records are being batched into S3 objects you can still have fairly high throughput. It used to be that the elegance of queues for such systems would tempt developers into the operational slough of despond that is running these types of systems. Again, I'm sure it's warranted for some applications, but S3, Lambda, and SQS as above work nicely together.
- pramodbiligiri 3y ago
  You can get rid of Step 2 if you're going to be creating a new S3 object for each batch. AWS has an inbuilt feature to trigger SQS notifications for S3 operations: https://docs.aws.amazon.com/AmazonS3/latest/userguide/enable-event-notifications.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/enable...
deleted 3y ago
[deleted]
pharmakom 3y ago
seems like there is a missing option? - Message retention (like Kafka) - Easy consumer scale out (like RabbitMQ) - No particular ordering guarantees (like RabbitMQ)
pbowyer 3y ago
What these articles always miss is what are the real-world scenarios that use Kafka or RabbitMQ? I've never used one professionally and beyond the async processing cases I come across in web dev (mostly sending emails asynchronously), or seeing devs using one to handle 5 req/s which doesn't need it, I don't have a good feel for when they are really needed - especially for companies at sub-Uber scale. What real-world cases would you use them for?
estiller 3y ago
Hi everyone, this is the article author here. I was genuinely surprised to see the article pop up here 3.5 years after I wrote it. Thank you for all the feedback! Given the advances in the past few years in this area, this article could use a serious update. A comparison to NATS and Pulsar is warranted, along with some extra explanations for some of the technical terms in the article. I'll carefully review the feedback here and try to make an update at some point soon.
opentokix 3y ago
Wow, this is the type of trash one can expect from someone calling themselves "Architect"
- rrr83 3y ago
  Be kind! Author has shared his findings based on his understanding, if you have more to add go ahead and create another page?
h1fra 3y ago
RabbitMQ is one the most impressive technology I have ever used I think. It's simple, efficient and reliable. In 10 years, I have never seen a crash or a bug, even with heavy load.
- vorticalbox 3y ago
  where i work we have some queues with millions of items in and the thing is perfectly stable.
onetimeuse92304 3y ago
If you have a throughput problem you are most likely doing it wrong. If your knee jerk reaction is to scale your messaging up you may want to reconsider. Messaging systems are usually hard to scale up and always very costly to do so compared to the amount of data they are transferring. The simplest thing you can do is to realise WHY you are using messages. Messages are there to trigger a process. Usually, you don't need a lot of data to trigger a process, the message just needs to let the system know enough to locate all necessary information. Also, when you are sending information at an extremely high rate, there usually is no difference if each message is processed separately or in batches. So what you can do in practice? 1) Get the producer to batch the messages. For example set rules like "batch up to 10.000 messages, up to 100ms, up to 100MB of data, whichever comes first". 2) Serialise the batch (for example, if it makes sense, create a compressed JSON file) 3) Upload the file to some high throughput, scalable, cheap storage (for example S3) 4) Send a message to the queue / topic / whatever else you are using with just enough to locate and process the message -- usually just the link to the S3 object. This usually can be modified depending on specific project needs. Now your messaging only ever sees a small number of very small messages and you will never have any scaling problems, at least not on messaging side.
- usrusr 3y ago
  If your use case can stomach the added failure modes this implies, yeah, that's what you can do.
  - onetimeuse92304 3y ago
    Everything has some cons, some failure modes. All engineering is about knowing, understanding and making tradeoffs. In my practice, I am happy if I can get rid of hard problems (my messaging platform being unable to process X messages per second) and replace them with relatively easier problems (my persistence might sometimes fail and then I can't send a message). I would argue that distributed persistence solutions are usually more reliable than messaging platforms and also what is a very large throughput for a messsaging solution is usually nothing much for monsters that are engineered to take much larger volumes of data. And so, in my experience, reducing load to messaging and increasing load to persistence is net positive for the overall reliability.

292 points by gslin 3y ago

deleted 3y ago

[deleted]

The same MQ patterns as mentioned in the article (exactly once, consumer groups) can also be done in kafka, contrary to what the article suggests.

robertlagrant 3y ago

> one is a message broker, and the other is a distributed streaming platform I think this is an odd way of putting it. One is smart messaging; dumb clients. The other is dumb messaging; smart clients. It turns out the latter (i.e. Kafka) scales wonderfully so you can send more data, but you add complexity to your clients, who can't just now pluck messages off a queue to process, or have messages retry upon the first 3 failures, as they could with RabbitMQ. Having said that, Kafka lets you keep all your data, so you don't have to worry about losing messages to unexpected interactions between RabbitMQ rules. But having said that, now you have to store all your data.

debadyutirc 3y ago
Every decision has a consequence. There are a lot more options depending on the use case.
supermatt 3y ago
> One is smart messaging; dumb clients. The other is dumb messaging; smart clients. All the smartness of the messaging can be implemented in the smart clients. Then you can expose that as a smart messaging api to dumb clients. The most obvious example is kafka streams which exposes a "simple" api rather than dealing directly with kafka, but obviously you could create a less featurefull wrapper than that.
- debadyutirc 3y ago
  That's a neat way to put it! - RabbitMQ: Smart messaging, dumb client - Kafka: Dumb messaging, smart client Have you heard of Fluvio? Fluvio: Smart messaging, Smart Client, Stateful Streaming Kafka + Flink in Rust + WASM Git Repo - https://github.com/infinyon/fluvio https://github.com/infinyon/fluvio
- waynesonfire 3y ago
  And reimplement rabbitmq? Great idea. Let's do it in rust too.
  - deleted 3y ago
    [deleted]
- eddythompson80 3y ago
  I can't help but think that this just gives you the worst of both worlds. You are now on the hook managing that non-standard "smart" wrapper which will quickly just become the status quo for the project. Anyone wanting to change how it works needs to understand exactly how "smart" you made it and all the side effects that will come with making a change there. I pushed against knative in our company particularly for that reason. Like we wanna use kafka because [Insert kafka sales pitch], but we don't want our developers to utilize any of the kafka features. We're just gonna define the kafka client in some yaml format and have our clients handle an http request per message. It didn't make sense to me.
  - robertlagrant 3y ago
    This would be my instinct too.
  - supermatt 3y ago
    Thats kind of like saying dont use any software libraries because they all use the standard lib indirectly so you may as well just use that? Its just an abstraction layer to make things less effort.
raducu 3y ago
> who can't just now pluck messages off a queue to process The problem is you cannot mark individual messages as read, for a given consumer&partition you can only update the offset for a partition. If a certain message processing takes very long, all other messages in that partition will have to wait. Also, with kafka, the max read concurrency is equal to the number of partitions, for something like rabbitMq it is much higher; but you do get nice message ordering for any given partition in kafka which you do not get in RabbitMq (afik); you are also get some really nice data locality with kafka because unless the consumers get the partitions re-assigned, all messages for the same key are served on the same physical consumer.
- math 3y ago
  Worth noting that Kafka is getting queues: https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A+Queues+for+Kafka https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A...
  - facorreia 3y ago
    And also Rabbit has streams[1]. There's a lot of overlap. [1] https://www.rabbitmq.com/streams.html https://www.rabbitmq.com/streams.html
- ryanjshaw 3y ago
  > The problem is you cannot mark individual messages as read, for a given consumer&partition you can only update the offset for a partition. Hence "smart clients". If you MUST process every message at least once, you will anyway be tracking messages individually on the client (e.g. a DB or file system plus logic for idempotent message processing) and thus disable auto-offset commits back to the cluster for your consumer. RabbitMQ says "let me track this for you", Kafka says "you already need to track this so why duplicate the data in the cluster and complicate the protocol". If you don't have reliable persistent storage available and insist on using the Kafka cluster to track offsets, you can track processed offsets in memory and whenever your lowest processed offset moves forward, you have your consumer commit that offset manually as part of its message loop. If your service restarts your downstream commands need to be idempotent of course because you will reconsume messages you may have previously processed, but this would be the case regardless of Kafka or RabbitMQ unless you're using distributed transactions (yuck). > If a certain message processing takes very long, all other messages in that partition will have to wait. You can stream messages into a buffer and process them in parallel, and commit the low watermark offset whenever it changes, as described above. I've implemented this in .NET with Channels and saturate the CPUs with no problem.
pnpnp 3y ago
Just my 2c but for anyone unaware, you should check out NATS. It combines the best of both Kafka and RabbitMQ IMO.
- doctor_eval 3y ago
  I thought NATS didn't actually store messages, am I mistaken? Looking at Wikipedia (https://en.wikipedia.org/wiki/NATS_Messaging https://en.wikipedia.org/wiki/NATS_Messaging) I see that I'm technically right, it's JetStream that does the storage layer - but it's part of the NATS server. From memory, I really liked the philosophy of NATS but found the nomenclature confusing.
  - pnpnp 3y ago
    I think they call that part of NATS “Jetstream” if I’m not mistaken. I haven’t used it, but I believe it has some form of message persistence. I have used it mostly for message-first services, and found subject-based messaging a breath of fresh air to decouple services. You can do the same thing with RabbitMQ topic exchanges, but it requires quite a bit more hand-waving.
    - zerbinxx 3y ago
      Jetstream does indeed have message persistence: I can issue queries like “get messages on topic since 5 minutes ago” - I do this a ton. However, that seems to be the extent of the storage/query API that it exposes for historical messages. I’m quite a big fan, and would recommend it with the caveat that Jetstream is considerably more complex than simple nats and I get the feeling I’m barely scratching the surface with it.
      - doctor_eval 3y ago
        I’m really interested in a Kafka like message broker in the Go ecosystem and look forward to checking it out for whatever my next project ends up being.
cbsmith 3y ago
You're getting to the key thing: You don't want to classify them by what they do. You want to classify them by what the clients must do/experience.
continuitylimit 3y ago
It is not odd, it is basically accurate. You are making a fetish of the S-C interaction but the essential matter is that Kafka is designed to store & distribute logs, whereas Rabbit is designed to route & send messages. The ‘store’ bit is very much a part of Kafka’s mission statement but not for Rabbit.

BWStearns 3y ago

My comment is mostly about part 2 of this post, but wrt message ordering being a kafka "win" I'd raise the point that in the actual use case of "a consumer fails in some way to process the message" you can still end up with out of order processing of the consumer's input since you might want to dump them into a DLQ or something. The fact that the message isn't reappended to the topic by default for processing is kind of an academic point no? Unrelatedly, I've been looking at Pulsar lately. Anyone have experience with Pulsar and either RMQ/Kafka want to throw out some opinions from having tried both?

EdwardDiego 3y ago
Pulsar can have both MQ semantics and pub/sub semantics. In pub/sub it's sorta like "Kafka with all bits people found it necessary to build later already built in", e.g., a proxy, schema registry, connectors, replication, tiered storage, all out of the box. It also has lightweight streaming functions built-in, but they operate per record, so good for lightweight transforms/routing, not for stream aggregations etc. It has more moving parts also, brokers are decoupled from storage, which is handled by BookKeeper, and replication between two clusters requires a ZooKeeper that both clusters can access, in addition to the ZK used by the brokers and bookies. And it's a reasonably new project, so last time I looked into it, some of the documentation was incorrect, especially around managing bookies.
- relay23 3y ago
  Massive and complex platform.. at a certain point why not just run 2 different platforms that are best of breed for each.
  - olavgg 3y ago
    You can just run Pulsar standalone which is super-simple, it is a great start and scales just fine for one machine. Once you outscale one machine, you can refactor Pulsar into a distributed setup. The advantage is that everything is still familiar.

bhouston 3y ago

If you use Confluent Kafka, the billing is pretty high. About 4 years ago it was much cheaper, but then they completely revamped the pricing to something ridiculous. I found that switching to Google Pub/Sub, at least if it meets your needs, is cheaper.

AndyPa32 3y ago
Yes, I can confirm that. Confluent is the most expensive part of our current infrastructure.
- bhouston 3y ago
  Can you switch away from it? Or do you need its advanced features?
- relay23 3y ago
  Sounds about right.
esafak 3y ago
I see they offer Kafka's exactly-once delivery: https://cloud.google.com/blog/products/data-analytics/cloud-pub-sub-exactly-once-delivery-feature-is-now-ga https://cloud.google.com/blog/products/data-analytics/cloud-...
- relay23 3y ago
  Kafka doesn't guarantee exactly once delivery at all, unless you're using Kafka Streams and even then your final output topic still won't get exactly once, the consumer group protocol doesn't allow for it.
relay23 3y ago
It's cheaper until you get to a sizeable workload, and the P90+ latency is ridiculous.. the kafka api is weak and when you're not using kafka api you're limited on integration tools unless you want to be super locked in to GCP.

dividedbyzero 3y ago

NATS (https://nats.io/ https://nats.io/) is another option, though I'm not sure if it's still considered a viable Kafka replacement.

vorpalhex 3y ago
Different set of promises. NATS is great but has a different tradeoff bargain from Rabbit or Kafka.
- kumarvvr 3y ago
  Could you expand on this a bit more? I am curious.
  - vorpalhex 3y ago
    NATs has a decent-ish guide here: https://docs.nats.io/nats-concepts/overview/compare-nats https://docs.nats.io/nats-concepts/overview/compare-nats A few things they get wrong mostly about Rabbit: + RabbitMQ does support replay, and also has a memory only mode which will support persistance in a cluster + RabbitMQ doesn't have that sensitive of a latency between cluster members (no more sensitive than NATS in some setups). + RabbitMQ also supports Prometheus A good (but incomplete) rule of thumb: + Kafka is a distributed Append-Only-Log that looks like a message bus. It allows time travel but has very simple server semantics. + RabbitMQ is a true message broker with all the abilities and patterns therein. It comes with more complexity. + NATs is primarily a straight forward streaming messaging platform. Also consider Redis' message queue mode, zeromq, mqtt brokers (Eclipse Mosquitto) and the option of just not using a message broker/queue system. Even as someone who really likes the pubsub pattern, there's a good chance you don't need it and you may be heading to a distributed monolith antipattern.
    - gunapologist99 3y ago
      > distributed monolith antipattern Beauty is in the eye of the beholder, I guess. (I would definitely not consider a distributed monolith an antipattern) Also wanted to add NSQ to your very good list.

bicijay 3y ago

Message ordering is an illusion. Unless you track/store the messages on the client and are willing to deal with stuck queues due to failures in one "poisoned" message.

klabb3 3y ago
There are different kinds of order. Yes, there’s no total order in a distributed system, but you can have certain partial order guarantees. It’s nice if something is added before it’s updated, for instance.
- bicijay 3y ago
  Could you expand on that? How would you achieve "partial order" guarantees?
  - klabb3 3y ago
    One type of partial order would be that a producer puts all the messages that are related in the same queue, so that A always precedes B. Basically the invariant becomes: If a consumer sees an event B, it will have certainly have seen the event A before that. Assuming business logic is correctly written, that saves you from having to write certain retry logic on the B handlers. This requires the message queue to be always available. If it goes down, the system would not make progress (like a db - in fact the MQ is a db). Once you add more actors/nodes to the same related events, maintaining a “causal order” can be very tricky and subtle, especially if you have an MQ and a DB as multiple sources of truth. So I’m not exactly endorsing it, even though MQ-as-a-DB (aka event sourcing) is a very interesting idea.
  - CogitoCogito 3y ago
    Let's say you have events coming in that result in inserts, updates, and deletes on a table with a certain primary key. Assuming no dependencies external to this table, you only need events involving a specific key to be ordered. I.e. it doesn't really matter if row_a gets updated before or after row_b. In either case, you end up with the same thing. So if you do something like kafka partitions and you send events to certain partitions based on their primary key, then those partitions will be ordered which will be enough. That doesn't fix your example of dealing with individual errors, but in many cases that's enough.

tannhaeuser 3y ago

I think this is an excellent article. The only thing I'd add is that RabbitMQ is an implementation of AQMP (optionally v1.0) as a standardized broker service protocol so is designed to be interchangeable with other extant implementations such as Apache Active MQ and Cupid whereas Kafka is one-of-a-kind software. Beyond that RabbitMQ has standardized client libs and frameworks in Java land if that matters to you - it did matter in the original context of message queue middlewares and SOA from where AMQP originated and where enterprise messaging sees major use. OTOH Kafka, with caveats, is in principle more "web scale" - though that is far from a free ride.

whalesalad 3y ago

One is a tomato, the other is an orange. From a distance they might look alike but they really are two completely different tools. This is a pretty solid explanation of the differences with good illustrations. Rabbit can do everything Kafka does - and much more - in a more configurable manner. Kafka is highly optimized for essentially one use case and does that well. Nothing in life is free, there are trade-offs everywhere. I am not privy to which one is theoretically faster - but once you reach that question methinks the particular workload is the deciding factor.

JohnMakin 3y ago
I am not an expert in either and have only worked with Kafka. At a past job I had to write a connector job to parse and sanitize some extremely dirty, unstructured data and pass it along somewhere else. RabbitMQ supports this? What is the one use case of kafka? I think you have it backwards.
- whalesalad 3y ago
  > parse and sanitize some extremely dirty, unstructured data and pass it along somewhere else can you be more specific? that to me sounds like hello world for either of these tools. "santize data" is an application level concern that neither rabbit or kafka would handle. as far as "pass along somewhere else" again both tools can do.
  - JohnMakin 3y ago
    It was a Sink Connector. I don’t know what it was or wasn’t supposed to do but I was asked to do it, as is often the case in tech. I could have done any number of transformations in that process though, which I’m not sure rabbitmq supports
    - whalesalad 3y ago
      It sounds to me like you aren't really even sure what you built. I have operated both rabbit and kafka at scale I definitely do not have it backwards :)

FridgeSeal 3y ago

I’m personally a fan of Kafka. I think the design of persisting the messages, and tracking offsets for progress instead of message acknowledgments is a much cleaner and more versatile design. You can get all the same advantages of message acknowledgments, but now you can also replay queues, let different applications use the messages (handy for cross cutting event/notification systems) and you get better scaling properties-which doesn’t hurt at the small scale, and provides further scaling when you need it.

whalesalad 3y ago
> You can get all the same advantages of message acknowledgments, but now you can also replay queues with rmq you can reject/nack a message and have it put back on the queue. rmq is not well suited for long term historical retention inside queues a-la kafka's logs but it is possible to do. > let different applications use the messages (handy for cross cutting event/notification systems) rmq also does a publish once and fanout to multiple queues to support this. data is replicated so that could be a deal breaker, but it is possible. how often have you had to diagnose a stuck consumer or some other kind of offset glitch where a consumer is unable to resume where it left off? not knocking kafka here but I do think it is a tool you should reach for when you need to solve a very hyper focused problem, while rabbit is a tool more suited to most cases where queuing is required. kafka is a code smell in a lot of organizations from my experience - most do not need it.
- raducu 3y ago
  > afka is a code smell in a lot of organizations from my experience - most do not need it. Kafka is really nice if you don't care that much about latency during peak load and you don't have absurd processing times for messages.
  - ceencee 3y ago
    Kafka can sustain sub 20ms at millions or even billions per second scale. Processing time delays is bad consumer code and partition design smell. Aka , your consumer shouldnt depend on a slower resource within an ordering domain. This can also be mitigated with an async consumer

purpleblue 3y ago

If someone is asking if they should decide between RabbitMQ vs Kafka, they should 100% use RabbitMQ. It means they have no idea what they're dealing with, the architectural differences, and the investment that the company needs in order to use Kafka. So use RabbitMQ.

patrec 3y ago
How do you create anything with RabbitMQ that a) has performance characteristics under load you can reason about and b) can handle individual node or networking failures without data loss? Kafka is overkill in most scenarios and you should probably just see if postgres isn't enough for your needs first (especially since you will almost certainly already need a database anyway). Kafka is more pain to setup and run than it ought to be. But underlying it is a useful and sensible abstraction for building robust distributed systems.

postalrat 3y ago

Redis isn't an option?

rafaelturk 3y ago
Yes, Don't why is not mentioned in the article, maybe because is more barebones.
noitpmeder 3y ago
I'd also like someone with experience to contrast redis for these use cases.
blindriver 3y ago
Redis is a cache, it's not a queue. Can you do something similar? I guess, just like you can use a highlighter to paint your house. But the semantics are not correct.
- relay23 3y ago
  There is Redis Streams, but certainly not without it's problems. Super obscure, not a lot of client support.
akvadrako 3y ago
Or postgres. If you are under 1000 messages a second it functions well as a transactional queue.
- sgarland 3y ago
  Until table bloat happens.

rafaelturk 3y ago

Nice post! RabbitMq is battle tested, exceptionally fast and low resources app. Capable of handling millions of transactions/second. RabbitMQ will handle vast majority of usecases. I'm puzzled why often startups, or even banks use Kafka, soley because is hype. Kafka on the order hand requires massive CPUs, Memory, often requiring its own K8S cluster just to be alive.

rafaelturk 3y ago
If your have a clean event-driven architecture, ie messages are completely agnostic and decoupled from one-another you don't need Kafka.
- ceencee 3y ago
  This is a ridiculous statement if you really build an EDA. Kafka is what enables the decoupling.
- inkyoto 3y ago
  Event-driven architecture is an architectural principle, and Kafka, RabbitMQ/ActiveMQ, Pulsar, NATS and so forth are implementations that support the event-driven architectural principle. Yet, all of them range in a variety, complexity and extent of features they provide which may or may not be a good fit for a particular use case. Traditional message brokers (RabbitMQ and similar) do support the event-driven architecture, yet the data they handle is ephemeral. Once a message has been processed, it is gone forever. Connecting a new raw data source is not an established practice and requires a technical «adapter» of sorts to be built. High concurrency levels is problematic for scenarios where the strict message processing ordering is required: the traditional message brokers do not handle it well in highly parallel scenarios out of the box. Kafka and similar also support event-driven architectures, yet they allow the data to be processed multiple times – by existing (i.e. a data replay) and, most importantly, new or unknown at the time consumers (note: this is distinct from the data replay!). This is allows to plug existing data source(s) into a data streaming platform (Kafka) and incrementally add new data consumers and processors over the time with the datasets being available intact. This is an important distinction. Kafka and similar also improve on the strict processing order guarantee by allowing a message source (a Kafka topic) to be explicitly partitioned out and guaranteeing that the message order will be retained and enforced for a consumer group receiving messages from that partition. To recap, traditional message brokers are a good fit for handling the ephemeral data, and data streaming platforms are a good fit for connecting data sources and allowing the data to be ingested multiple times. Both implement and support event-driven architectures in a variety of scenarios.

dventimi 3y ago

> Apache Kafka is not an implementation of a message broker. Instead, it is a distributed streaming platform. Unlike RabbitMQ, which is based on queues and exchanges, Kafka’s storage layer is implemented using a partitioned transaction log. Kafka also... This seems like an important passage, drawing the crucial and long-awaited distinction between RabbitMQ and Kafka, and yet without having defined a "partitioned transaction log" the author strands the reader without any help in absorbing the distinction.

zerbinxx 3y ago
This is sadly commonplace in tech blogs: rather than taking a great opportunity to hook the reader, the writer will drop a vocab term in bold and move on.
estiller 3y ago
Hi, this is the article author here. Thanks for the feedback! I've written this article 3.5 years ago, and it could definitely deserve a shake-up. I agree this should be cleared up a bit.

bob1029 3y ago

I've seen Tibco Rendezvous used in manufacturing. ~300 megabytes per hour of raw log generated 24/7/365 by tools and control systems in a factory setting. Probably on the order of 10k+ participants in the pub/sub network. If you are running something like a factory where thousands of independent systems need to communicate in some way, this kind of tech starts to look like the only option. If you are orchestrating the concerns of 5-10 services, I think you are making your life harder than it needs to be with ESB-style abstractions. Direct method invocation is much more reliable than whatever any one of these vendors could ever sell you. Put all your services into one exe. If you can't be bothered to use one language/repo, there are still ways to achieve this. The real architect's dilemma is shoving one's edifice-constructing ego into a box long enough to produce a useful shack for the business.

strangemonad 3y ago
Or NATS
- relay23 3y ago
  or kafka, or any other publish/subscribe system
kuchenbecker 3y ago
Our prod cluster generates that about every minute at O(1M) qps. We JUST turned on remote Logs because until now Kafka didn't have capacity.
- inkyoto 3y ago
  TIBCO Rendezvous is tech from 1998/99, millions of messages per second didn't exist at the time. Only NYSE and NASDAQ were capable of producing millions of events back then (still not by minute nor by second). TIBCO Rendezvous was one of the first successful large scale, low latency and near real-time pub/sub implementations, and it had a very efficient, Avro like, wire level serialisation format that made messages very compact and efficient to deliver. It was very popular in finance, banking and manufacturing, and is all but legacy now.

cma256 3y ago

The great thing about Kafka is the ability to batch operations. Collect a set of messages in memory and, when its time to commit, submit a bulk operation. If something fails you just rebuild the buffer from the last committed offset. Pretty neat piece of technology.

jpgvm 3y ago

If you want features of RabbitMQ (specifically queue like behavior) but the scalability of Kafka then you probably want Apache Pulsar. To elaborate on that a bit the main things Pulsar gives you are: 1. Still underlying distributed stream based architecture, this is what makes it able to do Kafka like things. 2. Broker side management of subscription state which allow out of order acknowledgement, this means you can use it like a queue. (Subscriptions sort of act like AMQP mailboxes but without the exchange routing semantics). Vs Kafka which can only do cumulative acknowledgement, i.e head of line blocking. 3. Separated "compute" and storage. By storing data in Bookkeeper you can scale your needs to support a lot of consumers separately from how you stash the data those consumers need to read vs Kafka where these 2 are coupled and an imbalance between the two becomes awkward. 4. In built offload with transparent pass-through read. When your data falls off the retention cliff for your standard broker cluster the data can be archived to object storage. The broker can transparently handle read request for these earlier messages though, just with higher startup latency to pull the archived ledgers. 5. Way more plugability than Kafka, in-fact similar plugability as RabbitMQ. You can implement your own authz/authn, a different listener to support a different protocol (there is a Kafka one, MQTT, AMQP etc). 6. Much greater metadata scalability. Before the new KRaft implementation the layout of metadata in ZK meant that you couldn't feasibly have more than about 10k topics. Especially because of how long the downtime would be on controller failover. Pulsar can easily support much larger numbers of topics which prevents needing to use a firehose design when you would prefer individual topics per tenant/customer/whatever.

mianos 3y ago
While pulsar on paper seems a superior solution, in my experience it is very still very immature and very buggy. I really want to use it over kafka but I would not bet my business on it. I am not a fan of Kafka, it's kinda old, and the code is a bit messy, a lot of the once only semantic problems 100% solved by Pulser are sorta kinda in Kfaka these days. All the newer stuff like built in RAFT makes it competitive with anything. But, anyone who has used Kafka at scale has to say it 100% does what it says on the box. Many people are used to it's idiosyncrasies at scale and can get it to scale. Now RabbitMQ. I have been burnt so badly by it breaking at scale I'll never be touching it again. Maybe my fault, maybe not, but replacing it with Kafka solved all my flakey issues and I never looked at it again.

deleted 3y ago

[deleted]

jonstewart 3y ago

I'm sure there remain good use cases for message buses that you have to run yourself, where there really are millions of messages that can't be batched up and need real-time and whatnot. But you can get pretty far with: 1. Write a bunch of records to an S3 object. 2. Trigger a lambda to process — infinite scale out! and if a queue really is needed due to constrained consumers, then: 1. Write a bunch of records to an S3 object. 2. The lambda trigger puts a message into an SQS queue with the S3 object's URI. 3. An auto-scaling group gets the message off the queue and processes the data in the object. If ordering's important, then the SQS queue can be made FIFO. That has pretty low numbers in messages/second, but since records are being batched into S3 objects you can still have fairly high throughput. It used to be that the elegance of queues for such systems would tempt developers into the operational slough of despond that is running these types of systems. Again, I'm sure it's warranted for some applications, but S3, Lambda, and SQS as above work nicely together.

pramodbiligiri 3y ago
You can get rid of Step 2 if you're going to be creating a new S3 object for each batch. AWS has an inbuilt feature to trigger SQS notifications for S3 operations: https://docs.aws.amazon.com/AmazonS3/latest/userguide/enable-event-notifications.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/enable...

deleted 3y ago

[deleted]

pharmakom 3y ago

seems like there is a missing option? - Message retention (like Kafka) - Easy consumer scale out (like RabbitMQ) - No particular ordering guarantees (like RabbitMQ)

pbowyer 3y ago

What these articles always miss is what are the real-world scenarios that use Kafka or RabbitMQ? I've never used one professionally and beyond the async processing cases I come across in web dev (mostly sending emails asynchronously), or seeing devs using one to handle 5 req/s which doesn't need it, I don't have a good feel for when they are really needed - especially for companies at sub-Uber scale. What real-world cases would you use them for?

estiller 3y ago

Hi everyone, this is the article author here. I was genuinely surprised to see the article pop up here 3.5 years after I wrote it. Thank you for all the feedback! Given the advances in the past few years in this area, this article could use a serious update. A comparison to NATS and Pulsar is warranted, along with some extra explanations for some of the technical terms in the article. I'll carefully review the feedback here and try to make an update at some point soon.

opentokix 3y ago

Wow, this is the type of trash one can expect from someone calling themselves "Architect"

rrr83 3y ago
Be kind! Author has shared his findings based on his understanding, if you have more to add go ahead and create another page?

h1fra 3y ago

RabbitMQ is one the most impressive technology I have ever used I think. It's simple, efficient and reliable. In 10 years, I have never seen a crash or a bug, even with heavy load.

vorticalbox 3y ago
where i work we have some queues with millions of items in and the thing is perfectly stable.

onetimeuse92304 3y ago

If you have a throughput problem you are most likely doing it wrong. If your knee jerk reaction is to scale your messaging up you may want to reconsider. Messaging systems are usually hard to scale up and always very costly to do so compared to the amount of data they are transferring. The simplest thing you can do is to realise WHY you are using messages. Messages are there to trigger a process. Usually, you don't need a lot of data to trigger a process, the message just needs to let the system know enough to locate all necessary information. Also, when you are sending information at an extremely high rate, there usually is no difference if each message is processed separately or in batches. So what you can do in practice? 1) Get the producer to batch the messages. For example set rules like "batch up to 10.000 messages, up to 100ms, up to 100MB of data, whichever comes first". 2) Serialise the batch (for example, if it makes sense, create a compressed JSON file) 3) Upload the file to some high throughput, scalable, cheap storage (for example S3) 4) Send a message to the queue / topic / whatever else you are using with just enough to locate and process the message -- usually just the link to the S3 object. This usually can be modified depending on specific project needs. Now your messaging only ever sees a small number of very small messages and you will never have any scaling problems, at least not on messaging side.

usrusr 3y ago
If your use case can stomach the added failure modes this implies, yeah, that's what you can do.
- onetimeuse92304 3y ago
  Everything has some cons, some failure modes. All engineering is about knowing, understanding and making tradeoffs. In my practice, I am happy if I can get rid of hard problems (my messaging platform being unable to process X messages per second) and replace them with relatively easier problems (my persistence might sometimes fail and then I can't send a message). I would argue that distributed persistence solutions are usually more reliable than messaging platforms and also what is a very large throughput for a messsaging solution is usually nothing much for monsters that are engineered to take much larger volumes of data. And so, in my experience, reducing load to messaging and increasing load to persistence is net positive for the overall reliability.