7 ms·

RabbitMQ 3.11 Feature Preview: Super Streams

95 points by codeadict 4y ago

FridgeSeal 4y ago
A bit of a weird article. It announces super streams-which are basically topic partitions in Kafka, acknowledges that, but then hand-waves “oh but they’re different” and then spends the rest of the article talking about all the features that are identical to how topic partitions work… It’s a good feature, there’s nothing wrong with that, but there’s nothing wrong with saying “we’ve brought this feature to RabbitMQ too” rather than trying to pretend it’s totally-not-the-same.
- simonw 4y ago
  Here's the full paragraph where they talk about Kafka: > Let’s talk about the elephant in the room: how does it compare to Kafka? We can compare a super stream to a Kafka topic and a stream to a partition of a Kafka topic. A RabbitMQ stream is a first-class, individually-named object though, whereas a Kafka partition is a subordinate of a Kafka topic. This explanation leaves a lot of details out, there is no real 1-to-1 mapping, but it is accurate enough for our point in this post. I think that's fine. This is a post about a new feature in RabbitMQ, from the maintainers of RabbitMQ. It's not intended as a RabbitMQ-Kafka comparison post. They acknowledge that it's equivalent to Kafka, and then chose not to spend time digging into the details of how it's different - I'm willing to take their word for it that there are all sorts of interesting technical differences here, and I'm fine with them not addressing those in the post where they announce their new feature to the world. I don't see this as them pretending that it's totally-not-the-same.
kungfufrog 4y ago
If I had to nominate a piece of software, as an SRE, that is as close to "set and forget" as possible, it'd absolutely be RabbitMQ standalone and a close runner up would be its clustered form. I've worked in 3 places where RabbitMQ has been a fundamental cornerstone of the architecture, and while it does require a little tuning around performance occasionally (generally because it's being used inappropriately or without full consideration of its limitations/best practices), it's rock solid, easy to debug/inspect, has an active and supportive community, and is generally just all around pleasant to work with and maintain. Kudos to RabbitMQ and its developers! As an aside, the addition of recent features such as super streams, streams and quorum queues make it a compelling all-in-one tool for solving a bunch of architectural based concerns/requirements in application and infrastructure development. I've often thought about why it's not more utilised in the ops side of the world for metrics gathering and other usecases. I have also wondered how useful it'd be for log ingestion with lazy queues etc. Does anyone out there have examples of unusual use cases for RabbitMQ where it's outshone some alternative product? Would love to hear about them!
- fmorel 4y ago
  Absolutely. My team uses .NET, so we use the NServiceBus library on top of it, and RabbitMQ has been rock-solid. We never have to think about it, it's just been running for years.
- dalyons 4y ago
  interesting, i have the opposite experience. When we got to high throughput in a cluster we had all sorts of crashes, partitions, and nasty failure modes that required stressful delicate rebuilds. It has similar challenges to a clustered M-M relational db. Moved the high volume events to kinesis which was far more reliable for that use case. At low volumes yeah sure, it just ticks along, and does what you want.
  - amock 4y ago
    I also found it to be unreliable. I only used it at one company, but it was the most unreliable and hard to diagnose piece of our infrastructure. We didn't even have high throughput and I wouldn't use it again without a really good reason.
  - florbo 4y ago
    What's low volume? At one point I used rmq to ingest ~20,000 messages per second of varying length. From Tweets to blog posts, all containing the full content of the activity with metadata. It was with 3 node cluster.. wish I remembered the specs but nothing crazy aside from the SSD IOPs. The one time it fell over was when the consumers were broke long enough to fill up the disks.
    - dalyons 4y ago
      around there & upwards. You've listed one of the big problems with rabbit @ volume - inevitably/unavoidably you are going to have consumers go down or so slow. At a high enough volume you're heading for a crash/partition quickly if you cant respond fast enough (where "fast enough" is a time window inverse to how high volume the queue is). Its a crappy failure mode to have a sword hanging over you like that. other log-based messaging technologies like kinesis, kafka, etc do not care if a consumer goes down & are thus much safer.
hnrodey 4y ago
My own blog post: What I Wish Someone Would Have Told Me About Using Rabbitmq Before It Was Too Late https://ryanrodemoyer.github.io/what-i-wish-someone-would-have-told-me-about-using-rabbitmq-before-it-was-too-late/ https://ryanrodemoyer.github.io/what-i-wish-someone-would-ha...
- jsmeaton 4y ago
  Some great points there, I’ve been hit by all of them. I particularly like the first which is to engage an expert to validate your design. There are so many kinds of deployments and configurations and knowing the best for your application isn’t at all obvious.
  - deleted 4y ago
    [deleted]
latch 4y ago
I read the blog post and the Java documentation (which is more informative), and maybe I missed it, but I can't figure out how you define the # of streams/partitions. Is it defined when the superstream is created? What's the general practice for multi-tenancy? Say we have millions of clients with thousand being added daily and for various reasons, at least for _some_ message types, we'd like to keep them separate (for example, maybe strict ordering is super important, so we can't throw messages away, but we don't want a poison message to impact all customers).
- rad_gruchalski 4y ago
  Yup. Or your clients can create their own queues. So you want tooling supporting things like list queues only of a single tenant.
scumola 4y ago
TIL: RabbitMQ is written in erlang

95 points by codeadict 4y ago

A bit of a weird article. It announces super streams-which are basically topic partitions in Kafka, acknowledges that, but then hand-waves “oh but they’re different” and then spends the rest of the article talking about all the features that are identical to how topic partitions work… It’s a good feature, there’s nothing wrong with that, but there’s nothing wrong with saying “we’ve brought this feature to RabbitMQ too” rather than trying to pretend it’s totally-not-the-same.

simonw 4y ago
Here's the full paragraph where they talk about Kafka: > Let’s talk about the elephant in the room: how does it compare to Kafka? We can compare a super stream to a Kafka topic and a stream to a partition of a Kafka topic. A RabbitMQ stream is a first-class, individually-named object though, whereas a Kafka partition is a subordinate of a Kafka topic. This explanation leaves a lot of details out, there is no real 1-to-1 mapping, but it is accurate enough for our point in this post. I think that's fine. This is a post about a new feature in RabbitMQ, from the maintainers of RabbitMQ. It's not intended as a RabbitMQ-Kafka comparison post. They acknowledge that it's equivalent to Kafka, and then chose not to spend time digging into the details of how it's different - I'm willing to take their word for it that there are all sorts of interesting technical differences here, and I'm fine with them not addressing those in the post where they announce their new feature to the world. I don't see this as them pretending that it's totally-not-the-same.

kungfufrog 4y ago

If I had to nominate a piece of software, as an SRE, that is as close to "set and forget" as possible, it'd absolutely be RabbitMQ standalone and a close runner up would be its clustered form. I've worked in 3 places where RabbitMQ has been a fundamental cornerstone of the architecture, and while it does require a little tuning around performance occasionally (generally because it's being used inappropriately or without full consideration of its limitations/best practices), it's rock solid, easy to debug/inspect, has an active and supportive community, and is generally just all around pleasant to work with and maintain. Kudos to RabbitMQ and its developers! As an aside, the addition of recent features such as super streams, streams and quorum queues make it a compelling all-in-one tool for solving a bunch of architectural based concerns/requirements in application and infrastructure development. I've often thought about why it's not more utilised in the ops side of the world for metrics gathering and other usecases. I have also wondered how useful it'd be for log ingestion with lazy queues etc. Does anyone out there have examples of unusual use cases for RabbitMQ where it's outshone some alternative product? Would love to hear about them!

fmorel 4y ago
Absolutely. My team uses .NET, so we use the NServiceBus library on top of it, and RabbitMQ has been rock-solid. We never have to think about it, it's just been running for years.
dalyons 4y ago
interesting, i have the opposite experience. When we got to high throughput in a cluster we had all sorts of crashes, partitions, and nasty failure modes that required stressful delicate rebuilds. It has similar challenges to a clustered M-M relational db. Moved the high volume events to kinesis which was far more reliable for that use case. At low volumes yeah sure, it just ticks along, and does what you want.
- amock 4y ago
  I also found it to be unreliable. I only used it at one company, but it was the most unreliable and hard to diagnose piece of our infrastructure. We didn't even have high throughput and I wouldn't use it again without a really good reason.
- florbo 4y ago
  What's low volume? At one point I used rmq to ingest ~20,000 messages per second of varying length. From Tweets to blog posts, all containing the full content of the activity with metadata. It was with 3 node cluster.. wish I remembered the specs but nothing crazy aside from the SSD IOPs. The one time it fell over was when the consumers were broke long enough to fill up the disks.
  - dalyons 4y ago
    around there & upwards. You've listed one of the big problems with rabbit @ volume - inevitably/unavoidably you are going to have consumers go down or so slow. At a high enough volume you're heading for a crash/partition quickly if you cant respond fast enough (where "fast enough" is a time window inverse to how high volume the queue is). Its a crappy failure mode to have a sword hanging over you like that. other log-based messaging technologies like kinesis, kafka, etc do not care if a consumer goes down & are thus much safer.

hnrodey 4y ago

My own blog post: What I Wish Someone Would Have Told Me About Using Rabbitmq Before It Was Too Late https://ryanrodemoyer.github.io/what-i-wish-someone-would-have-told-me-about-using-rabbitmq-before-it-was-too-late/ https://ryanrodemoyer.github.io/what-i-wish-someone-would-ha...

jsmeaton 4y ago
Some great points there, I’ve been hit by all of them. I particularly like the first which is to engage an expert to validate your design. There are so many kinds of deployments and configurations and knowing the best for your application isn’t at all obvious.
- deleted 4y ago
  [deleted]

latch 4y ago

I read the blog post and the Java documentation (which is more informative), and maybe I missed it, but I can't figure out how you define the # of streams/partitions. Is it defined when the superstream is created? What's the general practice for multi-tenancy? Say we have millions of clients with thousand being added daily and for various reasons, at least for _some_ message types, we'd like to keep them separate (for example, maybe strict ordering is super important, so we can't throw messages away, but we don't want a poison message to impact all customers).

rad_gruchalski 4y ago
Yup. Or your clients can create their own queues. So you want tooling supporting things like list queues only of a single tenant.

scumola 4y ago

TIL: RabbitMQ is written in erlang