5 ms·

how do you design avoiding message queues? Or do you use other alternatives around kafka for these things?

freedomben 3y ago
Most of my designs include a message queue, and I would not use Kafka unless there was a strong need. Right now my preference varies a bit depending on the rest of the tech stack, but for the most part I use Redis or RabbitMQ. If the stack is already hard dependent on AWS or another cloud, then SQS or whatever is also fine. I also wouldn't overlook just using your existing DB (like postgres)! At low and even medium scale this can be totally fine, and also comes with lots of benefits like single-source-of-truth, normal relational DB constraints, transaction wrapping, and more. One of the highest scale apps I've worked on uses Postgres for queueing. It's take a number of optimizations over the years as performance starts to fall due to scale, but it's doable.
dopidopHN 3y ago
RabbitMQ, ActiveMQ, MQtt All of those are fine is you only need pub/sub
- TedDoesntTalk 3y ago
  Those are also message queues. They even use MQ in the name.
- valenterry 3y ago
  How good does RabbitMQ do in terms of availability nowadays? Because one thing a message queue should offer is high availability - otherwie it loses one of it's most compelling benefits.
  - zbentley 3y ago
    Rabbit's quorum queues are an improvement on the extremely poor HA/clustering system they provided previously. Users can now choose between both. Rabbit's defaults are still unfortunate, in my opinion: queues and messages are not disk-persisted by default, though this can easily be enabled. As a result, many folks run and benchmark a "high availability rabbit" only to discover that they're benchmarking distributed state stored in memory, not disk. https://www.rabbitmq.com/quorum-queues.html https://www.rabbitmq.com/quorum-queues.html
  - regularfry 3y ago
    The one thing that made me pull back from RabbitMQ years ago was that using it between datacentres was a bad plan, because all the clustering was based on Erlang's underlying cluster implementation and the advice on that was not to use it between geographically distinct locations. I don't know if it's since improved or if that advice no longer holds, but working under an environment where we needed cross-DC redundancy made it impossible to select, for that reason.
- gkhartman 3y ago
  Just to clarify MQTT is a pub/sub protocol, while ActiveMQ and RabbitMQ are message broker implementations. As an example ActiveMQ implements MQTT as one of its optional protocols. Though if anyone's looking for a simple MQTT implantation I'd recommend Mosquitto. My 2 cents as someone who's worked with ActiveMQ pub/sub for quite a few years.
richdougherty 3y ago
They may still use some sort of event/message system. Kafka is lower level than other sorts message queue systems and requires more work to get correct (dedupe, ordering, retry logic), but has great performance. It's often easier to choose a different messaging system though.
- EdwardDiego 3y ago
  Kafka ain't a MQ, it's a distributed log. Pulsar is a Kafkaesque system that can act as distributed log _or_ an MQ.
whartung 3y ago
There was recently an article about distributed systems that showed up here. (Harry Doyle: Christ, I can't find it. To hell with it!) And the author made a very interesting point about message queues. Simply, any problem that could be resolved by a message queue could be resolved by load balancing or persistence, and, therefore, messages queues were actually kind of a bad idea. There were two basic issues. The first is that because of the nature of message queues, they're either empty or full. The second is that for many of the ways that queue are used, the unit putting the request on the queue in the first place may well be waiting for the response related to the request. So you've just turned an every day synchronous request into a more complicated, out of band, multi-party asynchronous request. If your queues are not empty, they they are filling up. And queue capacity is a two fold problem. One, is that you simply run out of space. But, more likely, referring to the earlier point about waiting for a response, is that you run out of time. The response does not return fast enough to manage your response window to the unit making the request. This is a load balancing problem. If the queue is filling you simply don't have the capacity to handle the current traffic. It's also a mechanically simpler thing to send out a request and wait for the response than to do the dance via a message queue. The second part is that if you're throwing items onto a message queue, and you "don't care" about them, that is it's a true "fire and forget" kind of request and direct response time is not a concern, then what does the queue gain you over simply posting it to a database table? If the request is Important, you certainly don't want to trust it to a message queue, a device not really designed for the storage of messages. Messages in a queue are kind of trapped in no mans land, where the easiest way to get to a message is to dig through the ones piled in front of it. They're interesting insights and worth scratching your chin and going "Hmmm" over.
- valenterry 3y ago
  > It's also a mechanically simpler thing to send out a request and wait for the response than to do the dance via a message queue. Yeah, this is simpler for the requester, but not for the counterpart that has to respond. Because now, the responder has to have 100% uptime and better not fail during the request, otherwise things get lost. Let's take sending emails as an example. You have a server A that can send emails, you have a server B that fulfills requests/actions by a user. Let's just assume that this is the (legacy) setting we are dealing with. Now, what do you do if you want to send the user an email on a certain action, e.g. a password reset or changing a payment information etc.? Is B then making a synchronous request to A? What if server A is currently down due to maintenance? What if A uses another entity to send the emails, which itself is down? How do you observe the current state of the system and e.g. detect that A is overloaded? With a message queue you get all those things for free and A doesn't even have to have any persistence layer for "retries" in case of crashes etc. While it's true that those issues can all be resolved by "by load balancing or persistence" it just means that you now traded one issue (having a message queue) for multiple issues (having database(s), having load balancer(s) and essentially re-implementing a part of a message queue). In most cases a message queue seems like a good trade-off.

Most of my designs include a message queue, and I would not use Kafka unless there was a strong need. Right now my preference varies a bit depending on the rest of the tech stack, but for the most part I use Redis or RabbitMQ. If the stack is already hard dependent on AWS or another cloud, then SQS or whatever is also fine. I also wouldn't overlook just using your existing DB (like postgres)! At low and even medium scale this can be totally fine, and also comes with lots of benefits like single-source-of-truth, normal relational DB constraints, transaction wrapping, and more. One of the highest scale apps I've worked on uses Postgres for queueing. It's take a number of optimizations over the years as performance starts to fall due to scale, but it's doable.

dopidopHN 3y ago

RabbitMQ, ActiveMQ, MQtt All of those are fine is you only need pub/sub

TedDoesntTalk 3y ago
Those are also message queues. They even use MQ in the name.
valenterry 3y ago
How good does RabbitMQ do in terms of availability nowadays? Because one thing a message queue should offer is high availability - otherwie it loses one of it's most compelling benefits.
- zbentley 3y ago
  Rabbit's quorum queues are an improvement on the extremely poor HA/clustering system they provided previously. Users can now choose between both. Rabbit's defaults are still unfortunate, in my opinion: queues and messages are not disk-persisted by default, though this can easily be enabled. As a result, many folks run and benchmark a "high availability rabbit" only to discover that they're benchmarking distributed state stored in memory, not disk. https://www.rabbitmq.com/quorum-queues.html https://www.rabbitmq.com/quorum-queues.html
- regularfry 3y ago
  The one thing that made me pull back from RabbitMQ years ago was that using it between datacentres was a bad plan, because all the clustering was based on Erlang's underlying cluster implementation and the advice on that was not to use it between geographically distinct locations. I don't know if it's since improved or if that advice no longer holds, but working under an environment where we needed cross-DC redundancy made it impossible to select, for that reason.
gkhartman 3y ago
Just to clarify MQTT is a pub/sub protocol, while ActiveMQ and RabbitMQ are message broker implementations. As an example ActiveMQ implements MQTT as one of its optional protocols. Though if anyone's looking for a simple MQTT implantation I'd recommend Mosquitto. My 2 cents as someone who's worked with ActiveMQ pub/sub for quite a few years.

richdougherty 3y ago

They may still use some sort of event/message system. Kafka is lower level than other sorts message queue systems and requires more work to get correct (dedupe, ordering, retry logic), but has great performance. It's often easier to choose a different messaging system though.

EdwardDiego 3y ago
Kafka ain't a MQ, it's a distributed log. Pulsar is a Kafkaesque system that can act as distributed log _or_ an MQ.

whartung 3y ago

There was recently an article about distributed systems that showed up here. (Harry Doyle: Christ, I can't find it. To hell with it!) And the author made a very interesting point about message queues. Simply, any problem that could be resolved by a message queue could be resolved by load balancing or persistence, and, therefore, messages queues were actually kind of a bad idea. There were two basic issues. The first is that because of the nature of message queues, they're either empty or full. The second is that for many of the ways that queue are used, the unit putting the request on the queue in the first place may well be waiting for the response related to the request. So you've just turned an every day synchronous request into a more complicated, out of band, multi-party asynchronous request. If your queues are not empty, they they are filling up. And queue capacity is a two fold problem. One, is that you simply run out of space. But, more likely, referring to the earlier point about waiting for a response, is that you run out of time. The response does not return fast enough to manage your response window to the unit making the request. This is a load balancing problem. If the queue is filling you simply don't have the capacity to handle the current traffic. It's also a mechanically simpler thing to send out a request and wait for the response than to do the dance via a message queue. The second part is that if you're throwing items onto a message queue, and you "don't care" about them, that is it's a true "fire and forget" kind of request and direct response time is not a concern, then what does the queue gain you over simply posting it to a database table? If the request is Important, you certainly don't want to trust it to a message queue, a device not really designed for the storage of messages. Messages in a queue are kind of trapped in no mans land, where the easiest way to get to a message is to dig through the ones piled in front of it. They're interesting insights and worth scratching your chin and going "Hmmm" over.

valenterry 3y ago
> It's also a mechanically simpler thing to send out a request and wait for the response than to do the dance via a message queue. Yeah, this is simpler for the requester, but not for the counterpart that has to respond. Because now, the responder has to have 100% uptime and better not fail during the request, otherwise things get lost. Let's take sending emails as an example. You have a server A that can send emails, you have a server B that fulfills requests/actions by a user. Let's just assume that this is the (legacy) setting we are dealing with. Now, what do you do if you want to send the user an email on a certain action, e.g. a password reset or changing a payment information etc.? Is B then making a synchronous request to A? What if server A is currently down due to maintenance? What if A uses another entity to send the emails, which itself is down? How do you observe the current state of the system and e.g. detect that A is overloaded? With a message queue you get all those things for free and A doesn't even have to have any persistence layer for "retries" in case of crashes etc. While it's true that those issues can all be resolved by "by load balancing or persistence" it just means that you now traded one issue (having a message queue) for multiple issues (having database(s), having load balancer(s) and essentially re-implementing a part of a message queue). In most cases a message queue seems like a good trade-off.