4 ms·

There was recently an article about distributed systems that showed up here. (Harry Doyle: Christ, I can't find it. To hell with it!) And the author made a ver

There was recently an article about distributed systems that showed up here. (Harry Doyle: Christ, I can't find it. To hell with it!) And the author made a very interesting point about message queues. Simply, any problem that could be resolved by a message queue could be resolved by load balancing or persistence, and, therefore, messages queues were actually kind of a bad idea. There were two basic issues. The first is that because of the nature of message queues, they're either empty or full. The second is that for many of the ways that queue are used, the unit putting the request on the queue in the first place may well be waiting for the response related to the request. So you've just turned an every day synchronous request into a more complicated, out of band, multi-party asynchronous request. If your queues are not empty, they they are filling up. And queue capacity is a two fold problem. One, is that you simply run out of space. But, more likely, referring to the earlier point about waiting for a response, is that you run out of time. The response does not return fast enough to manage your response window to the unit making the request. This is a load balancing problem. If the queue is filling you simply don't have the capacity to handle the current traffic. It's also a mechanically simpler thing to send out a request and wait for the response than to do the dance via a message queue. The second part is that if you're throwing items onto a message queue, and you "don't care" about them, that is it's a true "fire and forget" kind of request and direct response time is not a concern, then what does the queue gain you over simply posting it to a database table? If the request is Important, you certainly don't want to trust it to a message queue, a device not really designed for the storage of messages. Messages in a queue are kind of trapped in no mans land, where the easiest way to get to a message is to dig through the ones piled in front of it. They're interesting insights and worth scratching your chin and going "Hmmm" over.

valenterry 3y ago
> It's also a mechanically simpler thing to send out a request and wait for the response than to do the dance via a message queue. Yeah, this is simpler for the requester, but not for the counterpart that has to respond. Because now, the responder has to have 100% uptime and better not fail during the request, otherwise things get lost. Let's take sending emails as an example. You have a server A that can send emails, you have a server B that fulfills requests/actions by a user. Let's just assume that this is the (legacy) setting we are dealing with. Now, what do you do if you want to send the user an email on a certain action, e.g. a password reset or changing a payment information etc.? Is B then making a synchronous request to A? What if server A is currently down due to maintenance? What if A uses another entity to send the emails, which itself is down? How do you observe the current state of the system and e.g. detect that A is overloaded? With a message queue you get all those things for free and A doesn't even have to have any persistence layer for "retries" in case of crashes etc. While it's true that those issues can all be resolved by "by load balancing or persistence" it just means that you now traded one issue (having a message queue) for multiple issues (having database(s), having load balancer(s) and essentially re-implementing a part of a message queue). In most cases a message queue seems like a good trade-off.
- regularfry 3y ago
  This is where the "or persistence" bit comes in, but the trade-off isn't as you describe. One way to approach it would be for B to set a flag against the user in the database they must already have, because they have users so that A can process password resets on its own schedule. It's add a column versus add a message queue. Databases already come with locking primitives and well understood patterns, no reinvention needed.
  - valenterry 3y ago
    You are now assuming that they indeed have a database nd store users in there, and not use a 3rd party system that might not even allow to attach meta data (or at least make it difficult).
    - regularfry 3y ago
      In which case why is the password reset my problem at all? But even if I've handed off identity management entirely, I almost certainly do have some per-user state. Otherwise... what on earth am I doing with users in the first place?
      - valenterry 3y ago
        Well, maybe because the 3rd party service is not reliable or you want to send an additional mail. But okay, fair point - password-reset is maybe not the greatest example. But it doesn't invalidate the general point (I gave a couple of other examples). > Otherwise... what on earth am I doing with users in the first place? Maybe just making sure that the user is known and has paid for plan X (if the 3rd party service offers that).
hughesjj 3y ago
Kafka is less message queue (rabbitmq, sqs) and more ordered stream/write ahead log (ala kinesis) > The first is that because of the nature of message queues, they're either empty or full. ... Wat? > that is it's a true "fire and forget" kind of request and direct response time is not a concern, then what does the queue gain you over simply posting it to a database table? Performance is why. The fire and forget aspect is like udp in the sense that you don't need to ensure ordering of messages (packets) or hard persistence to the database. Also, dead letter queues exist for a reason. Message queues are super useful. The highest performance systems I've seen use the message queue + pool of workers paradigm, as it allows you to better smooth your load (unlike immediate republishing like in sns, which requires hardware available to accept) with minimal guarantees (unlike a write ahead log such as kinesis). The buffer is also great because it allows you a bit more time to scale up both your message queue fleet and worker fleet when you get a load spike.
- thephyber 3y ago
  > > The first is that because of the nature of message queues, they're either empty or full. > ... Wat? I interpreted it as "they are either trending towards empty or full". The statement doesn't seem well thought through. That might be true (either empty or full) most of the time (maybe) _if you squint_, but the entire point of the Message Queue is to provide buffering from the transient state (somewhere between empty and full) trending toward the empty state.
  - Raidion 3y ago
    Yea, I feel like these people never got a chron daily data dump (times N customers). Scaling is dead simple, and there isn't a need (or ability) to instantaneously handle large bursty workloads in like 99% of cases. Longer SLAs mean it's also easier to hit those SLAs. Giving the clients realistic SLAs is super important.
pronoiac 3y ago
If I'm thinking of the same article: was it about going from message queues to state machines for more operational robustness?
pronoiac 3y ago
You might be thinking of: https://blog.nelhage.com/post/systems-at-capacity/ https://blog.nelhage.com/post/systems-at-capacity/
Jasper_ 3y ago
Was it this one? https://ferd.ca/queues-don-t-fix-overload.html https://ferd.ca/queues-don-t-fix-overload.html
zbentley 3y ago
> If the request is Important, you certainly don't want to trust it to a message queue, a device not really designed for the storage of messages. That's false. Virtually all MQ systems are designed to persist (often with replication/redundancy) and store data. Most MQs also support non-persistent delivery, with the cost/benefit (ephemerality/performance) that entails, but that doesn't mean that durable storage is any less well-supported. Sure, folks have plenty of operational war stories regarding failures of persistence in their MQ broker/cluster/whatnot. Same as the DBAs who manage relational databases.

valenterry 3y ago

> It's also a mechanically simpler thing to send out a request and wait for the response than to do the dance via a message queue. Yeah, this is simpler for the requester, but not for the counterpart that has to respond. Because now, the responder has to have 100% uptime and better not fail during the request, otherwise things get lost. Let's take sending emails as an example. You have a server A that can send emails, you have a server B that fulfills requests/actions by a user. Let's just assume that this is the (legacy) setting we are dealing with. Now, what do you do if you want to send the user an email on a certain action, e.g. a password reset or changing a payment information etc.? Is B then making a synchronous request to A? What if server A is currently down due to maintenance? What if A uses another entity to send the emails, which itself is down? How do you observe the current state of the system and e.g. detect that A is overloaded? With a message queue you get all those things for free and A doesn't even have to have any persistence layer for "retries" in case of crashes etc. While it's true that those issues can all be resolved by "by load balancing or persistence" it just means that you now traded one issue (having a message queue) for multiple issues (having database(s), having load balancer(s) and essentially re-implementing a part of a message queue). In most cases a message queue seems like a good trade-off.

regularfry 3y ago
This is where the "or persistence" bit comes in, but the trade-off isn't as you describe. One way to approach it would be for B to set a flag against the user in the database they must already have, because they have users so that A can process password resets on its own schedule. It's add a column versus add a message queue. Databases already come with locking primitives and well understood patterns, no reinvention needed.
- valenterry 3y ago
  You are now assuming that they indeed have a database nd store users in there, and not use a 3rd party system that might not even allow to attach meta data (or at least make it difficult).
  - regularfry 3y ago
    In which case why is the password reset my problem at all? But even if I've handed off identity management entirely, I almost certainly do have some per-user state. Otherwise... what on earth am I doing with users in the first place?
    - valenterry 3y ago
      Well, maybe because the 3rd party service is not reliable or you want to send an additional mail. But okay, fair point - password-reset is maybe not the greatest example. But it doesn't invalidate the general point (I gave a couple of other examples). > Otherwise... what on earth am I doing with users in the first place? Maybe just making sure that the user is known and has paid for plan X (if the 3rd party service offers that).

hughesjj 3y ago

Kafka is less message queue (rabbitmq, sqs) and more ordered stream/write ahead log (ala kinesis) > The first is that because of the nature of message queues, they're either empty or full. ... Wat? > that is it's a true "fire and forget" kind of request and direct response time is not a concern, then what does the queue gain you over simply posting it to a database table? Performance is why. The fire and forget aspect is like udp in the sense that you don't need to ensure ordering of messages (packets) or hard persistence to the database. Also, dead letter queues exist for a reason. Message queues are super useful. The highest performance systems I've seen use the message queue + pool of workers paradigm, as it allows you to better smooth your load (unlike immediate republishing like in sns, which requires hardware available to accept) with minimal guarantees (unlike a write ahead log such as kinesis). The buffer is also great because it allows you a bit more time to scale up both your message queue fleet and worker fleet when you get a load spike.

thephyber 3y ago
> > The first is that because of the nature of message queues, they're either empty or full. > ... Wat? I interpreted it as "they are either trending towards empty or full". The statement doesn't seem well thought through. That might be true (either empty or full) most of the time (maybe) _if you squint_, but the entire point of the Message Queue is to provide buffering from the transient state (somewhere between empty and full) trending toward the empty state.
- Raidion 3y ago
  Yea, I feel like these people never got a chron daily data dump (times N customers). Scaling is dead simple, and there isn't a need (or ability) to instantaneously handle large bursty workloads in like 99% of cases. Longer SLAs mean it's also easier to hit those SLAs. Giving the clients realistic SLAs is super important.

pronoiac 3y ago

If I'm thinking of the same article: was it about going from message queues to state machines for more operational robustness?

pronoiac 3y ago

You might be thinking of: https://blog.nelhage.com/post/systems-at-capacity/ https://blog.nelhage.com/post/systems-at-capacity/

Jasper_ 3y ago

Was it this one? https://ferd.ca/queues-don-t-fix-overload.html https://ferd.ca/queues-don-t-fix-overload.html

zbentley 3y ago

> If the request is Important, you certainly don't want to trust it to a message queue, a device not really designed for the storage of messages. That's false. Virtually all MQ systems are designed to persist (often with replication/redundancy) and store data. Most MQs also support non-persistent delivery, with the cost/benefit (ephemerality/performance) that entails, but that doesn't mean that durable storage is any less well-supported. Sure, folks have plenty of operational war stories regarding failures of persistence in their MQ broker/cluster/whatnot. Same as the DBAs who manage relational databases.