4 ms·

Bad take. Most of the different behaviors you criticize about message queue systems arise from them needing to a) distribute work b) durably, c) over the networ

Bad take. Most of the different behaviors you criticize about message queue systems arise from them needing to a) distribute work b) durably, c) over the network. Most message queues want to be able to distribute work to multiple parallel consumers, either for performance, redundancy, re-deployment of the consumers, and so on. Like, sure, you can do that with named pipes, but they're not durable; their state can be lost in the event of a crash or reboot. Other than like ... in-process, thread-to-thread message queues, almost all message queues are used because they provide some form of durability (either by persisting messages to disk, or just by virtue of running on a separate server/process than much more crash-prone producers/consumers). > imperatively, rather than one end being imperative and the other being a declarative graph of callbacks (all those “reactive” libraries). MQ client libraries are heavily reactive because they need to work with MQs that are network services. If every consumer that ever wanted to fetch a message had to issue a network RPC poll/timeout cycle, and they were all doing that constantly, that'd be both a lot of traffic for the MQ to handle and a lot of network chatter. Some MQs do use (or support) the RPC poll model, but especially among the more performant ones, most of them are push-based so clients can just sit in an epoll/select statement waiting for the message queue to send them traffic. That said, you're totally right that plenty of MQ client frameworks/libraries are way over complicated and want to own your whole program's entry point and design. Callbacks are probably a necessary outgrowth of the underlying network behavior, but tortured whole-program-event-loop-ownership architecture is not. > Even using a term like “back pressure” is a tell to me that someone is confused snd doing something architecturally wrong. I don't think your other points support this; you said yourself that pipes' bounded nature is good, so what's the problem here?

groundzeros2015 3mo ago
I just talked about pipes as a useful conceptual tool. You’re assuming what applications I think they apply to. But I’ll do my best. One of my takeaways is that if you do not handle on data transfer in the http request/response then you are choosing to exit the pipe model. Once you do that, yes all kinds of crazy async problems exist which require very complex tools to wrangle. See the sibiling comment from Google engineer. It’s better to do work synchronously than delay it until later. So now we get your list of requirements that MQ is for. Ok but how did we get here? What problem are we trying to solve? The answer is usually a performance problem for which this is a bandaid. Another possible model for this problem is email. You can find some great threads here about large amounts of infrastructure being replaced with Unix email tools. Although I have yet to try it! > If every consumer that ever wanted to fetch a message had to issue a network RPC poll/timeout cycle, and they were all doing that constantly, that'd be both a lot of traffic for the MQ to handle and a lot of network chatter This makes no sense to me. A while loop blocked on a socket read with a producer with a write does not generate any extra network “chatter”. Anytime you are using callbacks you are choosing not to have a thread with imperative logic. > I don't think your other points support this Correct that was not a conclusion. What I’m trying to say is if you want to send data between two systems and you have producers and consumers at different rates, you don’t have to do anything special. The UNIX kernel is designed to solve this problem. So when I hear that, my inclination is that we don’t understand the capabilities UNIX already offers and I’ve yet to be wrong.
- zbentley 3mo ago
  > The answer is usually a performance problem for which this is a bandaid. I agree that if that's the problem you're trying to solve, MQs are a poor stop-gap and not a general-purpose solution. But many (most?) MQs aren't deployed for that reason; rather, they're used to either a) defer work whose latency characteristics are incompatible with the producer's runtime (e.g. sending email which might take minutes/hours to be accepted upstream can't be done synchronously from an HTTP request handler), b) handle work which doesn't need to be done transactionally and which has a high probability of needing to connect to resources whose uptime you don't control (queues make retries easy), or c) work that needs to be batched or otherwise completed on a schedule or dimensionality that's not available in the producer. > A while loop blocked on a socket read with a producer with a write does not generate any extra network “chatter”. True, and some MQs support that pattern, but it's rare for the same reason that most network server applications' core loop doesn't wrap a blocking read. While-blocking-read plays poorly with timeouts, restarts, and multiplexing/wait-any. This isn't unique to MQs. > Anytime you are using callbacks you are choosing not to have a thread with imperative logic. I mean, it's pretty easy in most languages/runtimes to turn a callback into an imperative wait. When it comes to network servers specifically, I think exposing the lowest-runtime-overhead model as the primary API (selectors and callbacks) and letting users add control flow primitives on top of that is preferable, but reasonable minds can differ here. This also isn't unique to MQs. > if you want to send data between two systems and you have producers and consumers at different rates, you don’t have to do anything special. The UNIX kernel is designed to solve this problem. For ephemeral/can-be-lost local traffic, you're partly right (pipes/fifos are a simple and widely available tool that can help with this, though you have to layer some protocol-ish logic on top to make them work reliably between consumers that expect particular structures and don't speak ASCII/newlines alone). For durable local traffic, or remote traffic, UNIX doesn't have a prebuilt primitive. TCP tx/rx window sizes aren't surfaced such that buffers and backpressure can be reliably interpreted by userland to make delivery decisions.
  - groundzeros2015 3mo ago
    Thanks for high quality reply. > latency characteristics are incompatible with the producer's runtime > handle work which doesn't need to be done transactionally and which has a high probability of needing to connect to resources whose uptime you don't control (queues make retries easy I agree. I think you do need to break the request/response pipe for this use case. I traditionally have treated this as a state in a database. I know there are limitations and I can see MQ being a solution. I want to try the email idea sometime. I do think queues are a poor technology for dealing with performance problems in a web stack. > This also isn't unique to MQs. Agreed. My ranting about async and callbacks is an ongoing project. I think pipes solve producer/consumer problems in a way most engineers don’t appreciate. > While-blocking-read plays poorly with timeouts, restarts, and multiplexing/wait-any. The primary problem solved by not using blocking primitives is to try to free up OS resources from threads (green threading). Why would an internal queue have that problem? It’s not accepting arbitrary connections.

groundzeros2015 3mo ago

I just talked about pipes as a useful conceptual tool. You’re assuming what applications I think they apply to. But I’ll do my best. One of my takeaways is that if you do not handle on data transfer in the http request/response then you are choosing to exit the pipe model. Once you do that, yes all kinds of crazy async problems exist which require very complex tools to wrangle. See the sibiling comment from Google engineer. It’s better to do work synchronously than delay it until later. So now we get your list of requirements that MQ is for. Ok but how did we get here? What problem are we trying to solve? The answer is usually a performance problem for which this is a bandaid. Another possible model for this problem is email. You can find some great threads here about large amounts of infrastructure being replaced with Unix email tools. Although I have yet to try it! > If every consumer that ever wanted to fetch a message had to issue a network RPC poll/timeout cycle, and they were all doing that constantly, that'd be both a lot of traffic for the MQ to handle and a lot of network chatter This makes no sense to me. A while loop blocked on a socket read with a producer with a write does not generate any extra network “chatter”. Anytime you are using callbacks you are choosing not to have a thread with imperative logic. > I don't think your other points support this Correct that was not a conclusion. What I’m trying to say is if you want to send data between two systems and you have producers and consumers at different rates, you don’t have to do anything special. The UNIX kernel is designed to solve this problem. So when I hear that, my inclination is that we don’t understand the capabilities UNIX already offers and I’ve yet to be wrong.

zbentley 3mo ago
> The answer is usually a performance problem for which this is a bandaid. I agree that if that's the problem you're trying to solve, MQs are a poor stop-gap and not a general-purpose solution. But many (most?) MQs aren't deployed for that reason; rather, they're used to either a) defer work whose latency characteristics are incompatible with the producer's runtime (e.g. sending email which might take minutes/hours to be accepted upstream can't be done synchronously from an HTTP request handler), b) handle work which doesn't need to be done transactionally and which has a high probability of needing to connect to resources whose uptime you don't control (queues make retries easy), or c) work that needs to be batched or otherwise completed on a schedule or dimensionality that's not available in the producer. > A while loop blocked on a socket read with a producer with a write does not generate any extra network “chatter”. True, and some MQs support that pattern, but it's rare for the same reason that most network server applications' core loop doesn't wrap a blocking read. While-blocking-read plays poorly with timeouts, restarts, and multiplexing/wait-any. This isn't unique to MQs. > Anytime you are using callbacks you are choosing not to have a thread with imperative logic. I mean, it's pretty easy in most languages/runtimes to turn a callback into an imperative wait. When it comes to network servers specifically, I think exposing the lowest-runtime-overhead model as the primary API (selectors and callbacks) and letting users add control flow primitives on top of that is preferable, but reasonable minds can differ here. This also isn't unique to MQs. > if you want to send data between two systems and you have producers and consumers at different rates, you don’t have to do anything special. The UNIX kernel is designed to solve this problem. For ephemeral/can-be-lost local traffic, you're partly right (pipes/fifos are a simple and widely available tool that can help with this, though you have to layer some protocol-ish logic on top to make them work reliably between consumers that expect particular structures and don't speak ASCII/newlines alone). For durable local traffic, or remote traffic, UNIX doesn't have a prebuilt primitive. TCP tx/rx window sizes aren't surfaced such that buffers and backpressure can be reliably interpreted by userland to make delivery decisions.
- groundzeros2015 3mo ago
  Thanks for high quality reply. > latency characteristics are incompatible with the producer's runtime > handle work which doesn't need to be done transactionally and which has a high probability of needing to connect to resources whose uptime you don't control (queues make retries easy I agree. I think you do need to break the request/response pipe for this use case. I traditionally have treated this as a state in a database. I know there are limitations and I can see MQ being a solution. I want to try the email idea sometime. I do think queues are a poor technology for dealing with performance problems in a web stack. > This also isn't unique to MQs. Agreed. My ranting about async and callbacks is an ongoing project. I think pipes solve producer/consumer problems in a way most engineers don’t appreciate. > While-blocking-read plays poorly with timeouts, restarts, and multiplexing/wait-any. The primary problem solved by not using blocking primitives is to try to free up OS resources from threads (green threading). Why would an internal queue have that problem? It’s not accepting arbitrary connections.