5 ms·

Genuine question from someone who doesn't know any better: what's the advantage of having a message queue like this (service publishes message to queue, recipie

Genuine question from someone who doesn't know any better: what's the advantage of having a message queue like this (service publishes message to queue, recipient gets notified about it, responds to payload) vs just sending an HTTP request directly to the recipient?

avmich 4y ago
Queues are used to be able to have bursts of requests without overloading the server, which may be less performant than needed to service those bursts of requests directly. With the message bus the clients can (roughly) send requests as frequently as they want, while servers will handle them as fast as they can, without the danger of missing a request or having clients to wait. The queue is the "asynchronization" mechanism.
- deleted 4y ago
  [deleted]
vladvasiliu 4y ago
Not an expert in this domain, but off the top of my head: It's easier to handle certain situations, like if you have multiple recipients, you may want to notify all of them, at least one, at most one, etc. It's also a way of handling consumers that may be unavailable for whatever reason and your source may not want to have to deal with that. For example, if the producer is a cron job, you may not want it to have to hang around for an arbitrary time if your consumers are already busy. Sure, this means that the queue is available, but in principle it's supposed to be more available than the consumers.
- brundolf 4y ago
  Fair enough I guess, though in that case my mind goes to just having the recipient queue its own tasks internally and respond immediately, if the sender doesn't want to wait on it to finish
  - nightpool 4y ago
    Which we know from the article that the recipient in this case was ALREADY doing. They were using RabbitMQ as a glorified cron job to poll the "status" of the other service every 5 minutes.
  - vladvasiliu 4y ago
    Sure you can do that, but I'd say in the general case "it depends". If your needs are fairly basic, that's probably the best approach. You also eliminate some overhead (piping messages through a middle man, operating said middle man, etc.). But it's my understanding that if your needs are a bit more involved, the complexities of building a solid queue mean that it'll take time away from building your actual product. In that case, you may be better served by an off-the-shelf solution which already handles the corner cases and is good to go. I think this is the usual decision between using an external product and rolling your own.
  - jojok 4y ago
    If there are multiple recipients then they may need to each maintain their own queues. But these queues may implicitly have an expectation of being in sync with each other. If you use a centralized message queue then there will be a single source of truth about what messages were received and in what order. This can help to avoid or debug problems that can arise if sender fails to send request to some receivers or if some receivers fail to process requests.
virtual_void 4y ago
With your http example you have to know who the recipient is. If there are multiple recipients then you’d have to call each one. With a queue you don’t need to know, and the recipients can subscribe to information on the queue themselves. Especially useful in event-driven architectures. Push vs pull.
- brundolf 4y ago
  I guess. Maybe I'm not thinking on the right scale of company, where there may be many services run by many teams that don't talk to each other. In that case I could see decoupling between senders and consumers being helpful
  - virtual_void 4y ago
    It’s not just about scale though, it can be useful where you own both components. The producer just produces events, as long as the queue is available. no retry is needed if the other side is having a problem or being updated. The messages sit in the queue until the other components can consume them. Especially useful when producer and consumers are running at different rates and the queue becomes a buffer. There’s also a difference in making a command to another service vs event or state notification. In the case of a command you often want to wait and find out if it went ok. Events or states published onto a queue are more of a case of “here is a thing that happened, deal with it how you wish”
  - xorcist 4y ago
    Scale isn't the point. It's the nature of event based systems. Why do we have interrupts? Having an abstraction there makes developing operating systems easier. Why do we have hooks in git? Because we want to add or remove software integrations at run time, without touching git itself. Queue based systems have similar characteristics as event based systems. It's not a bad idea at all, you probably use message based systems every day without thinking about it, culturally however the idea is irresistible for enterprise developers who live for abstractions. Just something to keep in mind.
silisili 4y ago
Two reasons, mainly. First, it handles ebbs and flows in requests much more gracefully. It's cheap to throw everything on a queue, and handle it as the system allows. Secondly, it's decoupled from your application. Trying to do things directly works for a while, then you do an in memory array, then your application crashes or gets updated and lost those messages.
codedokode 4y ago
1) Some tasks may take a long time to complete. For example, generating a report or converting a video file might take several minutes. You don't want a user to look at a blank page for so long, and browser might even close the connection because of timeout. 2) Reliability. Imagine, if a customer places an order in a store, but the server process crashes and the order is lost. The customer won't be happy about this. A mesasge queue might help in both cases. Note that you might implement a queue in a SQL database as well, but this still will be a message queue.
- TedDoesntTalk 4y ago
  And the downside is the http request can never have the full response. Client must somehow make another request to get results.
jimmyechan 4y ago
I thought I could instead describe practical use cases we have for message queues (and task queues) at our company. For context, we work on a product that helps people validate, clean, and import their CSVs/Excel files into SQL databases. We have two main use cases: First, we use them to allow app users to run (large) data processing tasks without keeping connections open for a long time. Basically letting our users start a data processing job on the client side and then let them continue doing something else, and not forced to stay until in the same page until the data processing job is completed. We check for status updates in the database; once our data processing job is complete, it updates the status of the job. Second, in the backend, we have multiple tasks (compute resources) running that are ready to take on these data processing requests based on messages in the queue. To allow for bursts of user requests for data processing, we make available multiple tasks whose job it is to run the data processing jobs. We place data processing request messages in the queue, which are then picked up by any of the available (non busy) data processing tasks. With a local/internal queue you suggested, we’d have a harder time figuring out and ensuring the right tasks picks up the data processing job, and that only one tasks does so to avoid duplicate effort. Or we might have to have local queues on each task, and somehow coordinate them. With the centralized queue, we don’t have to worry about this - whichever tasks is available picks up the message and runs the data processing job. When the tasks takes the message it pops it from the message queue and no other task picks up the same message (unless we want it to for some other reasons).
mst 4y ago
My favourite reason is that I can move retry logic into the thing sending the HTTP request and my front end web handler thing can be significantly less stateful as a result, which makes restarting it a lot less annoying etc. etc. I'd probably have started out with -just- the outbox though and bolted the HTTP request sending onto the database, then looked at switching to a dedicated queueing system later. But that's a "because I already understand the care and feeding of PostgreSQL" choice as much as anything else.
baq 4y ago
for a start - when you can't afford your requests to time out because there's so many of them but can afford to process them later and/or in bulk. then you notice that you can do much more than that, but most people start here.
Gordonjcp 4y ago
I want to send you an email from a web page, for example a "Click this link to complete your registration" type thing. Your mailserver is paranoid (and rightly so) and responds very slowly to greylisted mailservers like mine that it has never heard from before, so it takes about a minute to complete the message. The webpage I wrote twenty years ago would accept your POST request, wrap up everything into the email, send it, return a redirect to a GET to the same page, and finally send you a page saying "Thanks! I've sent you an email with a link to click on to confirm it's really you." You would not get this web page because your browser would have timed out, or maybe you would but you'd think "Jeez this site is so slow" not realising that it's just waiting for you. The webpage I wrote ten years ago would accept your POST request, wrap up everything into an email, send it to a message queue, and so on except maybe now I'd say "I am sending you" rather than "I have sent you", which you'd see immediately. Around the same time, a queue worker notices a new email in the queue, sends it, hangs around for a minute or so waiting for your server. You get the email, click the link, wow what a nice fast responsive site! The webpage I wrote last year would do much the same thing, but the queue worker might flip a flag in your account to say the email has been sent but the link has not been clicked. Meanwhile a wee bit of javascript is polling an endpoint watching for the flag, and flips the text round to say "Check your email, it's been sent".
rawgabbit 4y ago
It's about ensuring the message arrives. If you are creating a video game, who cares if a message gets lost. If $money is involved, you absolutely have to ensure the message arrives, even if destination server is dead/being updated/being attacked by hackers etc. With Kafka or enterprise service bus, the promise is when the destination server comes back online, it can replay or reprocess the events it missed.

avmich 4y ago

Queues are used to be able to have bursts of requests without overloading the server, which may be less performant than needed to service those bursts of requests directly. With the message bus the clients can (roughly) send requests as frequently as they want, while servers will handle them as fast as they can, without the danger of missing a request or having clients to wait. The queue is the "asynchronization" mechanism.

deleted 4y ago
[deleted]

vladvasiliu 4y ago

Not an expert in this domain, but off the top of my head: It's easier to handle certain situations, like if you have multiple recipients, you may want to notify all of them, at least one, at most one, etc. It's also a way of handling consumers that may be unavailable for whatever reason and your source may not want to have to deal with that. For example, if the producer is a cron job, you may not want it to have to hang around for an arbitrary time if your consumers are already busy. Sure, this means that the queue is available, but in principle it's supposed to be more available than the consumers.

brundolf 4y ago
Fair enough I guess, though in that case my mind goes to just having the recipient queue its own tasks internally and respond immediately, if the sender doesn't want to wait on it to finish
- nightpool 4y ago
  Which we know from the article that the recipient in this case was ALREADY doing. They were using RabbitMQ as a glorified cron job to poll the "status" of the other service every 5 minutes.
- vladvasiliu 4y ago
  Sure you can do that, but I'd say in the general case "it depends". If your needs are fairly basic, that's probably the best approach. You also eliminate some overhead (piping messages through a middle man, operating said middle man, etc.). But it's my understanding that if your needs are a bit more involved, the complexities of building a solid queue mean that it'll take time away from building your actual product. In that case, you may be better served by an off-the-shelf solution which already handles the corner cases and is good to go. I think this is the usual decision between using an external product and rolling your own.
- jojok 4y ago
  If there are multiple recipients then they may need to each maintain their own queues. But these queues may implicitly have an expectation of being in sync with each other. If you use a centralized message queue then there will be a single source of truth about what messages were received and in what order. This can help to avoid or debug problems that can arise if sender fails to send request to some receivers or if some receivers fail to process requests.

virtual_void 4y ago

With your http example you have to know who the recipient is. If there are multiple recipients then you’d have to call each one. With a queue you don’t need to know, and the recipients can subscribe to information on the queue themselves. Especially useful in event-driven architectures. Push vs pull.

brundolf 4y ago
I guess. Maybe I'm not thinking on the right scale of company, where there may be many services run by many teams that don't talk to each other. In that case I could see decoupling between senders and consumers being helpful
- virtual_void 4y ago
  It’s not just about scale though, it can be useful where you own both components. The producer just produces events, as long as the queue is available. no retry is needed if the other side is having a problem or being updated. The messages sit in the queue until the other components can consume them. Especially useful when producer and consumers are running at different rates and the queue becomes a buffer. There’s also a difference in making a command to another service vs event or state notification. In the case of a command you often want to wait and find out if it went ok. Events or states published onto a queue are more of a case of “here is a thing that happened, deal with it how you wish”
- xorcist 4y ago
  Scale isn't the point. It's the nature of event based systems. Why do we have interrupts? Having an abstraction there makes developing operating systems easier. Why do we have hooks in git? Because we want to add or remove software integrations at run time, without touching git itself. Queue based systems have similar characteristics as event based systems. It's not a bad idea at all, you probably use message based systems every day without thinking about it, culturally however the idea is irresistible for enterprise developers who live for abstractions. Just something to keep in mind.

silisili 4y ago

Two reasons, mainly. First, it handles ebbs and flows in requests much more gracefully. It's cheap to throw everything on a queue, and handle it as the system allows. Secondly, it's decoupled from your application. Trying to do things directly works for a while, then you do an in memory array, then your application crashes or gets updated and lost those messages.

codedokode 4y ago

1) Some tasks may take a long time to complete. For example, generating a report or converting a video file might take several minutes. You don't want a user to look at a blank page for so long, and browser might even close the connection because of timeout. 2) Reliability. Imagine, if a customer places an order in a store, but the server process crashes and the order is lost. The customer won't be happy about this. A mesasge queue might help in both cases. Note that you might implement a queue in a SQL database as well, but this still will be a message queue.

TedDoesntTalk 4y ago
And the downside is the http request can never have the full response. Client must somehow make another request to get results.

jimmyechan 4y ago

I thought I could instead describe practical use cases we have for message queues (and task queues) at our company. For context, we work on a product that helps people validate, clean, and import their CSVs/Excel files into SQL databases. We have two main use cases: First, we use them to allow app users to run (large) data processing tasks without keeping connections open for a long time. Basically letting our users start a data processing job on the client side and then let them continue doing something else, and not forced to stay until in the same page until the data processing job is completed. We check for status updates in the database; once our data processing job is complete, it updates the status of the job. Second, in the backend, we have multiple tasks (compute resources) running that are ready to take on these data processing requests based on messages in the queue. To allow for bursts of user requests for data processing, we make available multiple tasks whose job it is to run the data processing jobs. We place data processing request messages in the queue, which are then picked up by any of the available (non busy) data processing tasks. With a local/internal queue you suggested, we’d have a harder time figuring out and ensuring the right tasks picks up the data processing job, and that only one tasks does so to avoid duplicate effort. Or we might have to have local queues on each task, and somehow coordinate them. With the centralized queue, we don’t have to worry about this - whichever tasks is available picks up the message and runs the data processing job. When the tasks takes the message it pops it from the message queue and no other task picks up the same message (unless we want it to for some other reasons).

mst 4y ago

My favourite reason is that I can move retry logic into the thing sending the HTTP request and my front end web handler thing can be significantly less stateful as a result, which makes restarting it a lot less annoying etc. etc. I'd probably have started out with -just- the outbox though and bolted the HTTP request sending onto the database, then looked at switching to a dedicated queueing system later. But that's a "because I already understand the care and feeding of PostgreSQL" choice as much as anything else.

baq 4y ago

for a start - when you can't afford your requests to time out because there's so many of them but can afford to process them later and/or in bulk. then you notice that you can do much more than that, but most people start here.

Gordonjcp 4y ago

I want to send you an email from a web page, for example a "Click this link to complete your registration" type thing. Your mailserver is paranoid (and rightly so) and responds very slowly to greylisted mailservers like mine that it has never heard from before, so it takes about a minute to complete the message. The webpage I wrote twenty years ago would accept your POST request, wrap up everything into the email, send it, return a redirect to a GET to the same page, and finally send you a page saying "Thanks! I've sent you an email with a link to click on to confirm it's really you." You would not get this web page because your browser would have timed out, or maybe you would but you'd think "Jeez this site is so slow" not realising that it's just waiting for you. The webpage I wrote ten years ago would accept your POST request, wrap up everything into an email, send it to a message queue, and so on except maybe now I'd say "I am sending you" rather than "I have sent you", which you'd see immediately. Around the same time, a queue worker notices a new email in the queue, sends it, hangs around for a minute or so waiting for your server. You get the email, click the link, wow what a nice fast responsive site! The webpage I wrote last year would do much the same thing, but the queue worker might flip a flag in your account to say the email has been sent but the link has not been clicked. Meanwhile a wee bit of javascript is polling an endpoint watching for the flag, and flips the text round to say "Check your email, it's been sent".

rawgabbit 4y ago

It's about ensuring the message arrives. If you are creating a video game, who cares if a message gets lost. If $money is involved, you absolutely have to ensure the message arrives, even if destination server is dead/being updated/being attacked by hackers etc. With Kafka or enterprise service bus, the promise is when the destination server comes back online, it can replay or reprocess the events it missed.