4 ms·
I thought I could instead describe practical use cases we have for message queues (and task queues) at our company. For context, we work on a product that helps
by jimmyechan 4y ago
I thought I could instead describe practical use cases we have for message queues (and task queues) at our company. For context, we work on a product that helps people validate, clean, and import their CSVs/Excel files into SQL databases. We have two main use cases:
First, we use them to allow app users to run (large) data processing tasks without keeping connections open for a long time. Basically letting our users start a data processing job on the client side and then let them continue doing something else, and not forced to stay until in the same page until the data processing job is completed. We check for status updates in the database; once our data processing job is complete, it updates the status of the job.
Second, in the backend, we have multiple tasks (compute resources) running that are ready to take on these data processing requests based on messages in the queue. To allow for bursts of user requests for data processing, we make available multiple tasks whose job it is to run the data processing jobs. We place data processing request messages in the queue, which are then picked up by any of the available (non busy) data processing tasks. With a local/internal queue you suggested, we’d have a harder time figuring out and ensuring the right tasks picks up the data processing job, and that only one tasks does so to avoid duplicate effort. Or we might have to have local queues on each task, and somehow coordinate them. With the centralized queue, we don’t have to worry about this - whichever tasks is available picks up the message and runs the data processing job. When the tasks takes the message it pops it from the message queue and no other task picks up the same message (unless we want it to for some other reasons).