8 ms·

Can somebody help me better understand the possible uses for something like this?

by uptown 12y ago

hurin 12y ago
The author wrote a blogpost about it: http://antirez.com/news/88 http://antirez.com/news/88
dsugarman 12y ago
as a communication layer between modules or micro services or any distributed system. if you want to push global updates to a system, you can push a message out to a channel and all of the subscribers will get the message.
- uptown 12y ago
  Clarifies it perfectly for me. Thanks for the response.
antirez 12y ago
Let's say you have a web app that sends emails, and you have many worker processes actually sending emails. If you use something like Disque, instead of sending the email directly from the web app, you send a message to Disque, into the send_email queue: "Please send an email to foo@example.com". Workers sending emails will fetch from the send_email queue, and will get the message. Workers sending emails are supposed to acknowledged to Disque when they were actually able to process the message, so if they fail in some way, the message is re-queued and delivered again to the same or another worker, after a time you specify. So basically it adds a reliable middle-layer between the different moving parts of your application. The user will receive its email, or during failures it may receive the email multiple times, but it can never happen that no email is received by the user, since Disque guarantees that the message will be removed only once acknowledged.
- pyre 12y ago
  Other examples: - Queueing media files to be encoded/transcoded. - Trigger a build of your application from a web interface. Basically anything that you want to do where you might not want the web server to block until the entire task is done.
  - cdelsolar 12y ago
    Another one is making any sort of external web request; you should never use your web server to do this.
    - pyre 12y ago
      It depends on the use-case. You may want the request to your web server to wait for a response from your transaction processor (e.g. Stripe) before returning.
      - jonny_eh 12y ago
        That shouldn't be done in a web request. Not only can it lead to wasted time in the request handler, but it can lead to race conditions like double purchases.
        lumpypua 12y ago
        How do you implement "no really, this request should not return until the server has successfully completed external communication"? You can do some sort of websocket-y mechanism to tell the client that the server has completed a previous request, or polling, but both of those bring a lot of infrastructure overhead.
        jonny_eh 12y ago
        Polling is super easy. It's just one more API endpoint.
        12y ago
- jonny_eh 12y ago
  Isn't Redis already good for this, powering backgrounding software like Resque and Sidekiw? What does Disque offer that's different? BTW, thanks for Redis. It's one of my favourite pieces of software ever. Edit: NVM, found this: http://antirez.com/news/88 http://antirez.com/news/88
  - revskill 12y ago
    If your Redis died, you must have to saved the sent messages to process later. It's the most important fact about distributed message queue.
- uptown 12y ago
  Thanks very much for the explanation and the development. Fits nicely with a problem I've wanted to solve.
IgorPartola 12y ago
Personally, I look forward to using this instead of Redis and RabbitMQ as the message broker for Celery. Celery is a background tasks framework for Python. You write code that defines tasks, then certain things trigger tasks. Classic example: user uploads a video, then in the background the video is re-encoded in some standard format. So after the upload is done, the web application triggers a task by sending a message to the message broker (RabbitMQ, Redis, or now Disque). Why not just use RabbitMQ or Redis? Well, RabbitMQ is, in my experience, complex and fragile. It's got a ton of different features, which means you have to configure it to do anything beyond just the basics, and its management tools are somewhat lacking (why rabbitmqctl and rabbitmqadmin?). I recently started switching to Redis, because Redis is pretty much plug and play. It just works, and even minimal configuration that is necessary for this use case is very clear and simple. Moreover, it's got a very simple API for examining what's going on, no complex permissions management, vhosts, etc. It's only downside? It's not a message broker; it just has some of the right primitives to act like one. This is not to say that RabbitMQ or Redis are bad. They are great for what they do. I simply don't want to use them as the backend for Celery for the reasons stated above.
- misiti3780 12y ago
  "it's got a very simple API for examining what's going on" which api are you referring to specifically?
  - IgorPartola 12y ago
    INFO and LLEN. By contrast, here's my command to view the length of the queue with Rabbit: sudo rabbitmqctl -p example list_queues name messages messages_ready messages_unacknowledged consumers | grep 'celery \|other_celery ' Note that I need to use sudo to get this information here. I am sure the Management plug could help here. I do have Flower set up, but it gets terribly confused about multiple queues and multiple task servers and sets of workers, so it's basically useless. It seems to latch onto one of the queues and only shows the stats for that in most of its views.
- lobster_johnson 12y ago
rwmj 12y ago
Message queues are "email for applications". They allow you to break up your application into components that talk to each other across processes, machines and time. For example if you need to print and send a letter to your customer, you can have the web frontend send a message to the printer task. Those messages can be queued up (in case the printer is down), and handled slowly or quickly by the printer. If you have two printers you can have them both pulling jobs off the same queue. Good message queues can offer reliability guarantees like at most once delivery, and persistence (saving message queues to disk in case machines go down). Also primitives like pub/sub where applications subscribe to data (think: stock feeds), and other applications publish that data.
paulsutter 12y ago
Simple use cases for message queues in a web application include: - Improving response latency by moving deferrable tasks out of the page serving code - Improving response latency by simplifying implementation of concurrent tasks (whether across cores or servers) - Allowing capacity planning for average load instead of peak load for deferrable tasks The bigger story is that message queues make a good substrate for building fault-tolerant, highly available, scalable distributed systems that are easy to reason about and have predictable performance. This is because system components can be cleanly decoupled, implemented with redundancy and scalability, the message queue itself takes over much of the complexity of failover and delivery/ordering guarantees, and the message queue makes an excellent point for monitoring the system, giving visibility into logjams and problems in a live system.

Can somebody help me better understand the possible uses for something like this?

by uptown 12y ago

Can somebody help me better understand the possible uses for something like this?

hurin 12y ago

The author wrote a blogpost about it: http://antirez.com/news/88 http://antirez.com/news/88

dsugarman 12y ago

as a communication layer between modules or micro services or any distributed system. if you want to push global updates to a system, you can push a message out to a channel and all of the subscribers will get the message.

uptown 12y ago
Clarifies it perfectly for me. Thanks for the response.

antirez 12y ago

Let's say you have a web app that sends emails, and you have many worker processes actually sending emails. If you use something like Disque, instead of sending the email directly from the web app, you send a message to Disque, into the send_email queue: "Please send an email to foo@example.com". Workers sending emails will fetch from the send_email queue, and will get the message. Workers sending emails are supposed to acknowledged to Disque when they were actually able to process the message, so if they fail in some way, the message is re-queued and delivered again to the same or another worker, after a time you specify. So basically it adds a reliable middle-layer between the different moving parts of your application. The user will receive its email, or during failures it may receive the email multiple times, but it can never happen that no email is received by the user, since Disque guarantees that the message will be removed only once acknowledged.

pyre 12y ago
Other examples: - Queueing media files to be encoded/transcoded. - Trigger a build of your application from a web interface. Basically anything that you want to do where you might not want the web server to block until the entire task is done.
- cdelsolar 12y ago
  Another one is making any sort of external web request; you should never use your web server to do this.
  - pyre 12y ago
    It depends on the use-case. You may want the request to your web server to wait for a response from your transaction processor (e.g. Stripe) before returning.
    - jonny_eh 12y ago
      That shouldn't be done in a web request. Not only can it lead to wasted time in the request handler, but it can lead to race conditions like double purchases.
      - lumpypua 12y ago
        How do you implement "no really, this request should not return until the server has successfully completed external communication"? You can do some sort of websocket-y mechanism to tell the client that the server has completed a previous request, or polling, but both of those bring a lot of infrastructure overhead.
        jonny_eh 12y ago
        Polling is super easy. It's just one more API endpoint.
        12y ago
jonny_eh 12y ago
Isn't Redis already good for this, powering backgrounding software like Resque and Sidekiw? What does Disque offer that's different? BTW, thanks for Redis. It's one of my favourite pieces of software ever. Edit: NVM, found this: http://antirez.com/news/88 http://antirez.com/news/88
- revskill 12y ago
  If your Redis died, you must have to saved the sent messages to process later. It's the most important fact about distributed message queue.
uptown 12y ago
Thanks very much for the explanation and the development. Fits nicely with a problem I've wanted to solve.

IgorPartola 12y ago

Personally, I look forward to using this instead of Redis and RabbitMQ as the message broker for Celery. Celery is a background tasks framework for Python. You write code that defines tasks, then certain things trigger tasks. Classic example: user uploads a video, then in the background the video is re-encoded in some standard format. So after the upload is done, the web application triggers a task by sending a message to the message broker (RabbitMQ, Redis, or now Disque). Why not just use RabbitMQ or Redis? Well, RabbitMQ is, in my experience, complex and fragile. It's got a ton of different features, which means you have to configure it to do anything beyond just the basics, and its management tools are somewhat lacking (why rabbitmqctl and rabbitmqadmin?). I recently started switching to Redis, because Redis is pretty much plug and play. It just works, and even minimal configuration that is necessary for this use case is very clear and simple. Moreover, it's got a very simple API for examining what's going on, no complex permissions management, vhosts, etc. It's only downside? It's not a message broker; it just has some of the right primitives to act like one. This is not to say that RabbitMQ or Redis are bad. They are great for what they do. I simply don't want to use them as the backend for Celery for the reasons stated above.

misiti3780 12y ago
"it's got a very simple API for examining what's going on" which api are you referring to specifically?
- IgorPartola 12y ago
  INFO and LLEN. By contrast, here's my command to view the length of the queue with Rabbit: sudo rabbitmqctl -p example list_queues name messages messages_ready messages_unacknowledged consumers | grep 'celery \|other_celery ' Note that I need to use sudo to get this information here. I am sure the Management plug could help here. I do have Flower set up, but it gets terribly confused about multiple queues and multiple task servers and sets of workers, so it's basically useless. It seems to latch onto one of the queues and only shows the stats for that in most of its views.
lobster_johnson 12y ago

rwmj 12y ago

Message queues are "email for applications". They allow you to break up your application into components that talk to each other across processes, machines and time. For example if you need to print and send a letter to your customer, you can have the web frontend send a message to the printer task. Those messages can be queued up (in case the printer is down), and handled slowly or quickly by the printer. If you have two printers you can have them both pulling jobs off the same queue. Good message queues can offer reliability guarantees like at most once delivery, and persistence (saving message queues to disk in case machines go down). Also primitives like pub/sub where applications subscribe to data (think: stock feeds), and other applications publish that data.

paulsutter 12y ago

Simple use cases for message queues in a web application include: - Improving response latency by moving deferrable tasks out of the page serving code - Improving response latency by simplifying implementation of concurrent tasks (whether across cores or servers) - Allowing capacity planning for average load instead of peak load for deferrable tasks The bigger story is that message queues make a good substrate for building fault-tolerant, highly available, scalable distributed systems that are easy to reason about and have predictable performance. This is because system components can be cleanly decoupled, implemented with redundancy and scalability, the message queue itself takes over much of the complexity of failover and delivery/ordering guarantees, and the message queue makes an excellent point for monitoring the system, giving visibility into logjams and problems in a live system.