4 ms·
Let's say you have a web app that sends emails, and you have many worker processes actually sending emails. If you use something like Disque, instead of sending
by antirez 12y ago
Let's say you have a web app that sends emails, and you have many worker processes actually sending emails. If you use something like Disque, instead of sending the email directly from the web app, you send a message to Disque, into the send_email queue: "Please send an email to foo@example.com". Workers sending emails will fetch from the send_email queue, and will get the message. Workers sending emails are supposed to acknowledged to Disque when they were actually able to process the message, so if they fail in some way, the message is re-queued and delivered again to the same or another worker, after a time you specify. So basically it adds a reliable middle-layer between the different moving parts of your application. The user will receive its email, or during failures it may receive the email multiple times, but it can never happen that no email is received by the user, since Disque guarantees that the message will be removed only once acknowledged.
- pyre 12y agoOther examples: - Queueing media files to be encoded/transcoded. - Trigger a build of your application from a web interface. Basically anything that you want to do where you might not want the web server to block until the entire task is done.
- cdelsolar 12y agoAnother one is making any sort of external web request; you should never use your web server to do this.
- pyre 12y agoIt depends on the use-case. You may want the request to your web server to wait for a response from your transaction processor (e.g. Stripe) before returning.
- jonny_eh 12y agoThat shouldn't be done in a web request. Not only can it lead to wasted time in the request handler, but it can lead to race conditions like double purchases.
- lumpypua 12y agoHow do you implement "no really, this request should not return until the server has successfully completed external communication"? You can do some sort of websocket-y mechanism to tell the client that the server has completed a previous request, or polling, but both of those bring a lot of infrastructure overhead.
- jonny_eh 12y agoPolling is super easy. It's just one more API endpoint.
- StavrosK 12y agoBlocking is even easier, it's zero more endpoints.
- cdelsolar 12y agoYeah and if Stripe goes down your server goes down with it.
- StavrosK 12y agoHow are you architecting things that an upstream error will cause your server to die? Look, in both cases, the request needs to be synchronous. You need to tell the customer that their purchase went through. If you're going to add a whole other bus into the thing and make it asynchronous and have an elaborate polling solution on the client, that's fine, but I would (and do) just have them wait the two extra seconds and either say "your card has been charged" or handle the rare case with "there has been an upstream error, please don't close this window while we retry the request".
- jonny_eh 12y agoIsn't Redis already good for this, powering backgrounding software like Resque and Sidekiw? What does Disque offer that's different? BTW, thanks for Redis. It's one of my favourite pieces of software ever. Edit: NVM, found this: http://antirez.com/news/88 http://antirez.com/news/88
- revskill 12y agoIf your Redis died, you must have to saved the sent messages to process later. It's the most important fact about distributed message queue.
- uptown 12y agoThanks very much for the explanation and the development. Fits nicely with a problem I've wanted to solve.