3 ms·

You’re spot on really. Having the front end wait for a background task to complete broadly defeats the purpose. There are some caveats though: if you’re using a

You’re spot on really. Having the front end wait for a background task to complete broadly defeats the purpose. There are some caveats though: if you’re using an async/threaded web server then it may not matter that you have a pending request hanging around as your web server is free to continue serving other requests. It also may be that you need to run the task on specialist hardware for some reason. Really though, I think a lot of people use celery for offloading things like email sending and API calls which, IMHO, isn’t really worth the complexity (especially as SMTP is basically a queue anyway). Of course, YMMV depending on your use case. However, I find it is often more worthwhile for: 1. Tasks which take a long time to run 2. Tasks which need to happen on a schedule, rather than in response to a user request. There is an option 3 too, which is for inter-software communication. Eg events or RPCs, but I found Celery to be very much a square-peg-round-hole for this, which is why I developed Lightbus (http://lightbus.org http://lightbus.org). Lightbus also supports background tasks and scheduled tasks. /plug

doliveira 7y ago
It's not just about the time the operation takes, it's about reliability. Even if sending an email synchronously doesn't usually take more than a few milliseconds, you still need to handle cases like servers failing in the middle of the request, temporary upstream unavailability, some expired API, account limits reached, etc... Honestly, I think that mostly anything that doesn't depend directly in the current state of your infrastructure should be done asynchronously. I've had a lot of issues with systems that start up doing everything synchronously: you'll probably need to refactor it to be asynchronous in emergency mode during a crisis.
- marcosdumay 7y ago
  For email, that's why you set a relay within your control, that will accept messages without a fuss and send them around following SMTP conventions. But anyway, how is your application supposed to respond after any of those failures? Is it just supposed to ignore the failure and thread on like if nothing happened, leaving your users on the dark? Is it supposed to reliably log every task so that it can retry anything that fails and in the worst case feed failures into some monitoring system/process? Or is it supposed to inform the user of any success or failure before the user can move on? Queue software is only a good match for the first. For the second you will need to roll your own interface with the monitoring system anyway, so it's much easier to roll your own queues and get control of everything. The third one is best done synchronous, it doesn't matter the nature of the process or how long it takes. But funny thing is, I have never seen the first situation on the wild.

doliveira 7y ago

It's not just about the time the operation takes, it's about reliability. Even if sending an email synchronously doesn't usually take more than a few milliseconds, you still need to handle cases like servers failing in the middle of the request, temporary upstream unavailability, some expired API, account limits reached, etc... Honestly, I think that mostly anything that doesn't depend directly in the current state of your infrastructure should be done asynchronously. I've had a lot of issues with systems that start up doing everything synchronously: you'll probably need to refactor it to be asynchronous in emergency mode during a crisis.

marcosdumay 7y ago
For email, that's why you set a relay within your control, that will accept messages without a fuss and send them around following SMTP conventions. But anyway, how is your application supposed to respond after any of those failures? Is it just supposed to ignore the failure and thread on like if nothing happened, leaving your users on the dark? Is it supposed to reliably log every task so that it can retry anything that fails and in the worst case feed failures into some monitoring system/process? Or is it supposed to inform the user of any success or failure before the user can move on? Queue software is only a good match for the first. For the second you will need to roll your own interface with the monitoring system anyway, so it's much easier to roll your own queues and get control of everything. The third one is best done synchronous, it doesn't matter the nature of the process or how long it takes. But funny thing is, I have never seen the first situation on the wild.