3 ms·

You still should use some sort of work queue - your application process may need to restart (deploys), or for a period of time, work could overflow the amount o

You still should use some sort of work queue - your application process may need to restart (deploys), or for a period of time, work could overflow the amount of available resources (bursts), so having some place to put the task before it goes onto processing is useful regardless of the underlying concurrency primitives of the language.

rollingbarreler 7y ago
Transactionality is enough for most systems. Your order either succeeds or fails, it never stays in an incomplete state. Queues are not a panacea, and introduce their own problems like obscuring problems by delaying them long enough to bring everything to a halt.
- mattbillenstein 7y ago
  Yes, you wouldn't do all work in the task queue - commonly, we make some change in the database which can happen pretty fast, and after that transaction commits, we might defer a task that sends a notification, email, whatever.
acjohnson55 7y ago
It may not make sense to retain jobs across deployments. What if the contract of the job is changed by the code being deployed? Might be easier to keep it all in-process, letting queues drain in a graceful shutdown.
- sateesh 7y ago
  This might not be viable all the time, what if you have a stream of tasks being put to the queue ? The alternative would be to ensure that any change in the job contracts are backward compatible, and if any change in contract would need to have a remediation/migration plan for handling pending tasks.
- mattbillenstein 7y ago
  I haven't found that to be the case typically -- you could always serialize some information into the task to check for things like this. Also consider if the machine running that process just disappears and that process dies. Putting work into a task queue allows you to do it durably until it can be processed so that it's not lost in some typical "that machine/instance died" scenario.
momokoko 7y ago
In the business, customer, end-user sense, in many situations, you are much better off to have a transaction fail with an error as opposed to finally successfully happening after 2 hours once a service disruption is cleared. Every problem is not the same. Work queues introduce a magnitude or more of complexity to an http application. Sometimes that is very needed. Sometimes its overengineering. Sometimes it is a gray area.

rollingbarreler 7y ago

Transactionality is enough for most systems. Your order either succeeds or fails, it never stays in an incomplete state. Queues are not a panacea, and introduce their own problems like obscuring problems by delaying them long enough to bring everything to a halt.

mattbillenstein 7y ago
Yes, you wouldn't do all work in the task queue - commonly, we make some change in the database which can happen pretty fast, and after that transaction commits, we might defer a task that sends a notification, email, whatever.

acjohnson55 7y ago

It may not make sense to retain jobs across deployments. What if the contract of the job is changed by the code being deployed? Might be easier to keep it all in-process, letting queues drain in a graceful shutdown.

sateesh 7y ago
This might not be viable all the time, what if you have a stream of tasks being put to the queue ? The alternative would be to ensure that any change in the job contracts are backward compatible, and if any change in contract would need to have a remediation/migration plan for handling pending tasks.
mattbillenstein 7y ago
I haven't found that to be the case typically -- you could always serialize some information into the task to check for things like this. Also consider if the machine running that process just disappears and that process dies. Putting work into a task queue allows you to do it durably until it can be processed so that it's not lost in some typical "that machine/instance died" scenario.

momokoko 7y ago

In the business, customer, end-user sense, in many situations, you are much better off to have a transaction fail with an error as opposed to finally successfully happening after 2 hours once a service disruption is cleared. Every problem is not the same. Work queues introduce a magnitude or more of complexity to an http application. Sometimes that is very needed. Sometimes its overengineering. Sometimes it is a gray area.