5 ms·
Queues introduce entire other dimensions of complexity. Now you've got to monitor your queue size (and ideally autoscale when the queue backlog grows), and hav
by cle 7y ago
Queues introduce entire other dimensions of complexity. Now you've got to monitor your queue size (and ideally autoscale when the queue backlog grows), and have a dead letter queue for messages that failed processing and monitor that. Tracing requests is harder b/c now your logs are scattered around the worker fleet, so debugging becomes harder. You need more APIs for the client to poll the async state, and you need some data store to track the async state (and now you've got to worry about maintaining and monitoring that data store). It's a can of worms that should be avoided when possible.
The only way to know whether or not to accept this kind of complexity is to think about your use cases. Quite often it's fine (and desirable) to fail fast and make the client retry.
- staticassertion 7y ago> Now you've got to monitor your queue size (and ideally autoscale when the queue backlog grows), and have a dead letter queue for messages that failed processing and monitor that. These are both trivial things to do though. I don't see how it's any more complex than monitoring a circuit breaker, or setting up CI/CD. > Tracing requests is harder b/c now your logs are scattered around the worker fleet, so debugging becomes harder. Correlation IDs work just as well in a queue based system as a sync system. > You need more APIs for the client to poll the async state, and you need some data store to track the async state (and now you've got to worry about maintaining and monitoring that data store). Not sure what you mean. Again, in the Amazon Cart example, your state is just the cart - regardless of sync or async. You don't add any new state management at all.
- vageli 7y ago> Queues introduce entire other dimensions of complexity. Now you've got to monitor your queue size (and ideally autoscale when the queue backlog grows), and have a dead letter queue for messages that failed processing and monitor that. Wouldn't you need similar mechanisms without a queue? It seems to me queues give more visibility and more hooks for autoscaling without adding additional instrumentation to the app itself.
- cle 7y agoThe queue sits behind a service. If you don't do the work in the service, and do it in a queue instead, you've got more infrastructure to manage, monitor, and autoscale.
- mey 7y agoA few other things to consider when working w/ Queue/Message Bus systems. Back pressure complexity. Sharing a queue service, leading to a potential central point of failure/capacity issue. Message schema management.