3 ms·

I like this statement: > The first law of queues is: Every queue should have a maximum size. Queues must not grow unbounded. Unfortunately, unbounded resource

I like this statement: > The first law of queues is: Every queue should have a maximum size. Queues must not grow unbounded. Unfortunately, unbounded resource is often used as a hammer to try preventing other smaller issues, with https://github.com/grpc/grpc/issues/8603 https://github.com/grpc/grpc/issues/8603 being a perfect example.

Koshkin 5y ago
Sometimes it is useful to impose different size limits on ‘subqueues’ (each consisting of elements of a certain kind).
cassianoleal 5y ago
Alternatively, autoscale your consumers based on queue size and alert if it's growing indefinitely. Better than losing messages IMO.
- kqr 5y ago
  That sounds like instructions for turning a cheap, easy to diagnose problem into a very expensive, harder to diagnose problem. Not to mention the second-order effects: if you pretend to drop exactly 0 % of messages, then people will build things that depend on that condition. When the unexpected happens and that condition is violated, bad becomes worse. By deliberately designing in the option of dropping a reasonable amount of messages, you're creating a more robust system. (As well as a perfect point for a circuit breaker in an emergency.) Other than in some very specific domains, dropping messages is not a big deal and the design implications of allowing that are far preferable to designing for a fantasy world in which nothing goes wrong ever.
  - Joker_vD 5y ago
    While I agree with you, I think the parent's advice of "autoscale your consumers" is also a good advice, if interpreted in certain sense: that is, when you see that your queues start approaching the uncomfortable sizes, start dropping your clients. When they try to reconnect, they will probably get routed to a different, less busy node.
  - cassianoleal 5y ago
    I would absolutely not want to lose log messages. I'd rather have my queue grow infinitely until I solve the ingestion issues than lose messages, _especially_ but not exclusively audit logs. I don't think logging is _some very specific domain_ but I will concede that this might be due to my own biases because of my role.
- cryptica 5y ago
  The best strategy is usually to tell the sender that their message could not be delivered and let them decide how to deal with this. Unfortunately, most current message queue systems are not end-to-end (e.g. re-send or modify their message); they usually have an additional intermediate service like HTTP or a WebSocket API to interface with end users; this design makes it difficult to implement this kind of error-routing mechanism, especially at scale when different end users might be connected to different hosts.
sundbry 5y ago
Circular dependencies causing deadlocks? That should not be a surprise. Making the request queue unbounded is just trading one problem for another.

Koshkin 5y ago

Sometimes it is useful to impose different size limits on ‘subqueues’ (each consisting of elements of a certain kind).

cassianoleal 5y ago

Alternatively, autoscale your consumers based on queue size and alert if it's growing indefinitely. Better than losing messages IMO.

kqr 5y ago
That sounds like instructions for turning a cheap, easy to diagnose problem into a very expensive, harder to diagnose problem. Not to mention the second-order effects: if you pretend to drop exactly 0 % of messages, then people will build things that depend on that condition. When the unexpected happens and that condition is violated, bad becomes worse. By deliberately designing in the option of dropping a reasonable amount of messages, you're creating a more robust system. (As well as a perfect point for a circuit breaker in an emergency.) Other than in some very specific domains, dropping messages is not a big deal and the design implications of allowing that are far preferable to designing for a fantasy world in which nothing goes wrong ever.
- Joker_vD 5y ago
  While I agree with you, I think the parent's advice of "autoscale your consumers" is also a good advice, if interpreted in certain sense: that is, when you see that your queues start approaching the uncomfortable sizes, start dropping your clients. When they try to reconnect, they will probably get routed to a different, less busy node.
- cassianoleal 5y ago
  I would absolutely not want to lose log messages. I'd rather have my queue grow infinitely until I solve the ingestion issues than lose messages, _especially_ but not exclusively audit logs. I don't think logging is _some very specific domain_ but I will concede that this might be due to my own biases because of my role.
cryptica 5y ago
The best strategy is usually to tell the sender that their message could not be delivered and let them decide how to deal with this. Unfortunately, most current message queue systems are not end-to-end (e.g. re-send or modify their message); they usually have an additional intermediate service like HTTP or a WebSocket API to interface with end users; this design makes it difficult to implement this kind of error-routing mechanism, especially at scale when different end users might be connected to different hosts.

sundbry 5y ago

Circular dependencies causing deadlocks? That should not be a surprise. Making the request queue unbounded is just trading one problem for another.