3 ms·

Alternatively, autoscale your consumers based on queue size and alert if it's growing indefinitely. Better than losing messages IMO.

kqr 6y ago
That sounds like instructions for turning a cheap, easy to diagnose problem into a very expensive, harder to diagnose problem. Not to mention the second-order effects: if you pretend to drop exactly 0 % of messages, then people will build things that depend on that condition. When the unexpected happens and that condition is violated, bad becomes worse. By deliberately designing in the option of dropping a reasonable amount of messages, you're creating a more robust system. (As well as a perfect point for a circuit breaker in an emergency.) Other than in some very specific domains, dropping messages is not a big deal and the design implications of allowing that are far preferable to designing for a fantasy world in which nothing goes wrong ever.
- Joker_vD 6y ago
  While I agree with you, I think the parent's advice of "autoscale your consumers" is also a good advice, if interpreted in certain sense: that is, when you see that your queues start approaching the uncomfortable sizes, start dropping your clients. When they try to reconnect, they will probably get routed to a different, less busy node.
- cassianoleal 6y ago
  I would absolutely not want to lose log messages. I'd rather have my queue grow infinitely until I solve the ingestion issues than lose messages, _especially_ but not exclusively audit logs. I don't think logging is _some very specific domain_ but I will concede that this might be due to my own biases because of my role.
cryptica 6y ago
The best strategy is usually to tell the sender that their message could not be delivered and let them decide how to deal with this. Unfortunately, most current message queue systems are not end-to-end (e.g. re-send or modify their message); they usually have an additional intermediate service like HTTP or a WebSocket API to interface with end users; this design makes it difficult to implement this kind of error-routing mechanism, especially at scale when different end users might be connected to different hosts.

kqr 6y ago

That sounds like instructions for turning a cheap, easy to diagnose problem into a very expensive, harder to diagnose problem. Not to mention the second-order effects: if you pretend to drop exactly 0 % of messages, then people will build things that depend on that condition. When the unexpected happens and that condition is violated, bad becomes worse. By deliberately designing in the option of dropping a reasonable amount of messages, you're creating a more robust system. (As well as a perfect point for a circuit breaker in an emergency.) Other than in some very specific domains, dropping messages is not a big deal and the design implications of allowing that are far preferable to designing for a fantasy world in which nothing goes wrong ever.

Joker_vD 6y ago
While I agree with you, I think the parent's advice of "autoscale your consumers" is also a good advice, if interpreted in certain sense: that is, when you see that your queues start approaching the uncomfortable sizes, start dropping your clients. When they try to reconnect, they will probably get routed to a different, less busy node.
cassianoleal 6y ago
I would absolutely not want to lose log messages. I'd rather have my queue grow infinitely until I solve the ingestion issues than lose messages, _especially_ but not exclusively audit logs. I don't think logging is _some very specific domain_ but I will concede that this might be due to my own biases because of my role.

cryptica 6y ago

The best strategy is usually to tell the sender that their message could not be delivered and let them decide how to deal with this. Unfortunately, most current message queue systems are not end-to-end (e.g. re-send or modify their message); they usually have an additional intermediate service like HTTP or a WebSocket API to interface with end users; this design makes it difficult to implement this kind of error-routing mechanism, especially at scale when different end users might be connected to different hosts.