4 ms·

> For us, it's either a function that will retry the messages after some time, or manual intervention. What does manual intervention look like? Are all downstr

> For us, it's either a function that will retry the messages after some time, or manual intervention. What does manual intervention look like? Are all downstream consumers blocked until the DLQ is emptied or do all consumers know how to deal with late-arriving events?

giantg2 5y ago
Manual intervention could be moving them back to the main queue to try again, or looking up the data that failed like if the customer that is being transacted on exists. Most systems don't have an issue processing late messages. The ones that do would have to have some sort of retry/ingress process.
jayd16 5y ago
>What does manual intervention look like? A ticket or an ops alert depending on severity. The cause gets fixed and messages are retried or dropped depending on your business case. >Are all downstream consumers blocked until the DLQ is emptied No, its a diffent queue. The point is that you eventually pull them into the DLQ so you aren't wasting all your resources on failing messages. >do all consumers know how to deal with late-arriving events? Hopefully, but an outage is an outage. You're way past happy path at that point.
Licentia 5y ago
> Are all downstream consumers blocked until the DLQ is emptied No other service should know about the state of another services DLQ > do all consumers know how to deal with late-arriving events In my approach in architecting event driven systems, this question is meaningless. For me some of the core tenants (for me) of architecting these systems are 1. Message ordering should not matter 2. Message delays should not matter (no temporal coupling) 3. Message replays should have no side effects This means that Service B should have no knowledge of what happens in Service A. If something in Service A fails and gets sent to Service A's DLQ, it should have no impact on Service B. This often involves rethinking business requirements and processes to account for workflows that get interrupted - I've yet to find an insurmountable issue. If however, we are talking about a message broker trying to deliver a message from Service A to Service B, failing and forwarding to Service B's DLQ, then there is a very simple solution. Have your message broker deliver messages to queues. Service B then processes and handles messages from the queue rather than having to immediately handle all incoming messages from the broker. This protects against a lot of failure modes. The only thing(s) that arrive at Service B's DLQ are exceptions thrown internally in Service B

giantg2 5y ago

Manual intervention could be moving them back to the main queue to try again, or looking up the data that failed like if the customer that is being transacted on exists. Most systems don't have an issue processing late messages. The ones that do would have to have some sort of retry/ingress process.

jayd16 5y ago

>What does manual intervention look like? A ticket or an ops alert depending on severity. The cause gets fixed and messages are retried or dropped depending on your business case. >Are all downstream consumers blocked until the DLQ is emptied No, its a diffent queue. The point is that you eventually pull them into the DLQ so you aren't wasting all your resources on failing messages. >do all consumers know how to deal with late-arriving events? Hopefully, but an outage is an outage. You're way past happy path at that point.

Licentia 5y ago

> Are all downstream consumers blocked until the DLQ is emptied No other service should know about the state of another services DLQ > do all consumers know how to deal with late-arriving events In my approach in architecting event driven systems, this question is meaningless. For me some of the core tenants (for me) of architecting these systems are 1. Message ordering should not matter 2. Message delays should not matter (no temporal coupling) 3. Message replays should have no side effects This means that Service B should have no knowledge of what happens in Service A. If something in Service A fails and gets sent to Service A's DLQ, it should have no impact on Service B. This often involves rethinking business requirements and processes to account for workflows that get interrupted - I've yet to find an insurmountable issue. If however, we are talking about a message broker trying to deliver a message from Service A to Service B, failing and forwarding to Service B's DLQ, then there is a very simple solution. Have your message broker deliver messages to queues. Service B then processes and handles messages from the queue rather than having to immediately handle all incoming messages from the broker. This protects against a lot of failure modes. The only thing(s) that arrive at Service B's DLQ are exceptions thrown internally in Service B