3 ms·

In my experience designing systems like these: There are no hard and fast rules and you handle them in a way that’s been decided on a case-by-case basis for eac

In my experience designing systems like these: There are no hard and fast rules and you handle them in a way that’s been decided on a case-by-case basis for each system. Some messages may need to keep retrying with an exponential back off indefinitely. Others may need to retry only once and then send an email to someone because two failures in five minutes is a critical failure. You also have to design things so that if one failure occurs, maybe it stops the entire process, or maybe other messages can go through still and you just log the failure. It all comes down to the rules of your business and how critical these systems are. Maybe unknown means “failure” or maybe unknown means “someone should get an alert about this and check it out”. I think it’s hard because there is no highly visible “crash” that occurs like in a non-distributed system when an unexpected exception occurs and the entire program shuts down. Failures often happen silently and it’s difficult to tell where or why something failed. So you have to design each system with that in mind and figure out how each piece needs to deal with uncertainty.

stingraycharles 7y ago
For what it’s worth, in my experience it’s very effective to add this information to the error context: is it a permanent failure (eg validation) or a retryable error. If it is retryable, also add to the error context when it should be retried. This will allow you to handle these errors appropriately without having to handle these things on a case by case basis.
- clarry 7y ago
  > it’s very effective to add this information to the error context: is it a permanent failure (eg validation) or a retryable error The discussion was specifically about UNKNOWN errors, i.e. you sent a message but never got a reply back. You don't know whether it was a validation failure or temporary hiccup. For all you know, it's possible the message was received and processed correctly but the response never made it back. How to handle these unknowns is always going to be case by case. Some combination of retry and give up works for most cases, but there is no silver bullet and usually you have to think hard about the consequences of 1) retrying 2) giving up thinking the request failed even though it actually (silently) succeeded.

stingraycharles 7y ago

For what it’s worth, in my experience it’s very effective to add this information to the error context: is it a permanent failure (eg validation) or a retryable error. If it is retryable, also add to the error context when it should be retried. This will allow you to handle these errors appropriately without having to handle these things on a case by case basis.

clarry 7y ago
> it’s very effective to add this information to the error context: is it a permanent failure (eg validation) or a retryable error The discussion was specifically about UNKNOWN errors, i.e. you sent a message but never got a reply back. You don't know whether it was a validation failure or temporary hiccup. For all you know, it's possible the message was received and processed correctly but the response never made it back. How to handle these unknowns is always going to be case by case. Some combination of retry and give up works for most cases, but there is no silver bullet and usually you have to think hard about the consequences of 1) retrying 2) giving up thinking the request failed even though it actually (silently) succeeded.