4 ms·

That needs a few conditions to be accepted: - an isolated process which failure doesn’t cascade other part’s failure - as parent mentionned, where and what fa

That needs a few conditions to be accepted: - an isolated process which failure doesn’t cascade other part’s failure - as parent mentionned, where and what failed needs to be super clear - people are available to timely react to the error, so a rerun will succeed Failing any of the above, and you’ll need extensive and probably complex error handling that can at least help the system work in a degraded state until the error source is handled. In my experience, it’s really the last part that motivates engineers to try to deal with the maximum error cases automatically instead of having to deal with them weekends after weekends.

tsimionescu 4y ago
I believe one of the assumptions is that the failed process can automatically restart (e.g. using systemd, Kubernetes, hypervisor policies, top-level retries) - so that transient errors recover automatically, and at worst cost some performance or tiny bits of lost work (e.g. the setting an end user just hit apply on doesn't get applied, so they have to click again).
- makeitdouble 4y ago
  Yes, that’s really where it gets funny. In k8s, if I remember well, the default retry policy for failures will involve incremental back offs, which means if your transient error lasts 20 min your next retry might be in a few hours. It’s fine if your system is ok with that, otherwise jobs will need to “succeed” even when they fail, and so handle errors as gracefully as possible. Same actually for the user input one: you need to tell your user it’s a recoverable error and not just throw a random “oopsy” message, which means at least some handling of the error to come clean at the end of the tunnel. My take is, errors are complicated. It’s nice when a script can just die at the first error and not care about how or what happens from there, but that’s such a niche case.

tsimionescu 4y ago

I believe one of the assumptions is that the failed process can automatically restart (e.g. using systemd, Kubernetes, hypervisor policies, top-level retries) - so that transient errors recover automatically, and at worst cost some performance or tiny bits of lost work (e.g. the setting an end user just hit apply on doesn't get applied, so they have to click again).

makeitdouble 4y ago
Yes, that’s really where it gets funny. In k8s, if I remember well, the default retry policy for failures will involve incremental back offs, which means if your transient error lasts 20 min your next retry might be in a few hours. It’s fine if your system is ok with that, otherwise jobs will need to “succeed” even when they fail, and so handle errors as gracefully as possible. Same actually for the user input one: you need to tell your user it’s a recoverable error and not just throw a random “oopsy” message, which means at least some handling of the error to come clean at the end of the tunnel. My take is, errors are complicated. It’s nice when a script can just die at the first error and not care about how or what happens from there, but that’s such a niche case.