4 ms·
Resilience is one of the main characteristics of a stable codebase. Many failures, probably most failures after your developers hit mid-level, occur because an
by cookiecaper 8y ago
Resilience is one of the main characteristics of a stable codebase. Many failures, probably most failures after your developers hit mid-level, occur because an assumption about something stopped holding true.
Just this last week there was a major outage on an internal tool because the connection to the LDAP server got flaky, and the internal tool didn't know how to handle that -- instead of recovering gracefully, it freaked out until the whole server crashed.
That could've been avoided with just a small amount of recovery logic, but instead, the original author must've thought "If we can't keep our LDAP server connected, we have serious problems". But in the real world, things go sideways sometimes. In the real world, there are bugs in Microsoft's July 2018 patchset that ruin the TCP stack (cf. https://support.microsoft.com/en-us/help/4345421/windows-10-update-kb4345421 https://support.microsoft.com/en-us/help/4345421/windows-10-... among many others).
It's simple naivete to assume that you won't have problems as long as you keep your nose clean. There are externalities that impact even the most meticulous of us. If you want reliability, your systems must anticipate this.
- maxxxxx 8y agoThe problem is that you often can't anticipate in what ways other system may f..k you over. So automated restarts make sense. But you should then try to analyze the problem and handle it properly the next time.