3 ms·

There are plenty of similar examples out there. Take, for instance, Kubernetes indefinitely failing to start a pod and just retrying while taking no action to

There are plenty of similar examples out there. Take, for instance, Kubernetes indefinitely failing to start a pod and just retrying while taking no action to notify whoever started the process (just hanging). So, I can see two different aspects to this problem: * Lack of alerts. * Lack of RCA automation. For the first one -- it would've been nice if systemd had a "brother" program that could be used for alerts, so that no custom solution was necessary and that services could properly report intermittent problems etc. The second is contingent on several factors: re-envisioning error handling, declarative debugging and general popularization of the concept. There are several major problems with error handling in system programming languages today. Due to poor language runtime design, programmers learn to think about any and every error as essentially fatal. Recovery is typically seen as impossible, and if there's any sort of recovery code put in place, it's usually the one that tries to do the cleanup and start fresh. It's never really about fixing the problem. So, popularizing something like Common Lisp restart system with the ability to traverse the program stack back to the failing frame would've been a good first step in this direction. Declarative debugging, on the other hand, could be taking another step forward, where it could be made into a separate program which describes a complex recovery scheme. I.e. the idea of declarative debugging is that the programmer needs to describe the program in terms of constraints that should be checked when the program fails to identify the problematic place. The step forward would be to add automation for the cases when the failed constraint is discovered.

vbezhenar 3y ago
Nobody would test their complex recovery schemes. Fail fast is the way.
- marcosdumay 3y ago
  Yep, recovering is a losing proposition. Database management systems don't bother with it anymore, your almost stateless deamon has no hope of gaining anything from it. But wide logging and error identification are things too; and very useful ones. And those plug on the same spots that were created for error recovering. Because of this, those are also not very common on our current system software.
  - crabbone 3y ago
    I'm in the storage business, probably about a decade. This is the first time I hear that database management systems don't bother with recovery... where are you getting this from? I'm pretty sure it's the exact opposite.
- crabbone 3y ago
  Nobody in Web development? -- maybe. Nobody in system (storage / network / compute) or any other specialized highly-reliable system -- unlikely. In highly reliable systems this is already being done, except it's usually an afterthought, ad hoc, functionality added without any system or framework, and so it's hard to judge its performance because it's often partially interactive, and very uneven across different components. Any system that advertises itself as "HA" (for highly-available) will have some sort of a protocol for complex error recovery, except it's usually not a single program that's explicitly written with the purpose of definition and documentation of recovery process, instead different bits of this functionality are embedded in many different places across the entire surface of a program. A very common example that I had to implement myself multiple times: HTTP errors. These are usually very poorly reported by the HTTP client libraries. TLS handshake failed? -- you get some numerical exit code instead of proper error report, and so once you realize that this error happens every so often, you start writing code that digs up the internal state of the TLS library at the time of the error, finds the certificates involved and tries to trace down the chain of events leading to the problem. Similarly, various parsing and de/serialization errors... this stuff is usually atrocious in popular libraries. And in order to figure out why some JSON didn't parse the way you wanted, you need to write a lot of code to investigate and report the error properly. The most common cases I've encountered is when a client program expects a server to reply with a structured error message, but the message comes back botched, or an HTML page is sent instead etc. A lot of these things are quite predictable after some time of using the client, but in a situation where you wanted to properly report an error that was supposed to be encoded in some JSON message and instead you got an HTML page -- typically you end up reporting a wrong (and useless) error of failure of parsing HTML as JSON. To prevent that, you'd need a complicated recovery scheme that would anticipate such errors.

vbezhenar 3y ago

Nobody would test their complex recovery schemes. Fail fast is the way.

marcosdumay 3y ago
Yep, recovering is a losing proposition. Database management systems don't bother with it anymore, your almost stateless deamon has no hope of gaining anything from it. But wide logging and error identification are things too; and very useful ones. And those plug on the same spots that were created for error recovering. Because of this, those are also not very common on our current system software.
- crabbone 3y ago
  I'm in the storage business, probably about a decade. This is the first time I hear that database management systems don't bother with recovery... where are you getting this from? I'm pretty sure it's the exact opposite.
crabbone 3y ago
Nobody in Web development? -- maybe. Nobody in system (storage / network / compute) or any other specialized highly-reliable system -- unlikely. In highly reliable systems this is already being done, except it's usually an afterthought, ad hoc, functionality added without any system or framework, and so it's hard to judge its performance because it's often partially interactive, and very uneven across different components. Any system that advertises itself as "HA" (for highly-available) will have some sort of a protocol for complex error recovery, except it's usually not a single program that's explicitly written with the purpose of definition and documentation of recovery process, instead different bits of this functionality are embedded in many different places across the entire surface of a program. A very common example that I had to implement myself multiple times: HTTP errors. These are usually very poorly reported by the HTTP client libraries. TLS handshake failed? -- you get some numerical exit code instead of proper error report, and so once you realize that this error happens every so often, you start writing code that digs up the internal state of the TLS library at the time of the error, finds the certificates involved and tries to trace down the chain of events leading to the problem. Similarly, various parsing and de/serialization errors... this stuff is usually atrocious in popular libraries. And in order to figure out why some JSON didn't parse the way you wanted, you need to write a lot of code to investigate and report the error properly. The most common cases I've encountered is when a client program expects a server to reply with a structured error message, but the message comes back botched, or an HTML page is sent instead etc. A lot of these things are quite predictable after some time of using the client, but in a situation where you wanted to properly report an error that was supposed to be encoded in some JSON message and instead you got an HTML page -- typically you end up reporting a wrong (and useless) error of failure of parsing HTML as JSON. To prevent that, you'd need a complicated recovery scheme that would anticipate such errors.