3 ms·

Everything can fail. Things can fail without even telling you they failed. Writes to disk can fail even when it returns success, even with a synchronous write,

Everything can fail. Things can fail without even telling you they failed. Writes to disk can fail even when it returns success, even with a synchronous write, even after sync(). The world can lie to us. For your own sanity, I recommend not thinking about this often... > Neither of them fail often, but when they do, you can't just ignore it. You have to do something intelligent about it. Nope. You can often do very dumb shit in response to it. Just like you don't have to do anything intelligent when malloc() fails, you don't have to do anything intelligent when fork() fails. The most common thing to do is to restart whatever was failing , starting with the smallest "thing" and going up in size (threads->pids->pgroups->containers->vms->bare metal, tasks->services->clusters, etc). Of course, you should probably have some kind of cumulative alert to tell someone to fix it, but often the restart is good enough to keep requests being processed. Such is the reality of imperfect production systems.

asveikau 7y ago
> For your own sanity, I recommend not thinking about this often... If you mean this to say you shouldn't check for errors, I disagree. Write errors that fail with an error code are much more common than those that do not. The latter in my experience is usually a particularly bad hardware problem. For the most common write errors (including disk full, but also errors arising from hardware failures), I think you should make a best effort to check and flag it. The larger your base of installed machines, the more you will be thankful for this logging. You don't want to be diagnosing rare failures blindly. A low failure rate multiplied by a large install base leads to that problem daily. > Just like you don't have to do anything intelligent when malloc() fails This is also false in my experience. I have seen stuff in the field where malloc(40) [yes, that small] fails, and I am still able to log where that happens, write it to disk, and have it be recovered and read by me, the developer. I've also seen malloc fail and a UI program survive long enough to report the error to the user and maybe even retry the operation successfully. The reason for this has to do with how you react. In the first case: a log file write does not ordinarily result in memory allocation. [fprintf does, but there are other ways to log]. In the second case, firstly, mallocs that fail are often absurdly large, where a smaller malloc will succeed; and second, you will react to the malloc failure by unwinding your stack and this will have the side effect of many stack frames calling free(), which might give you enough space to keep going successfully.
- peterwwillis 7y ago
  It's a waste of time to check most errors. Usually they don't have enough context to diagnose a problem, and they can generate a lot of unnecessary noise. Catching exceptions is one thing, but just saying 'error!' or having specific logic for each and every potential failure just leads to spaghetti code that doesn't really help your overall system. Instead, you should measure the overall "golden signals" of application/service/system health, and if you find a problem with those, you can start tracing calls, which will show you the erroring system calls. Cross-referenced with other metrics, you find the root cause. This is for when your application is part of a larger system, anyway. If you're writing a monolithic application where the entire system is your application and it can never go down, then checking and handling every error is more useful, but then your monolith starts to internally resemble a complex system and you're back to too much noise.
  - asveikau 7y ago
    Ok, so this is pretty much counter to my experience. I've spent a significant chunk of my career just diagnosing errors based on error codes. It does work. When the kernel tells you some data is not found, permission is denied, or the filesystem is corrupt, or I/O failed, it really isn't kidding, it gives you specific advice and you should pay attention. You can also do things like look in kernel source to see when that error occurs, and depending on the error get really specific ideas about the problem. Keep in mind not every job will have you with total control of a machine as in a server use case. In many domains deploying code to someone else's device you won't know that you're running somewhere short of disk space, or whatever, until you see it in the error code. Further, I suspect you are working in higher level languages where somebody has already wrapped the error codes in exceptions, and already logs unhandled exceptions to stderr. So what I would call very basic "error handling" is already done for you. When talking about C, if this more manual work is not done, ignoring error codes and continuing to keep on chugging is just stupid in a lot of cases, will cause you to silently lose customer data if you don't stop yourself on the error, etc., which is bad times.

asveikau 7y ago

> For your own sanity, I recommend not thinking about this often... If you mean this to say you shouldn't check for errors, I disagree. Write errors that fail with an error code are much more common than those that do not. The latter in my experience is usually a particularly bad hardware problem. For the most common write errors (including disk full, but also errors arising from hardware failures), I think you should make a best effort to check and flag it. The larger your base of installed machines, the more you will be thankful for this logging. You don't want to be diagnosing rare failures blindly. A low failure rate multiplied by a large install base leads to that problem daily. > Just like you don't have to do anything intelligent when malloc() fails This is also false in my experience. I have seen stuff in the field where malloc(40) [yes, that small] fails, and I am still able to log where that happens, write it to disk, and have it be recovered and read by me, the developer. I've also seen malloc fail and a UI program survive long enough to report the error to the user and maybe even retry the operation successfully. The reason for this has to do with how you react. In the first case: a log file write does not ordinarily result in memory allocation. [fprintf does, but there are other ways to log]. In the second case, firstly, mallocs that fail are often absurdly large, where a smaller malloc will succeed; and second, you will react to the malloc failure by unwinding your stack and this will have the side effect of many stack frames calling free(), which might give you enough space to keep going successfully.

peterwwillis 7y ago
It's a waste of time to check most errors. Usually they don't have enough context to diagnose a problem, and they can generate a lot of unnecessary noise. Catching exceptions is one thing, but just saying 'error!' or having specific logic for each and every potential failure just leads to spaghetti code that doesn't really help your overall system. Instead, you should measure the overall "golden signals" of application/service/system health, and if you find a problem with those, you can start tracing calls, which will show you the erroring system calls. Cross-referenced with other metrics, you find the root cause. This is for when your application is part of a larger system, anyway. If you're writing a monolithic application where the entire system is your application and it can never go down, then checking and handling every error is more useful, but then your monolith starts to internally resemble a complex system and you're back to too much noise.
- asveikau 7y ago
  Ok, so this is pretty much counter to my experience. I've spent a significant chunk of my career just diagnosing errors based on error codes. It does work. When the kernel tells you some data is not found, permission is denied, or the filesystem is corrupt, or I/O failed, it really isn't kidding, it gives you specific advice and you should pay attention. You can also do things like look in kernel source to see when that error occurs, and depending on the error get really specific ideas about the problem. Keep in mind not every job will have you with total control of a machine as in a server use case. In many domains deploying code to someone else's device you won't know that you're running somewhere short of disk space, or whatever, until you see it in the error code. Further, I suspect you are working in higher level languages where somebody has already wrapped the error codes in exceptions, and already logs unhandled exceptions to stderr. So what I would call very basic "error handling" is already done for you. When talking about C, if this more manual work is not done, ignoring error codes and continuing to keep on chugging is just stupid in a lot of cases, will cause you to silently lose customer data if you don't stop yourself on the error, etc., which is bad times.