3 ms·

At my first job I wrote a new Java application to ingest/transform/store some data and after about 6 months it was done. It chugged along for years after withou

At my first job I wrote a new Java application to ingest/transform/store some data and after about 6 months it was done. It chugged along for years after without ever really requiring any maintenance or monitoring, it just kept working. One day the VP eng just showed up at my desk and started chatting, I didn't really know why. Turns out he was interested in knowing why it was so stable but it was an obvious part of software to me - every line I wrote where an exception was possible, I would look up what exceptions could be thrown and why, and then structure try catches to do the right thing in each case - sometimes that meant logging a warning, sometimes propagating to a higher level, sometimes shutting down the app etc. He then sent out some paper about the relationship between good error handling and software defects to the org. Like I don't get why this is so difficult for so many devs. Your job is to think about both the happy case and the error cases, and usually there are many error cases and 1 happy case so the bulk of time and code is dedicated to the errors. Since then I've seen this is not at all the way most devs write things. They are interested in writing the happy case and treat error handling like a chore to be done as quickly as possible. The tools and language constructs don't matter much, it's just do you dedicate time and care to thinking through errors or not. Yeah it sounds boastful but fuck it it's true. Same is true regardless of this study it's not about the structure and syntax of error handling,it's how much thought was given to understanding the failure modes and how to handle them.

MattPalmer1086 5y ago
Similarly, I've written systems that ran for years without needing much maintenance. I believe good error handling (and negative testing as well as positive testing) are the primary reasons for that. Unfortunately, it takes me longer to write code than my peers as a result. I'm not a professional developer anymore...
- zaptheimpaler 5y ago
  ouch i feel that. I do have the same problem all the time. if you don't mind me asking, what do you do now?
  - MattPalmer1086 5y ago
    I moved into information security. I still get to apply my dev knowledge. I don't write code in my job, but I do secure code reviews. I write open source code in my own time, and to be honest, enjoy it far more that way.
ravenstine 5y ago
I'm not exactly sure why that is either, but I do think most developers are appealed by the idea that a computer program is a closed system and thus any error that happens is a fluke, unexpected, and should be treated as catastrophic, when that is simply not true. Programs take input from the parent reality, asynchronously, and therefore aren't just subject to cosmic rays flipping bits in the hardware. Personally, I like to handle the possible errors intelligently, but many developers, especially web developers, love to just have any error result in an error page. Even worse, they sprinkle `try { ... } catch (err) { // do nothing }` all over the place, and that results in problems that are hard to track down or detect early.
metb 5y ago
>One day the VP eng just showed up at my desk and started chatting, I didn't really know why. Turns out he was interested in knowing why it was so stable Kudos to him, he paid attention, and noticed something that was stable. Usually something has to be broken to get attention at that level. Can't imagine leadership at most places thinking "hmm, this project we did several years ago had been running without bugs, let me go bring best practices from this developer."
hk-im-ad 5y ago
You assume that the only errors a program can encounter are logic errors. Logic errors are really the easiest class of error to fix. Here are examples of other errors that can bring down stable systems: 1) An API returns a list of items. In a new version of the API, the data structure used to generate the response list is changed from a list to a set, invalidating any implicit assumptions about response ordering. This happened to a service I worked on where there was an implicit assumption by callers that the results were ordered by date. This was true prior to using a set. 2) Resource leaks exposed by failure conditions. A network outage might cause infrequently tested code paths to leaks resources. In Go, I’ve seen this happen with network requests not tied to a context. This is an interesting case because the fact that an error occurs causes the leak even though the success path works correctly. 3) Missing back offs. A dependency may return errors in the case where a dependency becomes overloaded. This can cause a backup in something like an ingestion job. Without adequate back offs, the dependency may never be able to recover. There are plenty more examples where simply handling every error case is not sufficient for stability. As a question to you: why is this system more stable than other systems you write? Can’t you apply your error handling philosophy to everything?
- blub 5y ago
  Those errors can also be considered and mitigated though, if one thinks about what could go wrong instead of only thinking about what exceptions can be thrown: 1. One must either encode the assumption into a precondition or transfer the incoming data into a sorted data structure. But the gist is to always validate assumptions. 2. RAII’s pretty good at handling resources. Then one can inject failures to test the error handling paths and combine with code coverage measurements. 3. That sounds like an issue in the wider system which may be handled in the subsystem under development assuming that it can throttle back its requests, drop them, etc. But it may just as well be handled in another part of the larger system. It belongs more to the architecture realm, but it’s absolutely possible to foresee such issues.
  - hk-im-ad 5y ago
    For 1, the bug stems from unintentionally using undefined behavior due to a misunderstanding. You can never get the answer by "thinking about what could go wrong" because your view of the system makes the failure case impossible.
TheCoelacanth 5y ago
I think the hard part of handling all error cases like that is not the actual handling of the error cases, but making your code well structured enough that at any given place, you can actually enumerate all possible errors in a reasonably small list and so that you actually have a reasonable way of handling each error. In poorly structured code, all possible errors could often include just about anything and for many of the errors you will have no reasonable way of handling them.
journey_16162 5y ago
I'm refactoring a (not so big) TypeScript codebase from exceptions to returning error results for this reason. I don't know about Java, but TypeScript does not have a way to annotate that a function throws, not to mention what it throws - so carefully handling every error case would be difficult. With the new approach I know exactly what kind of error codes each function can return and how to handle them. Thanks for your comment, now I have more reassurance that I'm not wasting time with the refactor.

MattPalmer1086 5y ago

Similarly, I've written systems that ran for years without needing much maintenance. I believe good error handling (and negative testing as well as positive testing) are the primary reasons for that. Unfortunately, it takes me longer to write code than my peers as a result. I'm not a professional developer anymore...

zaptheimpaler 5y ago
ouch i feel that. I do have the same problem all the time. if you don't mind me asking, what do you do now?
- MattPalmer1086 5y ago
  I moved into information security. I still get to apply my dev knowledge. I don't write code in my job, but I do secure code reviews. I write open source code in my own time, and to be honest, enjoy it far more that way.

ravenstine 5y ago

I'm not exactly sure why that is either, but I do think most developers are appealed by the idea that a computer program is a closed system and thus any error that happens is a fluke, unexpected, and should be treated as catastrophic, when that is simply not true. Programs take input from the parent reality, asynchronously, and therefore aren't just subject to cosmic rays flipping bits in the hardware. Personally, I like to handle the possible errors intelligently, but many developers, especially web developers, love to just have any error result in an error page. Even worse, they sprinkle `try { ... } catch (err) { // do nothing }` all over the place, and that results in problems that are hard to track down or detect early.

metb 5y ago

>One day the VP eng just showed up at my desk and started chatting, I didn't really know why. Turns out he was interested in knowing why it was so stable Kudos to him, he paid attention, and noticed something that was stable. Usually something has to be broken to get attention at that level. Can't imagine leadership at most places thinking "hmm, this project we did several years ago had been running without bugs, let me go bring best practices from this developer."

hk-im-ad 5y ago

You assume that the only errors a program can encounter are logic errors. Logic errors are really the easiest class of error to fix. Here are examples of other errors that can bring down stable systems: 1) An API returns a list of items. In a new version of the API, the data structure used to generate the response list is changed from a list to a set, invalidating any implicit assumptions about response ordering. This happened to a service I worked on where there was an implicit assumption by callers that the results were ordered by date. This was true prior to using a set. 2) Resource leaks exposed by failure conditions. A network outage might cause infrequently tested code paths to leaks resources. In Go, I’ve seen this happen with network requests not tied to a context. This is an interesting case because the fact that an error occurs causes the leak even though the success path works correctly. 3) Missing back offs. A dependency may return errors in the case where a dependency becomes overloaded. This can cause a backup in something like an ingestion job. Without adequate back offs, the dependency may never be able to recover. There are plenty more examples where simply handling every error case is not sufficient for stability. As a question to you: why is this system more stable than other systems you write? Can’t you apply your error handling philosophy to everything?

blub 5y ago
Those errors can also be considered and mitigated though, if one thinks about what could go wrong instead of only thinking about what exceptions can be thrown: 1. One must either encode the assumption into a precondition or transfer the incoming data into a sorted data structure. But the gist is to always validate assumptions. 2. RAII’s pretty good at handling resources. Then one can inject failures to test the error handling paths and combine with code coverage measurements. 3. That sounds like an issue in the wider system which may be handled in the subsystem under development assuming that it can throttle back its requests, drop them, etc. But it may just as well be handled in another part of the larger system. It belongs more to the architecture realm, but it’s absolutely possible to foresee such issues.
- hk-im-ad 5y ago
  For 1, the bug stems from unintentionally using undefined behavior due to a misunderstanding. You can never get the answer by "thinking about what could go wrong" because your view of the system makes the failure case impossible.

TheCoelacanth 5y ago

I think the hard part of handling all error cases like that is not the actual handling of the error cases, but making your code well structured enough that at any given place, you can actually enumerate all possible errors in a reasonably small list and so that you actually have a reasonable way of handling each error. In poorly structured code, all possible errors could often include just about anything and for many of the errors you will have no reasonable way of handling them.

journey_16162 5y ago

I'm refactoring a (not so big) TypeScript codebase from exceptions to returning error results for this reason. I don't know about Java, but TypeScript does not have a way to annotate that a function throws, not to mention what it throws - so carefully handling every error case would be difficult. With the new approach I know exactly what kind of error codes each function can return and how to handle them. Thanks for your comment, now I have more reassurance that I'm not wasting time with the refactor.