7 ms·

Studying the relationship between exception handling and post-release defects

75 points by zdw 5y ago

kinow 5y ago
Interesting article and great blog post too. I was lucky to start my career with a mentor who considered well written exception handling core to software development. And he was always happy to stop his work (maintaining a core middleware in a telco with millions of transactions every minute) to review this intern's poorly written try/catch, show how he would write it, and why. To this day I am thankful and believe his concerns over exception handling, good and simple deployment techniques, knowing networking to OS/kernel level, helped me becoming a better developer (or at least knowing where I need to improve).
- jimbob45 5y ago
  To be clear, did he consider exceptions something to be avoided when possible or did he consider them to be a natural element of programming? I know exceptions do cause slow-downs on systems and most programming does its best to minimize occurrences within reason.
  - kaetemi 5y ago
    Throwing exceptions is slow. Adding exception handling code itself has little to no overhead. Only throw exceptions in exceptional cases, not for frequently expected outcomes.
    - kaetemi 5y ago
      Adding to that, assume any C++ code can throw exceptions unless it's specifically stated it won't. If you're doing manual memory allocation in a function, your exception handling might be leaky. (gsl::finally is your friend.)
    - renewiltord 5y ago
      Java exceptions build stacktrace at construct time, which makes for the fun fact that instantiating an exception is the majority of its cost. At least the last time I checked. Actually throwing it is some fifth of the cost. This is relevant in the log-and-throw-e situations.
    - amelius 5y ago
      Imagine you're writing a function that parses an integer from a string. Would you let it throw an exception on failure?
      - kaetemi 5y ago
        Depends on whether you call the function `parse` or `tryParse`. :) Throw if you expect that the input should never be invalid. Don't throw, and do friendly logging and reporting of your parsing errors, if this is uncontrollable third party input.
  - gwbas1c 5y ago
    The downvotes are unfair for an extremely good question. (I used to ask questions along this line when interviewing candidates.) Exceptions are used to communicate an unexpected error. Specifically, an error that the caller doesn't expect to handle during normal operations of the program. These errors can range from unusual situations like a network failure, to even more perverse situations like true bugs in the program. The example I discussed with job candidates was implementing a database access function called GetUserById. (In C#, pre compiler enforced null checking.) I would ask if the function should return null or throw an exception when there was no user with that ID. What followed, (with the candidates who passed,) was a discussion about the trade-offs of returning null versus throwing an exception. Null allows the caller to know that there was no user with that ID without the overhead of the exception. But, returning null increases the risk of a NullReferenceException. This is risky, because it's harder to debug then a strongly typed exception with a useful error message. Thus, the "right" approach depended on if it was anticipated that someone calling GetUserById expected the user ID to always be for a valid user. When there was time, we'd even get into the TryGet pattern that the .Net dictionaries use. (By the way, now it's a good time to check out how Rust's enum type is used with error handling. It's really slick with no overhead.)
    - wyager 5y ago
      > I would ask if the function should return null or throw an exception when there was no user with that ID. My answer would be “neither”. Both of these options are bad. Null is maybe less bad because it at least roughly approximates the algebraic type that accurately represents the image of the function. Ideally you would return some type like Optional<User>. I suspect this is what you’re referring to with the Rust reference. A slightly more generalized version of this is to use the Either/Result monad, which can nicely handle multiple types of exceptional behavior at once.
      - jwdunne 5y ago
        Hrm, not the OP but, personally, it depends. If you’re using a language without algebraic data types, you’d have to ask whether it’s perfectly normal for a record not to be found by ID at that point. If it’s unusual, you should probably just throw the exception at that point because: 1. You check null and throw anyway, which gives you a stack trace just a little off the mark. That isn’t too bad but it’s more code for little gain. 2. There’s every chance you could forget to handle the null (It’s not the nil, it’s my discipline, I know, but it happens, we’ve all been there) and then you get a null pointer exception at some point later on in the flow, which is like the above but even worse. If you’re using algebraic data types, you wouldn’t use an optional. There are a number of ways a DB look up can fail so Optional isn’t really ideal at all. Your suggestion of Either/Result is probably the ideal there. Chances that you expect an explicit DB look up by ID to return nothing is pretty rare. Where has the program or client code got that ID from in the first place? Either you want to know about the error in your own code or you want the client code to know the error so you probably want some form of exception or Either. In the rare case it is intended behaviour, yeah you could use null and push the responsibility for handling it to the calling code. Still, at that point, 2 kicks in - might be fine now but that assumption could easily change as the code evolves, which it can and often does. You could argue you can implement something like Optional in, say, Java w/o algebraic data types but they are simply not ergonomic when the language does not help you use them. But I don’t think you can hold up a single approach and say that’s the ideal. I doubt it was your intention but it’s rather dogmatic and doesn’t really play out nicely in real world code, especially legacy projects (not everything is written in Haskell|Rust).
  - kinow 5y ago
    I think he never considered the exceptions to the something to be avoided since he wrote Java most of his professional career - which ended when he decided to quit programming and go sell ornamental flowers in his family business in his hometown. The middleware we worked had libraries from other companies (e.g. online prepaid transactions were handled by code from a jar/lib from Ericsson). But in the case the platform threw an exception, we could still charge the customer using a slower process (i.e. there's a window of time where the customer could use data/talk in excess until we realize there's no more credit when processing the slower transaction). For him it was important to write these exceptions well. If something went wrong in production, we couldn't turn off the system, and if we were not charging users when we should to, the company would be losing millions (it was in Brazil, ~200 mi at the time, and the telco had ~60mi users I think, with mother's day and big brother being the craziest days with millions of messages per second.)
mindwok 5y ago
Personally I really dislike exceptions and try/catch in languages. I don't like having to worry about whether some function call is going to surprise me with an exception, and handling them with try/catch really breaks the flow of the program. I'm sure there's probably a lot I can learn to make this better for myself because I don't work with exception heavy code very often, but I find working with simple error value returns like in Go or error types in Haskell/Rust to be so much more ergonomic and comfortable to work with.
- planet-and-halo 5y ago
  I think the biggest place where it's really necessary is integration points. Within your system it's reasonable to try to define exceptions out of existence. Once you start accepting user input or depending upon some system outside of your control, though, you better have some kind of mechanism to handle whatever unknown asteroids come flying in from deep space ready to annihilate your entire planet and civilization.
  - gizmo686 5y ago
    That is still provided by error values or error types. The defining feature of exceptions is that they provide a secondary control flow path, and that control flow path automatically flows up the stack until it reaches an explicit catch. With error values/types, control flow happens normally, and the programmer is expected to explicitly branch on the the value using normal control flow constructs.
- rgharris 5y ago
  Error handling in Go is simple and verbose at the same time. Working in it alongside Python and JavaScript makes me rethink a lot of patterns I’m accustomed to - it’s a nice exercise. Though of course the use cases often vary significantly between the 3 languages so it is tough to even compare.
- hackinthebochs 5y ago
  The benefit of exceptions and try/catch is that it lets you separate your exception handling logic from the mainline logic of the function. Having error logic weaved in and out of mainline logic just obscures what is going on and increases cognitive load. I much prefer writing and reading code with exceptions than explicit error handling control flow.
lmilcin 5y ago
> The longer the exception handling blocks in a file, the more likely the file is to contain bugs. It is difficult to design a sophisticated error handling system. For various reasons. For example it is rarely talked about -- everybody focuses on how to get stuff working and errors aren't really hot topic people find interesting. I think, given the above, you have most chance of success with really simple error handling systems. The simplest is to only handle things you can really handle at the current frame and let everything else filter to the top and interrupt entire process. It is not perfect but it has the virtue of being simple and easy to make foolproof. > The Ignoring Interrupted Exception and Log and Throw patterns corelated with post-release defects in one of the projects they studied, but not all. Interrupted exception does not happen in real usage in a lot of applications, especially backends, where the environment is very well controlled. Log and throw can be a legitimate pattern. It makes sense when the exception travels outside some kind of boundary like module/library boundary. A REST API is an example of boundary and it is normal to log the error but still throw it (to the client). It also may possibly make sense to log additional information available in local scope and throw the error up the stack. It doesn't always make sense to convert the type of the exception to be able to add more information to it.
- gwbas1c 5y ago
  > Log and throw can be a legitimate pattern. It makes sense when the exception travels outside some kind of boundary like module/library boundary. A REST API is an example of boundary and it is normal to log the error but still throw it (to the client). It also may possibly make sense to log additional information available in local scope and throw the error up the stack. It doesn't always make sense to convert the type of the exception to be able to add more information to it. Of course it's going to correlate with a bug. The whole point of log and throw is to make bugs easier to find!
- TeMPOraL 5y ago
  > It also may possibly make sense to log additional information available in local scope and throw the error up the stack. Like, time. This is why I often do "log and throw" or "log unexpected value and return it", depending on whether the code is exception-oriented or uses sum types instead. It's because time is an important piece of information to encode in an error log. When your error is caused by an external resource (e.g. RPC failure), you want to log it immediately to later be able to correlate it with external resource's logs or some other monitoring. If you don't the error may take some time to reach its final logging place - or, it may never reach it at all, if the application crashes in the meantime.
zaptheimpaler 5y ago
At my first job I wrote a new Java application to ingest/transform/store some data and after about 6 months it was done. It chugged along for years after without ever really requiring any maintenance or monitoring, it just kept working. One day the VP eng just showed up at my desk and started chatting, I didn't really know why. Turns out he was interested in knowing why it was so stable but it was an obvious part of software to me - every line I wrote where an exception was possible, I would look up what exceptions could be thrown and why, and then structure try catches to do the right thing in each case - sometimes that meant logging a warning, sometimes propagating to a higher level, sometimes shutting down the app etc. He then sent out some paper about the relationship between good error handling and software defects to the org. Like I don't get why this is so difficult for so many devs. Your job is to think about both the happy case and the error cases, and usually there are many error cases and 1 happy case so the bulk of time and code is dedicated to the errors. Since then I've seen this is not at all the way most devs write things. They are interested in writing the happy case and treat error handling like a chore to be done as quickly as possible. The tools and language constructs don't matter much, it's just do you dedicate time and care to thinking through errors or not. Yeah it sounds boastful but fuck it it's true. Same is true regardless of this study it's not about the structure and syntax of error handling,it's how much thought was given to understanding the failure modes and how to handle them.
- MattPalmer1086 5y ago
  Similarly, I've written systems that ran for years without needing much maintenance. I believe good error handling (and negative testing as well as positive testing) are the primary reasons for that. Unfortunately, it takes me longer to write code than my peers as a result. I'm not a professional developer anymore...
  - zaptheimpaler 5y ago
    ouch i feel that. I do have the same problem all the time. if you don't mind me asking, what do you do now?
    - 5y ago
Too 5y ago
> there exist anti-patterns that can provide significant explanatory power to the probability > of post-release defects. > Therefore, development teams should consider allocating > more resources to improving their exception handling practices While an interesting correlation. The conclusion that plugging this particular hole might not be the best goal to focus on improving. It's two variables that are not statistically independent of each other. A novice developer is more likely to write bugs as they are to use anti-patterns of exception handling, or anti-patterns for anything else for that matter. Basically correlation is not causation.
Arech 5y ago
The statistician in me is severely troubled by a conclusion like "statistical relationship" when the conclusion is drawn from studying just 3 (three) projects. At worst case this means studying programming habits of just 3 (three) developers (appended: actually, 1 dev is the worst case). You just can't make any significant conclusions on such basis that generalize on whole population. Even if it was huge projects with hundreds of devs, you still can't make generalizable conclusions, because habits of all this devs are clearly not independent.
branko_d 5y ago
On the exception receiving side... The key, IMO, is to assume that every line of code can throw until proven otherwise, and then make sure you clean-up after yourself, using RAII in C++, `using` in C#, savepoints in SQL or whatever else mechanism is available in your language. After that, in 99% cases, you just let the exception propagate to the higher level. Eventually, it will result in an error dialog, or be logged or whatever, but the program is still in consistant state. It's a kind of a "soft reset" that doesn't destroy the state and lets you keep working. --- On the exception producing side... Don't throw a new exception unles you expect it can be reasonably treated as described above. If your immediate caller needs a `try...catch` wrapped directly around the call, this is probably a sign you should have returned an error result instead of throwing an exception.

75 points by zdw 5y ago

Interesting article and great blog post too. I was lucky to start my career with a mentor who considered well written exception handling core to software development. And he was always happy to stop his work (maintaining a core middleware in a telco with millions of transactions every minute) to review this intern's poorly written try/catch, show how he would write it, and why. To this day I am thankful and believe his concerns over exception handling, good and simple deployment techniques, knowing networking to OS/kernel level, helped me becoming a better developer (or at least knowing where I need to improve).

jimbob45 5y ago
To be clear, did he consider exceptions something to be avoided when possible or did he consider them to be a natural element of programming? I know exceptions do cause slow-downs on systems and most programming does its best to minimize occurrences within reason.
- kaetemi 5y ago
  Throwing exceptions is slow. Adding exception handling code itself has little to no overhead. Only throw exceptions in exceptional cases, not for frequently expected outcomes.
  - kaetemi 5y ago
    Adding to that, assume any C++ code can throw exceptions unless it's specifically stated it won't. If you're doing manual memory allocation in a function, your exception handling might be leaky. (gsl::finally is your friend.)
  - renewiltord 5y ago
    Java exceptions build stacktrace at construct time, which makes for the fun fact that instantiating an exception is the majority of its cost. At least the last time I checked. Actually throwing it is some fifth of the cost. This is relevant in the log-and-throw-e situations.
  - amelius 5y ago
    Imagine you're writing a function that parses an integer from a string. Would you let it throw an exception on failure?
    - kaetemi 5y ago
      Depends on whether you call the function `parse` or `tryParse`. :) Throw if you expect that the input should never be invalid. Don't throw, and do friendly logging and reporting of your parsing errors, if this is uncontrollable third party input.
- gwbas1c 5y ago
  The downvotes are unfair for an extremely good question. (I used to ask questions along this line when interviewing candidates.) Exceptions are used to communicate an unexpected error. Specifically, an error that the caller doesn't expect to handle during normal operations of the program. These errors can range from unusual situations like a network failure, to even more perverse situations like true bugs in the program. The example I discussed with job candidates was implementing a database access function called GetUserById. (In C#, pre compiler enforced null checking.) I would ask if the function should return null or throw an exception when there was no user with that ID. What followed, (with the candidates who passed,) was a discussion about the trade-offs of returning null versus throwing an exception. Null allows the caller to know that there was no user with that ID without the overhead of the exception. But, returning null increases the risk of a NullReferenceException. This is risky, because it's harder to debug then a strongly typed exception with a useful error message. Thus, the "right" approach depended on if it was anticipated that someone calling GetUserById expected the user ID to always be for a valid user. When there was time, we'd even get into the TryGet pattern that the .Net dictionaries use. (By the way, now it's a good time to check out how Rust's enum type is used with error handling. It's really slick with no overhead.)
  - wyager 5y ago
    > I would ask if the function should return null or throw an exception when there was no user with that ID. My answer would be “neither”. Both of these options are bad. Null is maybe less bad because it at least roughly approximates the algebraic type that accurately represents the image of the function. Ideally you would return some type like Optional<User>. I suspect this is what you’re referring to with the Rust reference. A slightly more generalized version of this is to use the Either/Result monad, which can nicely handle multiple types of exceptional behavior at once.
    - jwdunne 5y ago
      Hrm, not the OP but, personally, it depends. If you’re using a language without algebraic data types, you’d have to ask whether it’s perfectly normal for a record not to be found by ID at that point. If it’s unusual, you should probably just throw the exception at that point because: 1. You check null and throw anyway, which gives you a stack trace just a little off the mark. That isn’t too bad but it’s more code for little gain. 2. There’s every chance you could forget to handle the null (It’s not the nil, it’s my discipline, I know, but it happens, we’ve all been there) and then you get a null pointer exception at some point later on in the flow, which is like the above but even worse. If you’re using algebraic data types, you wouldn’t use an optional. There are a number of ways a DB look up can fail so Optional isn’t really ideal at all. Your suggestion of Either/Result is probably the ideal there. Chances that you expect an explicit DB look up by ID to return nothing is pretty rare. Where has the program or client code got that ID from in the first place? Either you want to know about the error in your own code or you want the client code to know the error so you probably want some form of exception or Either. In the rare case it is intended behaviour, yeah you could use null and push the responsibility for handling it to the calling code. Still, at that point, 2 kicks in - might be fine now but that assumption could easily change as the code evolves, which it can and often does. You could argue you can implement something like Optional in, say, Java w/o algebraic data types but they are simply not ergonomic when the language does not help you use them. But I don’t think you can hold up a single approach and say that’s the ideal. I doubt it was your intention but it’s rather dogmatic and doesn’t really play out nicely in real world code, especially legacy projects (not everything is written in Haskell|Rust).
- kinow 5y ago
  I think he never considered the exceptions to the something to be avoided since he wrote Java most of his professional career - which ended when he decided to quit programming and go sell ornamental flowers in his family business in his hometown. The middleware we worked had libraries from other companies (e.g. online prepaid transactions were handled by code from a jar/lib from Ericsson). But in the case the platform threw an exception, we could still charge the customer using a slower process (i.e. there's a window of time where the customer could use data/talk in excess until we realize there's no more credit when processing the slower transaction). For him it was important to write these exceptions well. If something went wrong in production, we couldn't turn off the system, and if we were not charging users when we should to, the company would be losing millions (it was in Brazil, ~200 mi at the time, and the telco had ~60mi users I think, with mother's day and big brother being the craziest days with millions of messages per second.)

mindwok 5y ago

Personally I really dislike exceptions and try/catch in languages. I don't like having to worry about whether some function call is going to surprise me with an exception, and handling them with try/catch really breaks the flow of the program. I'm sure there's probably a lot I can learn to make this better for myself because I don't work with exception heavy code very often, but I find working with simple error value returns like in Go or error types in Haskell/Rust to be so much more ergonomic and comfortable to work with.

planet-and-halo 5y ago
I think the biggest place where it's really necessary is integration points. Within your system it's reasonable to try to define exceptions out of existence. Once you start accepting user input or depending upon some system outside of your control, though, you better have some kind of mechanism to handle whatever unknown asteroids come flying in from deep space ready to annihilate your entire planet and civilization.
- gizmo686 5y ago
  That is still provided by error values or error types. The defining feature of exceptions is that they provide a secondary control flow path, and that control flow path automatically flows up the stack until it reaches an explicit catch. With error values/types, control flow happens normally, and the programmer is expected to explicitly branch on the the value using normal control flow constructs.
rgharris 5y ago
Error handling in Go is simple and verbose at the same time. Working in it alongside Python and JavaScript makes me rethink a lot of patterns I’m accustomed to - it’s a nice exercise. Though of course the use cases often vary significantly between the 3 languages so it is tough to even compare.
hackinthebochs 5y ago
The benefit of exceptions and try/catch is that it lets you separate your exception handling logic from the mainline logic of the function. Having error logic weaved in and out of mainline logic just obscures what is going on and increases cognitive load. I much prefer writing and reading code with exceptions than explicit error handling control flow.

lmilcin 5y ago

> The longer the exception handling blocks in a file, the more likely the file is to contain bugs. It is difficult to design a sophisticated error handling system. For various reasons. For example it is rarely talked about -- everybody focuses on how to get stuff working and errors aren't really hot topic people find interesting. I think, given the above, you have most chance of success with really simple error handling systems. The simplest is to only handle things you can really handle at the current frame and let everything else filter to the top and interrupt entire process. It is not perfect but it has the virtue of being simple and easy to make foolproof. > The Ignoring Interrupted Exception and Log and Throw patterns corelated with post-release defects in one of the projects they studied, but not all. Interrupted exception does not happen in real usage in a lot of applications, especially backends, where the environment is very well controlled. Log and throw can be a legitimate pattern. It makes sense when the exception travels outside some kind of boundary like module/library boundary. A REST API is an example of boundary and it is normal to log the error but still throw it (to the client). It also may possibly make sense to log additional information available in local scope and throw the error up the stack. It doesn't always make sense to convert the type of the exception to be able to add more information to it.

gwbas1c 5y ago
> Log and throw can be a legitimate pattern. It makes sense when the exception travels outside some kind of boundary like module/library boundary. A REST API is an example of boundary and it is normal to log the error but still throw it (to the client). It also may possibly make sense to log additional information available in local scope and throw the error up the stack. It doesn't always make sense to convert the type of the exception to be able to add more information to it. Of course it's going to correlate with a bug. The whole point of log and throw is to make bugs easier to find!
TeMPOraL 5y ago
> It also may possibly make sense to log additional information available in local scope and throw the error up the stack. Like, time. This is why I often do "log and throw" or "log unexpected value and return it", depending on whether the code is exception-oriented or uses sum types instead. It's because time is an important piece of information to encode in an error log. When your error is caused by an external resource (e.g. RPC failure), you want to log it immediately to later be able to correlate it with external resource's logs or some other monitoring. If you don't the error may take some time to reach its final logging place - or, it may never reach it at all, if the application crashes in the meantime.

zaptheimpaler 5y ago

At my first job I wrote a new Java application to ingest/transform/store some data and after about 6 months it was done. It chugged along for years after without ever really requiring any maintenance or monitoring, it just kept working. One day the VP eng just showed up at my desk and started chatting, I didn't really know why. Turns out he was interested in knowing why it was so stable but it was an obvious part of software to me - every line I wrote where an exception was possible, I would look up what exceptions could be thrown and why, and then structure try catches to do the right thing in each case - sometimes that meant logging a warning, sometimes propagating to a higher level, sometimes shutting down the app etc. He then sent out some paper about the relationship between good error handling and software defects to the org. Like I don't get why this is so difficult for so many devs. Your job is to think about both the happy case and the error cases, and usually there are many error cases and 1 happy case so the bulk of time and code is dedicated to the errors. Since then I've seen this is not at all the way most devs write things. They are interested in writing the happy case and treat error handling like a chore to be done as quickly as possible. The tools and language constructs don't matter much, it's just do you dedicate time and care to thinking through errors or not. Yeah it sounds boastful but fuck it it's true. Same is true regardless of this study it's not about the structure and syntax of error handling,it's how much thought was given to understanding the failure modes and how to handle them.

MattPalmer1086 5y ago
Similarly, I've written systems that ran for years without needing much maintenance. I believe good error handling (and negative testing as well as positive testing) are the primary reasons for that. Unfortunately, it takes me longer to write code than my peers as a result. I'm not a professional developer anymore...
- zaptheimpaler 5y ago
  ouch i feel that. I do have the same problem all the time. if you don't mind me asking, what do you do now?
  - 5y ago

Too 5y ago

> there exist anti-patterns that can provide significant explanatory power to the probability > of post-release defects. > Therefore, development teams should consider allocating > more resources to improving their exception handling practices While an interesting correlation. The conclusion that plugging this particular hole might not be the best goal to focus on improving. It's two variables that are not statistically independent of each other. A novice developer is more likely to write bugs as they are to use anti-patterns of exception handling, or anti-patterns for anything else for that matter. Basically correlation is not causation.

Arech 5y ago

The statistician in me is severely troubled by a conclusion like "statistical relationship" when the conclusion is drawn from studying just 3 (three) projects. At worst case this means studying programming habits of just 3 (three) developers (appended: actually, 1 dev is the worst case). You just can't make any significant conclusions on such basis that generalize on whole population. Even if it was huge projects with hundreds of devs, you still can't make generalizable conclusions, because habits of all this devs are clearly not independent.

branko_d 5y ago

On the exception receiving side... The key, IMO, is to assume that every line of code can throw until proven otherwise, and then make sure you clean-up after yourself, using RAII in C++, `using` in C#, savepoints in SQL or whatever else mechanism is available in your language. After that, in 99% cases, you just let the exception propagate to the higher level. Eventually, it will result in an error dialog, or be logged or whatever, but the program is still in consistant state. It's a kind of a "soft reset" that doesn't destroy the state and lets you keep working. --- On the exception producing side... Don't throw a new exception unles you expect it can be reasonably treated as described above. If your immediate caller needs a `try...catch` wrapped directly around the call, this is probably a sign you should have returned an error result instead of throwing an exception.