3 ms·

I'm not sure what to take away from this post. Unit tests just saved my bacon when I accidentally introduced a serious bug in refactoring some network-related

I'm not sure what to take away from this post. Unit tests just saved my bacon when I accidentally introduced a serious bug in refactoring some network-related code. The bug wouldn't have shown up outside of production in some very specific cases, but it would have had a serious customer impact. Unit tests catch these things for me all the time. Every 5 minutes I spend writing tests easily saves me 30+ minutes I'd spend debugging when some code 6 levels down from the apparent point of failure caused the issue.

jshen 17y ago
I wonder why the apparent point of failure is so far removed from the actual point of failure. With my work that is rarely the case, and I wonder which of us is the anomaly. Don't get me wrong, I write tests, but not nearly as many as a lot of people propose. I do a lot of social sites, so the context may be different, but it's not uncommon for a release to introduce a bug immediately comes to my attention when a user hits it. I can usually deploy a fix very fast, and I feel this is a more effective approach than maintaining a test set approaching comprehensive coverage. People usually fail to mention the time it takes to maintain such a test set over time, which is higher than the time it takes to write the initial test. Note, I always thoroughly test high impact pieces of code. i.e. account creation, anything involving money or personal information, and data validation code. Having said that, the bulk of the code is controller and view code which isn't earth shattering if it breaks for 10 minutes.
- chadaustin 17y ago
  I've noticed that code connectedness has a huge impact on the kind of test coverage needed. For example, if your website back-end is fairly stable and the API rarely changes, then you don't typically need 100% code coverage on each page. As you mentioned, if you fix a bug, it won't randomly break later... On the other hand, with a bunch of connected objects and events that fire, causing other events... it's very possible for a change in one system to affect others. In this situation, I recommend a ton of unit tests _and_ acceptance tests. In addition, decouple the components in the system and stabilize your base APIs. This will allow you to write a new feature without affecting stability elsewhere.
  - DrJokepu 17y ago
    I also noticed that (at least in the code I write) bugs are much more prevalent in "boring code" than in "smart code"; that is, if I got some elegant but rather advanced piece of code I had put a lot of thought into (like some smart math stuff or something similar), it almost always works right away without any bugs whatsoever, however, boring and tedious-to-write code (such as wiring up GUI, integration glue code etc) tends to have a lot more bugs in it. Unfortunately, this first type of code is a lot easier to test automatically than the second type. Smart code usually has clean and simple interfaces, while writing tests for "boring code" is also boring and tedious, like writing a test for each textbox on a form etc.
    - chadaustin 17y ago
      Maybe there's a way to automate the boring code? Write a system that lets you build UI without manually specifying the behavior of every control, etc.
- antonovka 17y ago
  I wonder why the apparent point of failure is so far removed from the actual point of failure. With my work that is rarely the case, and I wonder which of us is the anomaly. Neither of us, it just sounds like we just work on different software with varying levels of complexity.
- akeefer 17y ago
  The tradeoffs largely depend on 1) your deployment model and 2) the cost of the bug itself. If you're running a web service, diagnosing and fixing bugs is a whole lot easier than if you're shipping pre-packaged, installed software. If the cost of failure is just "some user action failed, they'll try again when it works" it's far less critical to prevent it than if it's "your data is now corrupted." Something like a social media site is on one end of the scale, where you're running a service and where most bugs are non-fatal. Installed backend financial transactions systems for large companies, for example, are at the other end of the scale, where fixing bugs in deployed software is hard (you might not have the information to really diagnose the failure or the ability to reproduce it in-house, and the fix will require a patch that someone else has to apply and deploy), and the failure can be incredibly costly.
zmimon 17y ago
Can you say for sure that a good suite of functional tests wouldn't have saved your bacon as well? And they might have been much easier to write and also catch a whole lot broader set of problems at the same time. The post isn't advocating having NO tests. It's not even saying that unit tests don't work. Just that they're not necessarily the best value in every situation.

jshen 17y ago

I wonder why the apparent point of failure is so far removed from the actual point of failure. With my work that is rarely the case, and I wonder which of us is the anomaly. Don't get me wrong, I write tests, but not nearly as many as a lot of people propose. I do a lot of social sites, so the context may be different, but it's not uncommon for a release to introduce a bug immediately comes to my attention when a user hits it. I can usually deploy a fix very fast, and I feel this is a more effective approach than maintaining a test set approaching comprehensive coverage. People usually fail to mention the time it takes to maintain such a test set over time, which is higher than the time it takes to write the initial test. Note, I always thoroughly test high impact pieces of code. i.e. account creation, anything involving money or personal information, and data validation code. Having said that, the bulk of the code is controller and view code which isn't earth shattering if it breaks for 10 minutes.

chadaustin 17y ago
I've noticed that code connectedness has a huge impact on the kind of test coverage needed. For example, if your website back-end is fairly stable and the API rarely changes, then you don't typically need 100% code coverage on each page. As you mentioned, if you fix a bug, it won't randomly break later... On the other hand, with a bunch of connected objects and events that fire, causing other events... it's very possible for a change in one system to affect others. In this situation, I recommend a ton of unit tests _and_ acceptance tests. In addition, decouple the components in the system and stabilize your base APIs. This will allow you to write a new feature without affecting stability elsewhere.
- DrJokepu 17y ago
  I also noticed that (at least in the code I write) bugs are much more prevalent in "boring code" than in "smart code"; that is, if I got some elegant but rather advanced piece of code I had put a lot of thought into (like some smart math stuff or something similar), it almost always works right away without any bugs whatsoever, however, boring and tedious-to-write code (such as wiring up GUI, integration glue code etc) tends to have a lot more bugs in it. Unfortunately, this first type of code is a lot easier to test automatically than the second type. Smart code usually has clean and simple interfaces, while writing tests for "boring code" is also boring and tedious, like writing a test for each textbox on a form etc.
  - chadaustin 17y ago
    Maybe there's a way to automate the boring code? Write a system that lets you build UI without manually specifying the behavior of every control, etc.
antonovka 17y ago
I wonder why the apparent point of failure is so far removed from the actual point of failure. With my work that is rarely the case, and I wonder which of us is the anomaly. Neither of us, it just sounds like we just work on different software with varying levels of complexity.
akeefer 17y ago
The tradeoffs largely depend on 1) your deployment model and 2) the cost of the bug itself. If you're running a web service, diagnosing and fixing bugs is a whole lot easier than if you're shipping pre-packaged, installed software. If the cost of failure is just "some user action failed, they'll try again when it works" it's far less critical to prevent it than if it's "your data is now corrupted." Something like a social media site is on one end of the scale, where you're running a service and where most bugs are non-fatal. Installed backend financial transactions systems for large companies, for example, are at the other end of the scale, where fixing bugs in deployed software is hard (you might not have the information to really diagnose the failure or the ability to reproduce it in-house, and the fix will require a patch that someone else has to apply and deploy), and the failure can be incredibly costly.

zmimon 17y ago

Can you say for sure that a good suite of functional tests wouldn't have saved your bacon as well? And they might have been much easier to write and also catch a whole lot broader set of problems at the same time. The post isn't advocating having NO tests. It's not even saying that unit tests don't work. Just that they're not necessarily the best value in every situation.