8 ms·

It's OK Not to Write Unit Tests

103 points by bensummers 17y ago

dschobel 17y ago
If you're a Java programmer and want to have a rude awakening, go download Jester. Jester is an automated mutation testing tool—it goes in and replaces “<=” with “<”, “&&” with “||”, “!” with whitespace. And then it re-runs the tests. And then you get to watch in horror as your tests still all pass, regardless of what the product code actually does. I've always wanted to write a unit-test testing library which inverts all of the assertions to catch the false positives. This sounds like it's a lot more sadistic (and useful) though.
- gruseom 17y ago
  That is indeed a devilishly clever idea.
- petercooper 17y ago
  A tip for Rubyists.. there's a library that does stuff like this: http://ruby.sadi.st/Heckle.html http://ruby.sadi.st/Heckle.html
- bbest86 17y ago
  Would be interesting to automatically run something like this and find a way to assert that the tests should fail. Almost an automated test of your tests.
  - ZeroGravitas 17y ago
    Unless I'm misunderstanding something, that is what these type of tools do.
chadaustin 17y ago
IMVU did not consistently write unit tests on day one. So why are we all rabid unit testing / test-driven development advocates now? http://www.startuplessonslearned.com/2008/11/five-whys.html http://www.startuplessonslearned.com/2008/11/five-whys.html Every time you ship a bug to customers, ask yourself "How will we prevent this bug from ever reaching customers again?" and "Why did we write this bug in the first place?" Almost every time, unit or acceptance tests would have caught the bug initially or prevented the regression. That's not to say unit tests are the only solution. Sometimes: * you need better knowledge sharing between engineers * you should spend more time refactoring your code * your internal APIs are confusing and inherently cause bugs Often a follow-up to a bug will involve changes to all of the above. When you start an experimental project, don't focus on automated tests if you don't need to... I'm all for lazily pulling in the processes you need as you need them. :) Whether you write unit tests or not is entirely context-sensitive. The author's points might make sense in some context, but they certainly don't apply to every team or even every engineer. Maybe this guideline is helpful: "I just wrote a new component. What kind of test coverage do I need to allow a new engineer to refactor arbitrarily and still be convinced my code works?"
- jshen 17y ago
  "How will we prevent this bug from ever reaching customers again?" I think the cost of this is far higher than it's worth.
  - j_baker 17y ago
    Are you a programmer presently? That works on a codebase that's at least in the tens of thousands of lines of code? Tell me what software you write so that I know to never, ever use it.
    - nkohari 17y ago
      I'll play along. Have you ever owned your own business, and had to make the trade off between absolute quality and efficiency?
      - rbanffy 17y ago
        The one between correct and works-for-the-customer?
    - jshen 17y ago
      lines of code is very dependent on the language. I have one rails app that's in the 5k range, excluding tests, and it's been running for over a year with no restarts and no need for bug fixes. I think that says something about the effectiveness of my process. I question the design decisions of any code base with tens of thousands lines of code.
      - munctional 17y ago
        If your app has been running for over a year, you might want to look into this (before someone else does)... http://groups.google.com/group/rubyonrails-security/msg/7f57cd7794e1d1b4?pli=1 http://groups.google.com/group/rubyonrails-security/msg/7f57...
        17y ago
antonovka 17y ago
I'm not sure what to take away from this post. Unit tests just saved my bacon when I accidentally introduced a serious bug in refactoring some network-related code. The bug wouldn't have shown up outside of production in some very specific cases, but it would have had a serious customer impact. Unit tests catch these things for me all the time. Every 5 minutes I spend writing tests easily saves me 30+ minutes I'd spend debugging when some code 6 levels down from the apparent point of failure caused the issue.
- jshen 17y ago
  I wonder why the apparent point of failure is so far removed from the actual point of failure. With my work that is rarely the case, and I wonder which of us is the anomaly. Don't get me wrong, I write tests, but not nearly as many as a lot of people propose. I do a lot of social sites, so the context may be different, but it's not uncommon for a release to introduce a bug immediately comes to my attention when a user hits it. I can usually deploy a fix very fast, and I feel this is a more effective approach than maintaining a test set approaching comprehensive coverage. People usually fail to mention the time it takes to maintain such a test set over time, which is higher than the time it takes to write the initial test. Note, I always thoroughly test high impact pieces of code. i.e. account creation, anything involving money or personal information, and data validation code. Having said that, the bulk of the code is controller and view code which isn't earth shattering if it breaks for 10 minutes.
  - chadaustin 17y ago
    I've noticed that code connectedness has a huge impact on the kind of test coverage needed. For example, if your website back-end is fairly stable and the API rarely changes, then you don't typically need 100% code coverage on each page. As you mentioned, if you fix a bug, it won't randomly break later... On the other hand, with a bunch of connected objects and events that fire, causing other events... it's very possible for a change in one system to affect others. In this situation, I recommend a ton of unit tests _and_ acceptance tests. In addition, decouple the components in the system and stabilize your base APIs. This will allow you to write a new feature without affecting stability elsewhere.
jon_dahl 17y ago
Summary: another post saying that "unit tests aren't a panacea". Which is true, of course. But that doesn't mean that unit testing isn't terribly useful.
- joe_the_user 17y ago
  Uh, He actually makes some subsidiary points that are worthwhile and don't just fit mold. * A lot of tests people write aren't really unit tests * Unit tests aren't the armor that protects you from error during refactoring. * Unit test don't necessarily improve design, indeed they can have a bad effect on design. So, there's more to the article than the summary - read the article, I think it has merit.
petercooper 17y ago
I'm a relative newcomer to unit testing and man.. has it saved me a lot of time already. Forgetting the gigantic advantages it gives you over mere bugs, I'm just as impressed at how it helps you resolve higher level issues. For example, recently I moved a project I was working on months ago from one machine to another. Tests had lots of failures, despite no changes. Turns out I had an unpatched library installed, a few libraries of the wrong version, a slightly different database version, stuff like that. My tests picked all that up and I could vendorize stuff or tighten up my dependencies. Without the tests, I'd have been scratching my head for ages figuring out "what" wasn't working.
- forensic 17y ago
  >Without the tests, I'd have been scratching my head for ages figuring out "what" wasn't working. The real question though is how all that time you spent writing the tests compares to the time you'd have spent troubleshooting.
  - jon_dahl 17y ago
    Have you done much unit testing? Done right, writing unit tests often doesn't take any extra time at all.
    - munctional 17y ago
      I disagree. We spend a considerable amount of time writing our behavioural tests (we have around 1,700 for our large web app), but they have saved us countless times from having to track down bugs after deploying code. There was an article published in the last few weeks which showed that teams who write tests spend 10-15% longer in the development process, but they also save 25-30% of the development process time by not having to fix regression defects. Unfortunately, I don't remember which site it was linked from or what the article's title was. If I can find it, I'll link it here.
    - bphogan 17y ago
      I have to agree with Jon. Really, once you get into the cycle, and you write them first, it doesn't tend to add much time. It does, however, really hurt your productivity when you start doing TDD. It's akin to the hit you take when you move to a new programming language. You'll feel faster if you don't do it. But as you get more comfortable and get into a rhythm, it can be fairly insignificant. Just don't get ahead of yourself, and don't let the tests get out of sync. Rails is a good example where this sync stuff is hard. It's wicked easy to use things like associations and named scopes to query records while forgetting to write a test to ensure that those queries actually return what you specify. But keep in mind that tests aren't just for you. They're for your team, and the next guy who follows you. You can learn a lot about intent from tests too.
MartinCron 17y ago
Wow, people are still writing more and more words on unit testing with absolutely no new ideas. Nothing to see here. Move along.
- rbanffy 17y ago
  Quite the contrary. This attitude in a MSDN blog, from a Microsoft employee explain a lot of problems their users have. Way to go, Microsoft. Testing is for sissies! Real Men write correct code. Now I can use Exchange and know why it sucks so bad.
  - MartinCron 17y ago
    What one random developer writes on his MSDN blog doesn't mean anything more than what any other random developer writes on an MSDN blog. Microsoft is way too big and has way too many developers to draw conclusions like that. I've done consulting work for Microsoft, and I've never conceptualized them as one company with one voice. Microsoft is more like lots and lots of little companies which run the entire spectrum from total dysfunction to jaw-dropping awesomeness.
  - j_baker 17y ago
    "Testing is for sissies! Real Men write correct code." I seriously laughed when I read this. I'm willing to sacrifice the extra couple of inches of programmer penis size by writing tests.
    - rbanffy 17y ago
      You can also refrain from writing the bugs ;-)
    - forensic 17y ago
      But the tests themselves _are_ code. Do you have tests that check the tests? And then tests that check the tests that check the tests? At one point or another you just have to learn to live with bugs.
j_baker 17y ago
Being a huge testing advocatem I agree with this post ...for the most part (and it's easy enough to disagree with if you didn't read it all the way). There are just certain parts of code that are just more trouble than they're worth to test. However, I'm surprised at just how small an amount of code that really is. In fact, it's really difficult to say what those portions of code are unless you stick to TDD. Another thing that I've noticed is that not only do functional languages lead to a more testable design, but testable code tends to be more functional in nature. So perhaps tests are design tools. Plus, while unit tests really only catch a small number of bugs (I think studies have shown something like 25%), they're the fastest way of catching and preventing the bugs they prevent. Words can't describe how frustrated I get when I deploy to the test server and get an error because I mistyped a variable name in some obscure part of my code. Some times I think programming in a dynamic language without unit tests is like screwing without a condom. Sure it feels good for a while, but eventually it will come back to bite you (no pun intended).
deleted 17y ago
[deleted]
Freebytes 17y ago
The book that changed my life: http://www.amazon.com/Clean-Code-Handbook-Software-Craftsmanship/dp/0132350882 http://www.amazon.com/Clean-Code-Handbook-Software-Craftsman... "Clean Code" by Robert Martin is one of the best programming books ever written, and it barely has any code in it. However, it is relevant for every programming language. He treats unit tests like mandatory rituals, though, and unfortunately, I have not progressed quite that far yet. Still, it is a must read and gives good insights into this topic.
- plinkplonk 17y ago
  "one of the best programming books ever written, and it barely has any code in it." Sounds very fishy, but from Robert Martin, who thinks people who don't do TDD (not unit testing, TDD) are "unprofessional" and "stone age programmers", I wouldn't expect any less. "He treats unit tests like mandatory rituals," cargo cult alert! fwiw my conclusion based on my experience, unit tests are (by and large)a good idea. Test Driven Development is dubious, especially the "TDD is a design method" idea. Conflating the two notions isn't very useful but happens surprisingly often.
  - bphogan 17y ago
    Even if you don't get into the TDD stuff he evangelizes, it's a pretty darn good read. It has some really thought-provoking things to think about in terms of how to write your code. I wouldn't dismiss it offhandedly.
wisty 17y ago
Unit tests have another advantage - they keep you coding and motivated without an upfront schedule. Sure, you should be motivated already. Why was it you were reading hacker news then?
- dmoney 17y ago
  I find unit testing to be a drag sometimes. Creating a class is much easier than creating code to reproduce every situation it can be in. Then there's troubleshooting random failures that turn out to be bugs in the test rather than the code being tested. In what way does it keep you motivated?
- mfukar 17y ago
  I got tired of that argument before I even read it. If you're coding non-stop, without distractions, for the whole of your work day, teach us how to do it.
danek 17y ago
re: pains of unit tests when refactoring -- i.e. my experience with poorly written unit tests i took over a project once from a crazy person that had a ton of unit tests in it. about 50% of the tests were dumb, so i deleted those (seriously, things like instantiating an object and then testing that it wasn't null). I tried to maintain the rest of the tests as I worked with the code, but what I found was that there were so many tests, and most of them were testing the most pointless trivial things, that keeping them around was taking more time than it was worth, so I axed those too. the remaining tests that actually tested stuff, were testing parts of the design that were done so ridiculously that they needed unit tests. So I just got rid of all the tests and refactored the hell out of the code. I ran it through the debugger a few times and all was good. the end. oh, then a few months later the company changed its business model and they didn't need that subsystem anymore. go figure.
JoeAltmaier 17y ago
My mantra: "if you haven't tried it, it doesn't work" Unit test, dry run, big bang, but in the end no significant piece of code works the first time. Ok, once or twice in my life it has, and I still talk about that experience. But that just shows how necessary testing is most of the time.

103 points by bensummers 17y ago

If you're a Java programmer and want to have a rude awakening, go download Jester. Jester is an automated mutation testing tool—it goes in and replaces “<=” with “<”, “&&” with “||”, “!” with whitespace. And then it re-runs the tests. And then you get to watch in horror as your tests still all pass, regardless of what the product code actually does. I've always wanted to write a unit-test testing library which inverts all of the assertions to catch the false positives. This sounds like it's a lot more sadistic (and useful) though.

gruseom 17y ago
That is indeed a devilishly clever idea.
petercooper 17y ago
A tip for Rubyists.. there's a library that does stuff like this: http://ruby.sadi.st/Heckle.html http://ruby.sadi.st/Heckle.html
bbest86 17y ago
Would be interesting to automatically run something like this and find a way to assert that the tests should fail. Almost an automated test of your tests.
- ZeroGravitas 17y ago
  Unless I'm misunderstanding something, that is what these type of tools do.

chadaustin 17y ago

IMVU did not consistently write unit tests on day one. So why are we all rabid unit testing / test-driven development advocates now? http://www.startuplessonslearned.com/2008/11/five-whys.html http://www.startuplessonslearned.com/2008/11/five-whys.html Every time you ship a bug to customers, ask yourself "How will we prevent this bug from ever reaching customers again?" and "Why did we write this bug in the first place?" Almost every time, unit or acceptance tests would have caught the bug initially or prevented the regression. That's not to say unit tests are the only solution. Sometimes: * you need better knowledge sharing between engineers * you should spend more time refactoring your code * your internal APIs are confusing and inherently cause bugs Often a follow-up to a bug will involve changes to all of the above. When you start an experimental project, don't focus on automated tests if you don't need to... I'm all for lazily pulling in the processes you need as you need them. :) Whether you write unit tests or not is entirely context-sensitive. The author's points might make sense in some context, but they certainly don't apply to every team or even every engineer. Maybe this guideline is helpful: "I just wrote a new component. What kind of test coverage do I need to allow a new engineer to refactor arbitrarily and still be convinced my code works?"

jshen 17y ago
"How will we prevent this bug from ever reaching customers again?" I think the cost of this is far higher than it's worth.
- j_baker 17y ago
  Are you a programmer presently? That works on a codebase that's at least in the tens of thousands of lines of code? Tell me what software you write so that I know to never, ever use it.
  - nkohari 17y ago
    I'll play along. Have you ever owned your own business, and had to make the trade off between absolute quality and efficiency?
    - rbanffy 17y ago
      The one between correct and works-for-the-customer?
  - jshen 17y ago
    lines of code is very dependent on the language. I have one rails app that's in the 5k range, excluding tests, and it's been running for over a year with no restarts and no need for bug fixes. I think that says something about the effectiveness of my process. I question the design decisions of any code base with tens of thousands lines of code.
    - munctional 17y ago
      If your app has been running for over a year, you might want to look into this (before someone else does)... http://groups.google.com/group/rubyonrails-security/msg/7f57cd7794e1d1b4?pli=1 http://groups.google.com/group/rubyonrails-security/msg/7f57...
      - 17y ago

antonovka 17y ago

I'm not sure what to take away from this post. Unit tests just saved my bacon when I accidentally introduced a serious bug in refactoring some network-related code. The bug wouldn't have shown up outside of production in some very specific cases, but it would have had a serious customer impact. Unit tests catch these things for me all the time. Every 5 minutes I spend writing tests easily saves me 30+ minutes I'd spend debugging when some code 6 levels down from the apparent point of failure caused the issue.

jshen 17y ago
I wonder why the apparent point of failure is so far removed from the actual point of failure. With my work that is rarely the case, and I wonder which of us is the anomaly. Don't get me wrong, I write tests, but not nearly as many as a lot of people propose. I do a lot of social sites, so the context may be different, but it's not uncommon for a release to introduce a bug immediately comes to my attention when a user hits it. I can usually deploy a fix very fast, and I feel this is a more effective approach than maintaining a test set approaching comprehensive coverage. People usually fail to mention the time it takes to maintain such a test set over time, which is higher than the time it takes to write the initial test. Note, I always thoroughly test high impact pieces of code. i.e. account creation, anything involving money or personal information, and data validation code. Having said that, the bulk of the code is controller and view code which isn't earth shattering if it breaks for 10 minutes.
- chadaustin 17y ago
  I've noticed that code connectedness has a huge impact on the kind of test coverage needed. For example, if your website back-end is fairly stable and the API rarely changes, then you don't typically need 100% code coverage on each page. As you mentioned, if you fix a bug, it won't randomly break later... On the other hand, with a bunch of connected objects and events that fire, causing other events... it's very possible for a change in one system to affect others. In this situation, I recommend a ton of unit tests _and_ acceptance tests. In addition, decouple the components in the system and stabilize your base APIs. This will allow you to write a new feature without affecting stability elsewhere.

jon_dahl 17y ago

Summary: another post saying that "unit tests aren't a panacea". Which is true, of course. But that doesn't mean that unit testing isn't terribly useful.

joe_the_user 17y ago
Uh, He actually makes some subsidiary points that are worthwhile and don't just fit mold. * A lot of tests people write aren't really unit tests * Unit tests aren't the armor that protects you from error during refactoring. * Unit test don't necessarily improve design, indeed they can have a bad effect on design. So, there's more to the article than the summary - read the article, I think it has merit.

petercooper 17y ago

I'm a relative newcomer to unit testing and man.. has it saved me a lot of time already. Forgetting the gigantic advantages it gives you over mere bugs, I'm just as impressed at how it helps you resolve higher level issues. For example, recently I moved a project I was working on months ago from one machine to another. Tests had lots of failures, despite no changes. Turns out I had an unpatched library installed, a few libraries of the wrong version, a slightly different database version, stuff like that. My tests picked all that up and I could vendorize stuff or tighten up my dependencies. Without the tests, I'd have been scratching my head for ages figuring out "what" wasn't working.

forensic 17y ago
>Without the tests, I'd have been scratching my head for ages figuring out "what" wasn't working. The real question though is how all that time you spent writing the tests compares to the time you'd have spent troubleshooting.
- jon_dahl 17y ago
  Have you done much unit testing? Done right, writing unit tests often doesn't take any extra time at all.
  - munctional 17y ago
    I disagree. We spend a considerable amount of time writing our behavioural tests (we have around 1,700 for our large web app), but they have saved us countless times from having to track down bugs after deploying code. There was an article published in the last few weeks which showed that teams who write tests spend 10-15% longer in the development process, but they also save 25-30% of the development process time by not having to fix regression defects. Unfortunately, I don't remember which site it was linked from or what the article's title was. If I can find it, I'll link it here.
  - bphogan 17y ago
    I have to agree with Jon. Really, once you get into the cycle, and you write them first, it doesn't tend to add much time. It does, however, really hurt your productivity when you start doing TDD. It's akin to the hit you take when you move to a new programming language. You'll feel faster if you don't do it. But as you get more comfortable and get into a rhythm, it can be fairly insignificant. Just don't get ahead of yourself, and don't let the tests get out of sync. Rails is a good example where this sync stuff is hard. It's wicked easy to use things like associations and named scopes to query records while forgetting to write a test to ensure that those queries actually return what you specify. But keep in mind that tests aren't just for you. They're for your team, and the next guy who follows you. You can learn a lot about intent from tests too.

MartinCron 17y ago

Wow, people are still writing more and more words on unit testing with absolutely no new ideas. Nothing to see here. Move along.

rbanffy 17y ago
Quite the contrary. This attitude in a MSDN blog, from a Microsoft employee explain a lot of problems their users have. Way to go, Microsoft. Testing is for sissies! Real Men write correct code. Now I can use Exchange and know why it sucks so bad.
- MartinCron 17y ago
  What one random developer writes on his MSDN blog doesn't mean anything more than what any other random developer writes on an MSDN blog. Microsoft is way too big and has way too many developers to draw conclusions like that. I've done consulting work for Microsoft, and I've never conceptualized them as one company with one voice. Microsoft is more like lots and lots of little companies which run the entire spectrum from total dysfunction to jaw-dropping awesomeness.
- j_baker 17y ago
  "Testing is for sissies! Real Men write correct code." I seriously laughed when I read this. I'm willing to sacrifice the extra couple of inches of programmer penis size by writing tests.
  - rbanffy 17y ago
    You can also refrain from writing the bugs ;-)
  - forensic 17y ago
    But the tests themselves _are_ code. Do you have tests that check the tests? And then tests that check the tests that check the tests? At one point or another you just have to learn to live with bugs.

j_baker 17y ago

Being a huge testing advocatem I agree with this post ...for the most part (and it's easy enough to disagree with if you didn't read it all the way). There are just certain parts of code that are just more trouble than they're worth to test. However, I'm surprised at just how small an amount of code that really is. In fact, it's really difficult to say what those portions of code are unless you stick to TDD. Another thing that I've noticed is that not only do functional languages lead to a more testable design, but testable code tends to be more functional in nature. So perhaps tests are design tools. Plus, while unit tests really only catch a small number of bugs (I think studies have shown something like 25%), they're the fastest way of catching and preventing the bugs they prevent. Words can't describe how frustrated I get when I deploy to the test server and get an error because I mistyped a variable name in some obscure part of my code. Some times I think programming in a dynamic language without unit tests is like screwing without a condom. Sure it feels good for a while, but eventually it will come back to bite you (no pun intended).

deleted 17y ago

[deleted]

Freebytes 17y ago

The book that changed my life: http://www.amazon.com/Clean-Code-Handbook-Software-Craftsmanship/dp/0132350882 http://www.amazon.com/Clean-Code-Handbook-Software-Craftsman... "Clean Code" by Robert Martin is one of the best programming books ever written, and it barely has any code in it. However, it is relevant for every programming language. He treats unit tests like mandatory rituals, though, and unfortunately, I have not progressed quite that far yet. Still, it is a must read and gives good insights into this topic.

plinkplonk 17y ago
"one of the best programming books ever written, and it barely has any code in it." Sounds very fishy, but from Robert Martin, who thinks people who don't do TDD (not unit testing, TDD) are "unprofessional" and "stone age programmers", I wouldn't expect any less. "He treats unit tests like mandatory rituals," cargo cult alert! fwiw my conclusion based on my experience, unit tests are (by and large)a good idea. Test Driven Development is dubious, especially the "TDD is a design method" idea. Conflating the two notions isn't very useful but happens surprisingly often.
- bphogan 17y ago
  Even if you don't get into the TDD stuff he evangelizes, it's a pretty darn good read. It has some really thought-provoking things to think about in terms of how to write your code. I wouldn't dismiss it offhandedly.

wisty 17y ago

Unit tests have another advantage - they keep you coding and motivated without an upfront schedule. Sure, you should be motivated already. Why was it you were reading hacker news then?

dmoney 17y ago
I find unit testing to be a drag sometimes. Creating a class is much easier than creating code to reproduce every situation it can be in. Then there's troubleshooting random failures that turn out to be bugs in the test rather than the code being tested. In what way does it keep you motivated?
mfukar 17y ago
I got tired of that argument before I even read it. If you're coding non-stop, without distractions, for the whole of your work day, teach us how to do it.

danek 17y ago

re: pains of unit tests when refactoring -- i.e. my experience with poorly written unit tests i took over a project once from a crazy person that had a ton of unit tests in it. about 50% of the tests were dumb, so i deleted those (seriously, things like instantiating an object and then testing that it wasn't null). I tried to maintain the rest of the tests as I worked with the code, but what I found was that there were so many tests, and most of them were testing the most pointless trivial things, that keeping them around was taking more time than it was worth, so I axed those too. the remaining tests that actually tested stuff, were testing parts of the design that were done so ridiculously that they needed unit tests. So I just got rid of all the tests and refactored the hell out of the code. I ran it through the debugger a few times and all was good. the end. oh, then a few months later the company changed its business model and they didn't need that subsystem anymore. go figure.

JoeAltmaier 17y ago

My mantra: "if you haven't tried it, it doesn't work" Unit test, dry run, big bang, but in the end no significant piece of code works the first time. Ok, once or twice in my life it has, and I still talk about that experience. But that just shows how necessary testing is most of the time.