6 ms·

Working Effectively with Unit Tests

83 points by GarethX 12y ago

struppi 12y ago
A nice article, and I think I should add the book to my (already too large) to-read pile. I totally agree with "DRY might not be great in all cases" - In tests, it's often better to make each single test as expressive as possible. I once read (and forgot who said it): "Code should be either DRY (don't repeat yourself) or WET (write expressive tests), but never damp)" And here's a shameless plug: I recently wrote an article about how tests can help you achieve simple design, and the next article in the series will be coming soon: http://quickglance.at/en/simple_design/passes_its_tests http://quickglance.at/en/simple_design/passes_its_tests
- zak_mc_kracken 12y ago
  > "Code should be either DRY (don't repeat yourself) or WET (write expressive tests), but never damp) I'm getting so tired of people who try to capture the intricacies and complexities of our profession with witty one liners that have to be taken like dogma. Just use your common sense, let your experience guide you and if you're not sure about something, ask your teammates.
  - struppi 12y ago
    Who says that they have to be taken like dogma? Sometimes the one-liners are just easy to remember. Often they are funny. They can be a great tool to quickly illustrate a point or start a conversation. The one liners can never replace a conversation or explanation. They cannot replace common sense or real knowlege. I would never treat them as laws or follow them by the letter...
  - Gupie 12y ago
    > Just use your common sense, let your experience guide you and if you're not sure about something, ask your teammates. Brilliant, added to our bugzilla quips list.
vowelless 12y ago
I wish the audio quality was a little better. Is there any post process work that can be done to improve the quality?
- GarethX 12y ago
  Sadly that's with post-processing improvements. Without the guests recording their own audio and then syncing it after (which is something of a big ask), we're reliant on the quality of the connection. Something to work on for sure. I'm hopeful solutions like http://podclear.com http://podclear.com can help.
lukasm 12y ago
You may want to invest a few bucks in a good mic.
- GarethX 12y ago
  Yeah, sorry about that. Derrick has one, but was moving house at the time of recording and it was packed away. So future ones should be using that, at least on the host side.
collyw 12y ago
This is one of the most sensible articles I have seen on unit testing. Rather than religiously insist on tests first or 100% coverage, judge when it is valuable to have these things and when they are not important. Far too many articles on testing test the framework / language being used.
zak_mc_kracken 12y ago
Terrible audio. Next time you interview someone, ask them to install Audacity and to run it during the interview. Then have them send you the audio file via Google Drive.
colinramsay 12y ago
I constantly have problems with people posting the same information on unit tests, again and again. I know how to set up (for example) RSpec, MiniTest, Cucumber, FactoryGirl. The thing that people struggle with is how to use these in a larger, practical sense, rather than in just the trivial. I think this puts a lot of people off testing.
- Tyguy7 12y ago
  Agreed. I have only used TDD once as a front-end web developer, and I've found that I do struggle with the practicality of it. I spent half my time managing and maintaining my tests. When I did the math, It seemed that all that extra time spent was more than It would have taken to simply chase down bugs that would have arisen in a non-tdd environment. It just seems to take more time than is practical. Especially in agile startups.
  - CmonDev 12y ago
    Actually in this case he seems to be past his personal TDD hype peak: "... a proponent of the selective use of Test-Driven Development." Hopefully this will become a mainstream thought. Then maybe we will fix the legacy browser languages problem. Man can dream...
  - ZeroGravitas 12y ago
    Is the only metric your time spent preventing/chasing bugs? What if those bugs get out to the customer and impact them?
  - deleted 12y ago
    [deleted]
  - atilaneves 12y ago
    Perhaps. The difference is that in one case you'll have the machine automatically tell you when and why your code stops working because of a regression, and in the other you have no clue until the customer hits it. It's not only about proving it worked once, it's about proving it with every commit.
  - 12y ago
henrik_w 12y ago
I like Jay's comments about how tests are wasted if you don't know exactly what you want, and just try out different things (because then you need to change/delete the tests when you end up changing your code). As for common mistakes with unit tests, my top 5 list is: 1. Testing algorithms together with coordinators. 2. Mocking too much. 3. Not using asserts. 4. Leaving print statements in the tests. 5. Checking the log statements, not the result. More details here: 5 Unit Testing Mistakes http://henrikwarne.com/2014/02/19/5-unit-testing-mistakes/ http://henrikwarne.com/2014/02/19/5-unit-testing-mistakes/
- Iftheshoefits 12y ago
  Here is another one: failing to understand that tests introduce additional maintenance and design overhead. Quis custodiet ipsos custodes? Tests are additional code that is tightly coupled to the target code that is meant to be used in production. This additional code requires design, debugging, and maintenance just like the target code. It is also tightly coupled to the target codebase: a change in either necessitates at least an inspection to determine what, if any, changes must then be made in the other. This all adds overhead to the whole process, and too often I see casual, glib statements implying or even asserting that the overhead is nonexistent or else pays for itself. It isn't free and it is especially not free the more dogmatic the test proponents and tests themselves are.
  - henrik_w 12y ago
    Agreed. I think Jay touches on this when talking about how you don't need to aim for 100% unit test coverage. It's also what I mean by testing algorithmic code (where the pay off from the effort is large), versus testing coordinator code, where you are better off not unit testing it (because it is not complicated).
    - rane 12y ago
      What's coordinator code like?
      - 12y ago
encoderer 12y ago
Much of what passes for testing is a near-criminal waste of corporate resources. I've seen so much rubbish test code.
- Tyguy7 12y ago
  I have similar experiences.
  - UK-AL 12y ago
    Unit Tests are a god send when you have good coverage on a legacy large project. Refactor code, and it gives you that much more confidence you have not broken anything. They're seriously useful mid-later stages of a project, which is probably why startups devs aren't doing them...
    - donchipotle 12y ago
      I think what they are saying is the kind of tests they see in a corporate environment are not the kind that add any value to the refactoring process (or worse, inhibit it).
- jakub_g 12y ago
  Tests are a great thing when everyone involved understands their value. In my former team we had a big number of unit & integration tests that saved my ass many times. On the other hand, when the team has a KPI "we need a big test coverage!" and people are lazy or simply bad developers, you end up with a code like try { runTheWholeApp() } catch (e) {} assertTrue(itRan)
  - ZeroGravitas 12y ago
    Mutation Testing can test your tests in these situations. Well actually if the organization is that dysfunctional it's probably game over, but Mutation Testing can catch you when you do this accidentally.
- Cymen 12y ago
Juraci 12y ago
Rant about "unity". The idea of unity is vague. Almost every person has a different definition of unity. Do not think about unity think about granularity, or even better facets of behavior. The amount of integration can scale to infinity. You can have infinity layers of integration between unity and end2end tests. The attempt to classify granularity is a waste of time.
abuehrle 12y ago
I write lots of unit tests. One thing I'm struggling with now is the organization of a large number of tests. It's hard to know what I have tests for without going into my tests directory and reading comments. Even then, I feel like I could do more to have a "big picture" overview of my testing strategy rather than seeing a bunch of one-off tests. I haven't quite figured out what that should look like. I think the IDE could do more in some cases. For example, when unit testing a method, I'd like to be able to tag a reference to the method I'm testing. Then, when I'm in my code, my IDE could show me the docstrings of tests I have for that method. It's the difference between "hello world" and software design principles used to build large scale systems. I see lots of "hello world" for testing, but not much of the other side. Does anyone have suggestions?
- ereyes01 12y ago
  The way I personally solve this problem is by organizing my code and my tests in a similar structure. Split application code up into reasonable files, and split the tests up the same way. Depending on your language/test framework, it also helps to put the test right next to the code... i.e. in Go, you can have component.go and component_test.go in the same directory. I do agree though that it would be nice to see an IDE have some smarts about tests...
- ZeroGravitas 12y ago
  Some IDEs will read unit test code coverage logs and be able to show you which lines are covered by tests, and which tests exactly cover each line.
- PJDK 12y ago
  Visual Studio + Resharper lets you do a "Find Usages" of a type or method which is very useful for finding relevant tests. Intellij has the same functionality and I imagine the other JetBrains IDEs are similar. More general advice is that your test structure should mirror your code structure. So a folder called Services should have a matching test folder called Services containing tests for the classes within it.
dkrich 12y ago
I have to say that the trade-off argument brought up between DRY and readability is pretty lame. If you have a junior developer who can't make sense of the test because you put the test data in a separate class or are using a setup method, then teach him or her how to trace the call to the setup method and then read through it. If we follow this logic then all code should follow the same methodology of readability because tracing through method calls is too cumbersome. Obviously that isn't the case for most code, so why would it be true for unit tests?
- woah 12y ago
  Part of the benefit of tests is that they are a form of documentation.
  - dkrich 12y ago
    I don't think having readable tests and DRY are necessarily mutually-exclusive. If you name methods and variables with lengthy but very descriptive names your tests can still be readable. In fact the more you modularize your setup methods the more readable they can become. For example: TestData.CreatePerson(); TestData.AddSsnToPerson(); TestData.PersonModule.checkWhetherPersonHasValidSsn(); ...is pretty readable despite being broken up. It may or may not be true that if you just went through the setup here it may be more readable, but the benefit to this approach is that you only have to write a method to add an SSN to a person object one time and then everyone can use it without needing to understand the structure of different objects. Notwithstanding readability, I would argue that DRY makes writing tests easier than cramming a bunch of setters into every test case. Once somebody understands what a setup method does, in every place that it is used in other tests the tester can skip over that part and move on to testing their specific use case.
platz 12y ago
For tests we should use DAMP, Descriptive And Meaningful Phrases, instead of full DRY
chatmasta 12y ago
Does anyone have any resources for best practices unit testing web apps written with frameworks like Django, Rails, etc? Whenever I try to apply TDD to a framework, I find myself testing the functionality of the framework (which should already be tested), rather than my app itself. How do you decide what to test when you are effectively cobbling together framework concepts? Is this the time for focus on integration tests over unit tests?

83 points by GarethX 12y ago

A nice article, and I think I should add the book to my (already too large) to-read pile. I totally agree with "DRY might not be great in all cases" - In tests, it's often better to make each single test as expressive as possible. I once read (and forgot who said it): "Code should be either DRY (don't repeat yourself) or WET (write expressive tests), but never damp)" And here's a shameless plug: I recently wrote an article about how tests can help you achieve simple design, and the next article in the series will be coming soon: http://quickglance.at/en/simple_design/passes_its_tests http://quickglance.at/en/simple_design/passes_its_tests

zak_mc_kracken 12y ago
> "Code should be either DRY (don't repeat yourself) or WET (write expressive tests), but never damp) I'm getting so tired of people who try to capture the intricacies and complexities of our profession with witty one liners that have to be taken like dogma. Just use your common sense, let your experience guide you and if you're not sure about something, ask your teammates.
- struppi 12y ago
  Who says that they have to be taken like dogma? Sometimes the one-liners are just easy to remember. Often they are funny. They can be a great tool to quickly illustrate a point or start a conversation. The one liners can never replace a conversation or explanation. They cannot replace common sense or real knowlege. I would never treat them as laws or follow them by the letter...
- Gupie 12y ago
  > Just use your common sense, let your experience guide you and if you're not sure about something, ask your teammates. Brilliant, added to our bugzilla quips list.

vowelless 12y ago

I wish the audio quality was a little better. Is there any post process work that can be done to improve the quality?

GarethX 12y ago
Sadly that's with post-processing improvements. Without the guests recording their own audio and then syncing it after (which is something of a big ask), we're reliant on the quality of the connection. Something to work on for sure. I'm hopeful solutions like http://podclear.com http://podclear.com can help.

lukasm 12y ago

You may want to invest a few bucks in a good mic.

GarethX 12y ago
Yeah, sorry about that. Derrick has one, but was moving house at the time of recording and it was packed away. So future ones should be using that, at least on the host side.

collyw 12y ago

This is one of the most sensible articles I have seen on unit testing. Rather than religiously insist on tests first or 100% coverage, judge when it is valuable to have these things and when they are not important. Far too many articles on testing test the framework / language being used.

zak_mc_kracken 12y ago

Terrible audio. Next time you interview someone, ask them to install Audacity and to run it during the interview. Then have them send you the audio file via Google Drive.

colinramsay 12y ago

I constantly have problems with people posting the same information on unit tests, again and again. I know how to set up (for example) RSpec, MiniTest, Cucumber, FactoryGirl. The thing that people struggle with is how to use these in a larger, practical sense, rather than in just the trivial. I think this puts a lot of people off testing.

Tyguy7 12y ago
Agreed. I have only used TDD once as a front-end web developer, and I've found that I do struggle with the practicality of it. I spent half my time managing and maintaining my tests. When I did the math, It seemed that all that extra time spent was more than It would have taken to simply chase down bugs that would have arisen in a non-tdd environment. It just seems to take more time than is practical. Especially in agile startups.
- CmonDev 12y ago
  Actually in this case he seems to be past his personal TDD hype peak: "... a proponent of the selective use of Test-Driven Development." Hopefully this will become a mainstream thought. Then maybe we will fix the legacy browser languages problem. Man can dream...
- ZeroGravitas 12y ago
  Is the only metric your time spent preventing/chasing bugs? What if those bugs get out to the customer and impact them?
- deleted 12y ago
  [deleted]
- atilaneves 12y ago
  Perhaps. The difference is that in one case you'll have the machine automatically tell you when and why your code stops working because of a regression, and in the other you have no clue until the customer hits it. It's not only about proving it worked once, it's about proving it with every commit.
- 12y ago

henrik_w 12y ago

I like Jay's comments about how tests are wasted if you don't know exactly what you want, and just try out different things (because then you need to change/delete the tests when you end up changing your code). As for common mistakes with unit tests, my top 5 list is: 1. Testing algorithms together with coordinators. 2. Mocking too much. 3. Not using asserts. 4. Leaving print statements in the tests. 5. Checking the log statements, not the result. More details here: 5 Unit Testing Mistakes http://henrikwarne.com/2014/02/19/5-unit-testing-mistakes/ http://henrikwarne.com/2014/02/19/5-unit-testing-mistakes/

Iftheshoefits 12y ago
Here is another one: failing to understand that tests introduce additional maintenance and design overhead. Quis custodiet ipsos custodes? Tests are additional code that is tightly coupled to the target code that is meant to be used in production. This additional code requires design, debugging, and maintenance just like the target code. It is also tightly coupled to the target codebase: a change in either necessitates at least an inspection to determine what, if any, changes must then be made in the other. This all adds overhead to the whole process, and too often I see casual, glib statements implying or even asserting that the overhead is nonexistent or else pays for itself. It isn't free and it is especially not free the more dogmatic the test proponents and tests themselves are.
- henrik_w 12y ago
  Agreed. I think Jay touches on this when talking about how you don't need to aim for 100% unit test coverage. It's also what I mean by testing algorithmic code (where the pay off from the effort is large), versus testing coordinator code, where you are better off not unit testing it (because it is not complicated).
  - rane 12y ago
    What's coordinator code like?
    - 12y ago

encoderer 12y ago

Much of what passes for testing is a near-criminal waste of corporate resources. I've seen so much rubbish test code.

Tyguy7 12y ago
I have similar experiences.
- UK-AL 12y ago
  Unit Tests are a god send when you have good coverage on a legacy large project. Refactor code, and it gives you that much more confidence you have not broken anything. They're seriously useful mid-later stages of a project, which is probably why startups devs aren't doing them...
  - donchipotle 12y ago
    I think what they are saying is the kind of tests they see in a corporate environment are not the kind that add any value to the refactoring process (or worse, inhibit it).
jakub_g 12y ago
Tests are a great thing when everyone involved understands their value. In my former team we had a big number of unit & integration tests that saved my ass many times. On the other hand, when the team has a KPI "we need a big test coverage!" and people are lazy or simply bad developers, you end up with a code like try { runTheWholeApp() } catch (e) {} assertTrue(itRan)
- ZeroGravitas 12y ago
  Mutation Testing can test your tests in these situations. Well actually if the organization is that dysfunctional it's probably game over, but Mutation Testing can catch you when you do this accidentally.
Cymen 12y ago

Juraci 12y ago

Rant about "unity". The idea of unity is vague. Almost every person has a different definition of unity. Do not think about unity think about granularity, or even better facets of behavior. The amount of integration can scale to infinity. You can have infinity layers of integration between unity and end2end tests. The attempt to classify granularity is a waste of time.

abuehrle 12y ago

I write lots of unit tests. One thing I'm struggling with now is the organization of a large number of tests. It's hard to know what I have tests for without going into my tests directory and reading comments. Even then, I feel like I could do more to have a "big picture" overview of my testing strategy rather than seeing a bunch of one-off tests. I haven't quite figured out what that should look like. I think the IDE could do more in some cases. For example, when unit testing a method, I'd like to be able to tag a reference to the method I'm testing. Then, when I'm in my code, my IDE could show me the docstrings of tests I have for that method. It's the difference between "hello world" and software design principles used to build large scale systems. I see lots of "hello world" for testing, but not much of the other side. Does anyone have suggestions?

ereyes01 12y ago
The way I personally solve this problem is by organizing my code and my tests in a similar structure. Split application code up into reasonable files, and split the tests up the same way. Depending on your language/test framework, it also helps to put the test right next to the code... i.e. in Go, you can have component.go and component_test.go in the same directory. I do agree though that it would be nice to see an IDE have some smarts about tests...
ZeroGravitas 12y ago
Some IDEs will read unit test code coverage logs and be able to show you which lines are covered by tests, and which tests exactly cover each line.
PJDK 12y ago
Visual Studio + Resharper lets you do a "Find Usages" of a type or method which is very useful for finding relevant tests. Intellij has the same functionality and I imagine the other JetBrains IDEs are similar. More general advice is that your test structure should mirror your code structure. So a folder called Services should have a matching test folder called Services containing tests for the classes within it.

dkrich 12y ago

I have to say that the trade-off argument brought up between DRY and readability is pretty lame. If you have a junior developer who can't make sense of the test because you put the test data in a separate class or are using a setup method, then teach him or her how to trace the call to the setup method and then read through it. If we follow this logic then all code should follow the same methodology of readability because tracing through method calls is too cumbersome. Obviously that isn't the case for most code, so why would it be true for unit tests?

woah 12y ago
Part of the benefit of tests is that they are a form of documentation.
- dkrich 12y ago
  I don't think having readable tests and DRY are necessarily mutually-exclusive. If you name methods and variables with lengthy but very descriptive names your tests can still be readable. In fact the more you modularize your setup methods the more readable they can become. For example: TestData.CreatePerson(); TestData.AddSsnToPerson(); TestData.PersonModule.checkWhetherPersonHasValidSsn(); ...is pretty readable despite being broken up. It may or may not be true that if you just went through the setup here it may be more readable, but the benefit to this approach is that you only have to write a method to add an SSN to a person object one time and then everyone can use it without needing to understand the structure of different objects. Notwithstanding readability, I would argue that DRY makes writing tests easier than cramming a bunch of setters into every test case. Once somebody understands what a setup method does, in every place that it is used in other tests the tester can skip over that part and move on to testing their specific use case.

platz 12y ago

For tests we should use DAMP, Descriptive And Meaningful Phrases, instead of full DRY

chatmasta 12y ago

Does anyone have any resources for best practices unit testing web apps written with frameworks like Django, Rails, etc? Whenever I try to apply TDD to a framework, I find myself testing the functionality of the framework (which should already be tested), rather than my app itself. How do you decide what to test when you are effectively cobbling together framework concepts? Is this the time for focus on integration tests over unit tests?