20 ms·

Questions to ask yourself when writing tests

203 points by michalc 9y ago

V2hLe0ThslzRaV2 9y ago
>> "For every part of my code change, if I break the code, will a test fail?" Since "breaking code" is super subjective, and generally speaking, trying to "cover everything" is a recipe for hell. Anyone able to expand on what the author meant by this?
- brucephillips 9y ago
  > since breaking code is super subjective. Only to the extent that your spec is incomplete.
- jcrben 9y ago
  Meaning: if I break the feature, will I know about it from a test failure? Basically, you put the customer first. Make sure your features are tested such that they can't fail without a failing test before writing lower-level tests. This is also the approach advocated by the popular book Growing Object-Oriented Software Guided by Tests.
- atticusCr 9y ago
  That's actually, IMHO, a good asset that comes out of a good test and code coverage. I would be worried if after adding new piece of code or modifying an existing code if there is not a test that tells me that something is broken due my code. Here is where testing becomes very helpful.
- michalc 9y ago
  Perhaps this part is not clear / well defined, but roughly, I meant that code coverage is (about) 100% for the lines added/changed, and some "reasonable" subset of possible breaking changes would be picked up by failing tests. What I had in mind specifically in the answer, was the case of changing "interfaces" between parts of code. For example, the case of changing a function's arguments, or how it uses them, but omitting to change a call site. Checking that the call site just calls the function would not be enough to produce a failing test, especially without type safety. The test would actually need to assert on what the function does, e.g. its return value for a pure function. Yes, I think trying to test against every single possible breaking change is not valuable.
discreteevent 9y ago
"consider instead writing higher level tests that each each test more of the code." I really like the way they keep repeating this as the answer to so many questions. To me it reads as a softly softly approach to weaning people off what appears to be a mania for micro level unit testing, driven by people like Uncle Bob.
- ajmurmann 9y ago
  I'm not sure I agree with the "instead" in his recommendation. "micro level unit testing" comes with its own benefits, like faster feedback and also design feedback that integration or acceptance style tests won't give you. I like to start out with a high level acceptance tests and then write unit tests for every change to involved classes. You really need a healthy testing pyramid.
  - discreteevent 9y ago
    The problem is that, in my observation over a number of codebases with many developers: A lot of unit tests end up testing the implementation. When you go to refactor, these tests break which slows you down. To me it seems nearly impossible not to end up with these kinds of net negative tests when you follow a micro approach. I mean the approach outlined in books and blogs where you write a test and then write only a few lines of code. It's all very neat and satisfying but for me it's just a form of fooling myself. I can see this if I honestly measure the net effect of the these kinds of tests in the large. The only thing is that they might sometimes help me locate a problem a few seconds faster, but that's all.
    - cle 9y ago
      Agree. I am so sick of having to code in extremely brittle codebases that are impossible to refactor because of the sheer volume of UTs that are tightly coupled to very specific implementation details--added to satisfy a minimum code coverage metric. UTs were supposed to save us from the pain of refactoring, but in many cases they just trade one time-sink for another, and the cost equation remains the same (not to refactor, because it's too time consuming). In this same vein, I propose another question: How similar does the test look to the production code? These kinds of super-granular, brittle tests have a way of looking very similar to the code they are testing (e.g. super strict mocking of every dependency almost looks exactly the same as the code under test!). That style of testing is basically writing the code twice, and verifying that the second copy does indeed equal the first. I'd rather just write the damn code twice, and save myself from having to figure out all the minute differences between testing libraries that framework jockies bikeshed over.
simonhamp 9y ago
The most important question to ask when writing tests: Can I do this faster if I don’t write a test? The answer is always no. Even if you are the only person building something, future you will lick your boots clean in gratitude if there are tests. Because even the best developers have to work with their own code sometimes.
- mooreds 9y ago
  This! I became a believer in automated testing when I worked with a large ETL process that worked with real estate data. This data was input by real estate agents, varied wildly in quality, and had both image files and structured text data. Releasing this code before automated testing was fearful. We'd push changes to staging and then wait for three (!) days of data, then examine the staging and production databases manually and see if there were material differences. Needless to say, we didn't like to release this code very often. When I left, we weren't doing automated testing on the image processing, but we were on the text side of things. It became far easier to do a release because we had a set of regression tests that gave us confidence we weren't breaking anything. If something did pop up, we could add it to the suite. Human beings systematically undervalue their future selves. This is why we have a hard time saving for retirement, exercising and writing tests. Think of your future self and write tests.
  - maksimum 9y ago
    > Human beings systematically undervalue their future selves. This is why we have a hard time saving for retirement, exercising and writing tests. Think of your future self and write tests. This! What's also worrisome is that sometimes your incentives are misaligned; e.g. when product development and maintenance are handled by different teams, or features have tight deadlines and screw the debt that will be the future and we must deal with now.
- qaq 9y ago
  I like having tests but I would bet if you audit you production stack it's very likely that a very large portion of it does not really have tests yet is prob most stable part of your stack like Linux kernel etc.
atticusCr 9y ago
The author does not cover any question related to application security. Things like is this parameter/input value properly sanitized, does this piece is/is not vulnerable to injection attacks, does this piece of code performs authentication/authorization checks? Is RBAC properly implemented for this method?
- viraptor 9y ago
  I agree with some cases, but "is this parameter/input value properly sanitized" is a bit weird. It should only every apply to a) the db framework, b) those N really weird cases that have to break the abstraction and don't use the db framework. If you have to test every input, then the problem is on a completely different level than missing a test.
  - atticusCr 9y ago
    Kind of, if you have a centralized place to perform input data validation, as it should, then it is just a matter to test that piece of code same if you are using a framework. However, I don't understand why you refer to a db in the first place? Is it because I used the injection attack as an example? if that's the case bare in mind that Injection target other interprets as well not only a db. But getting back to my original idea, what I want to highlight the need of adding cases to cover application security.
    - viraptor 9y ago
      Yeah, my mind substituted parameter with query parameter. Too much database stuff at my $dayjob recently and I get tunnel vision ;-)
      - atticusCr 9y ago
        lol! thanks for your comments.
spion 9y ago
Here are 3 more questions: * Are these tests relying on API that is likely to change? * Can I make the API surface area used by all tests even smaller? * Can I make a library that wraps the existing API of the unit to get a smaller and/or more stable API for use in the tests only? These 3 help get tests that withstand refactors. An example: An acceptance test for an editing form is relying on the save button having a certain CSS style to find it and click it. This is API that is likely to change. An unrelated change in the looks of a button may break the test. If we switch to using the text of the button ("Save"), that's better because that is what the user is likely to rely on too when they try to find a sav button. But its still not perfect as the text of the button could change. The final step is to make a little library function that finds a save button within a certain form. Then we can encode the logic of the test but vary the kinds of text that are considered "save" (or even the method of finding a save button - perhaps a standard CSS class of save buttons in the future!); the test logic itself remains "permanent" since it doesn't rely on any implementation detail anymore.
- atoko 9y ago
  A small library function like document.getElementById?
  - spion 9y ago
    The example wasn't perfect, but no, it wouldn't be that. What if in the future we have a screen that needs to show more than one save form? Then we'll need to stop using the "save" element id as a mechanism to denote save buttons; to do that, we need to update all the tests that rely on this mechanism. The small library function would be `getSaveButton(form)` or even `save(form)` - now every form save test no longer encodes the knowledge of how a form's save buttons are made, whether that's by using a certain ID, class, text or anything else. Now when we get that requirement for a screen with two forms, we'll no longer get mad and try to attack the idea (two save forms on one screen? that's inconsistent with our product, its confusing the users, etc etc) when the real reason is that it creates pain updating our tests. Instead we just update the save function. The general idea is to encode the meaning of the test and separate the implementation details (clicking the save button might even be too concrete - "saving the form" is probably about the right level of abstraction). A good way to do this is to describe this test in prose and check if its encoding details that may vary - does this sound like something universally true / something that will be true forever, or accidentally true due to current circumstances?
KirinDave 9y ago
I'm glad to see that increasingly people are recognizing that shallow mocks of an API interface are not very good at testing anything.
yawaramin 9y ago
Looks like Michal has been immersed in Haskell for at least the past year. I wonder if he will have something to say about balancing testing and coverage with static typing.
- michalc 9y ago
  Thinking about this, at the moment, I don't think static typing is a replacement for testing (or vice versa!). Although, apparently, with something like LiquidHaskell, you can get more logic "into" the types and checked by the compiler, but I'm unsure how much you can do when it comes to more complex business logic. Regarding testing "glue": static typing often gives evidence (but not proof) that code is glued together appropriately [refactoring even small projects without tests in Haskell is a joy: the compiler essentially tells you what to change]. However, it doesn't give evidence that the high level behaviour is what it needs to be. So I think higher level tests are still needed. I think maybe changing the first question from... > Am I confident the feature I wrote today works because of the tests and not because I loaded up the application? to... > Am I confident the feature I wrote today works because of the tests and type checking, and not because I loaded up the application? will probably help you to answer the question about how much static types allow you to forgo tests. My instinct is that in most cases, high level tests are still worthwhile.
  - yawaramin 9y ago
    I'm thinking much the same, though I do think of tests as looking after the dynamic behaviour and types after the static behaviour. Which seems obvious in retrospect, but once you wrap your head around it you can build neat abstractions like lightweight static capabilities: https://github.com/yawaramin/lightweght-static-capabilities https://github.com/yawaramin/lightweght-static-capabilities
lambdabit 9y ago
Here's one more. Is maintaining this test more time and work than testing manually?
securingsincity 9y ago
I'll add a couple more Can I run these tests more than once? Will they ever go stale? Can I run these tests and they'll clean themselves up? Having to update tests because they didn't take into account the date changing (Happy birthday Joe Test!) or manually cleaning up data is a huge time suck.
sethammons 9y ago
> Mocking introduces assumptions, which introduce risk. This boils down something I've had on my mind a lot of late. Though, with a different spin. I write a lot of Go. I prefer testing interfaces while some others what to use mock generators. This quote captures part of my reasoning behind avoiding mocks. I plan to write a detailed post at some point full of examples. I think this quote will work its way in there.
- michalc 9y ago
  I would be interested in reading it... if / when, if you can send me a link, that would be great!
dvh 9y ago
1. Am I paid to write tests?
- jyriand 9y ago
  Yes. You are paid to write working software.
conatus 9y ago
The most important thing here seems to be "For every part of my code change, if I break the code, will a test fail?". Or as I'd put it "if you break the code deliberately does the test actually fail?". Seen quite a few tests in my time that don't capture the functionality they think they are. They pass but wouldn't be able to tell if the underlying functionality they capture is genuinely broken. This is why I guess the standard practice is to go test red before you go test green.
falsedan 9y ago
Here are some more questions: How much business value is this test adding? That is, if this test failed and we ignored it, how much would the business suffer? Is the code easy to test? That is, does the design have lots of self-contained components with well-described input/outputs & conditions/assumptions? Do the docs clearly communicate that? Will the test still work if we change the implementation? How much work to update/remove the tests if the behaviour has to change to follow new business requirements?
nexfitter 9y ago
> Have I just made something public in order to test it? > > If yes, consider instead writing higher level tests that each each test more of the code. This is one that I struggle with in JS with React.js components. If you have a little helper component in a file that isn't exported but used in the same file by a component that is exported, it is sometimes difficult to test that non-exported component. Because of how enzyme shallow rendering works you don't get the full tree so that component, if sufficiently nested, might not ever be touched. This forces me to export the component just to test it. example: https://imgur.com/a/gqgcS https://imgur.com/a/gqgcS
- reledi 9y ago
  I run into the same problem with React and it bugs me from a testing standpoint. Extracting code from a big component to helper functions and extracting those functions from the component can lead to cleaner code, and it makes it much easier to test the behaviour of the helpers directly than having to render the component with enzyme. A good example of this is moving state changes to pure functions [1] which makes them much easier to test, but you'll need to export those functions to test them. 1: https://twitter.com/dan_abramov/status/824310320399319040 https://twitter.com/dan_abramov/status/824310320399319040
  - nexfitter 9y ago
    that is a really cool technique, thanks for the link!
reledi 9y ago
Couple more which haven't been mentioned yet: - Can I run the tests in random order? - Are the tests optimised for readability? - Are the tests unnecessarily testing third-party code? - Do individual tests contain the whole story? (Or do you have to look in many places to understand each test?)
- baristaGeek 9y ago
  Being able to run tests in a random order is indeed pretty important. It is not only an indicator that the features were written with high quality, but that the tests are high quality as well. Things such as the mocks mutating happen and the developer writing the tests should avoid those kinds of things happening.

203 points by michalc 9y ago

>> "For every part of my code change, if I break the code, will a test fail?" Since "breaking code" is super subjective, and generally speaking, trying to "cover everything" is a recipe for hell. Anyone able to expand on what the author meant by this?

brucephillips 9y ago
> since breaking code is super subjective. Only to the extent that your spec is incomplete.
jcrben 9y ago
Meaning: if I break the feature, will I know about it from a test failure? Basically, you put the customer first. Make sure your features are tested such that they can't fail without a failing test before writing lower-level tests. This is also the approach advocated by the popular book Growing Object-Oriented Software Guided by Tests.
atticusCr 9y ago
That's actually, IMHO, a good asset that comes out of a good test and code coverage. I would be worried if after adding new piece of code or modifying an existing code if there is not a test that tells me that something is broken due my code. Here is where testing becomes very helpful.
michalc 9y ago
Perhaps this part is not clear / well defined, but roughly, I meant that code coverage is (about) 100% for the lines added/changed, and some "reasonable" subset of possible breaking changes would be picked up by failing tests. What I had in mind specifically in the answer, was the case of changing "interfaces" between parts of code. For example, the case of changing a function's arguments, or how it uses them, but omitting to change a call site. Checking that the call site just calls the function would not be enough to produce a failing test, especially without type safety. The test would actually need to assert on what the function does, e.g. its return value for a pure function. Yes, I think trying to test against every single possible breaking change is not valuable.

discreteevent 9y ago

"consider instead writing higher level tests that each each test more of the code." I really like the way they keep repeating this as the answer to so many questions. To me it reads as a softly softly approach to weaning people off what appears to be a mania for micro level unit testing, driven by people like Uncle Bob.

ajmurmann 9y ago
I'm not sure I agree with the "instead" in his recommendation. "micro level unit testing" comes with its own benefits, like faster feedback and also design feedback that integration or acceptance style tests won't give you. I like to start out with a high level acceptance tests and then write unit tests for every change to involved classes. You really need a healthy testing pyramid.
- discreteevent 9y ago
  The problem is that, in my observation over a number of codebases with many developers: A lot of unit tests end up testing the implementation. When you go to refactor, these tests break which slows you down. To me it seems nearly impossible not to end up with these kinds of net negative tests when you follow a micro approach. I mean the approach outlined in books and blogs where you write a test and then write only a few lines of code. It's all very neat and satisfying but for me it's just a form of fooling myself. I can see this if I honestly measure the net effect of the these kinds of tests in the large. The only thing is that they might sometimes help me locate a problem a few seconds faster, but that's all.
  - cle 9y ago
    Agree. I am so sick of having to code in extremely brittle codebases that are impossible to refactor because of the sheer volume of UTs that are tightly coupled to very specific implementation details--added to satisfy a minimum code coverage metric. UTs were supposed to save us from the pain of refactoring, but in many cases they just trade one time-sink for another, and the cost equation remains the same (not to refactor, because it's too time consuming). In this same vein, I propose another question: How similar does the test look to the production code? These kinds of super-granular, brittle tests have a way of looking very similar to the code they are testing (e.g. super strict mocking of every dependency almost looks exactly the same as the code under test!). That style of testing is basically writing the code twice, and verifying that the second copy does indeed equal the first. I'd rather just write the damn code twice, and save myself from having to figure out all the minute differences between testing libraries that framework jockies bikeshed over.

simonhamp 9y ago

The most important question to ask when writing tests: Can I do this faster if I don’t write a test? The answer is always no. Even if you are the only person building something, future you will lick your boots clean in gratitude if there are tests. Because even the best developers have to work with their own code sometimes.

mooreds 9y ago
This! I became a believer in automated testing when I worked with a large ETL process that worked with real estate data. This data was input by real estate agents, varied wildly in quality, and had both image files and structured text data. Releasing this code before automated testing was fearful. We'd push changes to staging and then wait for three (!) days of data, then examine the staging and production databases manually and see if there were material differences. Needless to say, we didn't like to release this code very often. When I left, we weren't doing automated testing on the image processing, but we were on the text side of things. It became far easier to do a release because we had a set of regression tests that gave us confidence we weren't breaking anything. If something did pop up, we could add it to the suite. Human beings systematically undervalue their future selves. This is why we have a hard time saving for retirement, exercising and writing tests. Think of your future self and write tests.
- maksimum 9y ago
  > Human beings systematically undervalue their future selves. This is why we have a hard time saving for retirement, exercising and writing tests. Think of your future self and write tests. This! What's also worrisome is that sometimes your incentives are misaligned; e.g. when product development and maintenance are handled by different teams, or features have tight deadlines and screw the debt that will be the future and we must deal with now.
qaq 9y ago
I like having tests but I would bet if you audit you production stack it's very likely that a very large portion of it does not really have tests yet is prob most stable part of your stack like Linux kernel etc.

atticusCr 9y ago

The author does not cover any question related to application security. Things like is this parameter/input value properly sanitized, does this piece is/is not vulnerable to injection attacks, does this piece of code performs authentication/authorization checks? Is RBAC properly implemented for this method?

viraptor 9y ago
I agree with some cases, but "is this parameter/input value properly sanitized" is a bit weird. It should only every apply to a) the db framework, b) those N really weird cases that have to break the abstraction and don't use the db framework. If you have to test every input, then the problem is on a completely different level than missing a test.
- atticusCr 9y ago
  Kind of, if you have a centralized place to perform input data validation, as it should, then it is just a matter to test that piece of code same if you are using a framework. However, I don't understand why you refer to a db in the first place? Is it because I used the injection attack as an example? if that's the case bare in mind that Injection target other interprets as well not only a db. But getting back to my original idea, what I want to highlight the need of adding cases to cover application security.
  - viraptor 9y ago
    Yeah, my mind substituted parameter with query parameter. Too much database stuff at my $dayjob recently and I get tunnel vision ;-)
    - atticusCr 9y ago
      lol! thanks for your comments.

spion 9y ago

Here are 3 more questions: * Are these tests relying on API that is likely to change? * Can I make the API surface area used by all tests even smaller? * Can I make a library that wraps the existing API of the unit to get a smaller and/or more stable API for use in the tests only? These 3 help get tests that withstand refactors. An example: An acceptance test for an editing form is relying on the save button having a certain CSS style to find it and click it. This is API that is likely to change. An unrelated change in the looks of a button may break the test. If we switch to using the text of the button ("Save"), that's better because that is what the user is likely to rely on too when they try to find a sav button. But its still not perfect as the text of the button could change. The final step is to make a little library function that finds a save button within a certain form. Then we can encode the logic of the test but vary the kinds of text that are considered "save" (or even the method of finding a save button - perhaps a standard CSS class of save buttons in the future!); the test logic itself remains "permanent" since it doesn't rely on any implementation detail anymore.

atoko 9y ago
A small library function like document.getElementById?
- spion 9y ago
  The example wasn't perfect, but no, it wouldn't be that. What if in the future we have a screen that needs to show more than one save form? Then we'll need to stop using the "save" element id as a mechanism to denote save buttons; to do that, we need to update all the tests that rely on this mechanism. The small library function would be `getSaveButton(form)` or even `save(form)` - now every form save test no longer encodes the knowledge of how a form's save buttons are made, whether that's by using a certain ID, class, text or anything else. Now when we get that requirement for a screen with two forms, we'll no longer get mad and try to attack the idea (two save forms on one screen? that's inconsistent with our product, its confusing the users, etc etc) when the real reason is that it creates pain updating our tests. Instead we just update the save function. The general idea is to encode the meaning of the test and separate the implementation details (clicking the save button might even be too concrete - "saving the form" is probably about the right level of abstraction). A good way to do this is to describe this test in prose and check if its encoding details that may vary - does this sound like something universally true / something that will be true forever, or accidentally true due to current circumstances?

KirinDave 9y ago

I'm glad to see that increasingly people are recognizing that shallow mocks of an API interface are not very good at testing anything.

yawaramin 9y ago

Looks like Michal has been immersed in Haskell for at least the past year. I wonder if he will have something to say about balancing testing and coverage with static typing.

michalc 9y ago
Thinking about this, at the moment, I don't think static typing is a replacement for testing (or vice versa!). Although, apparently, with something like LiquidHaskell, you can get more logic "into" the types and checked by the compiler, but I'm unsure how much you can do when it comes to more complex business logic. Regarding testing "glue": static typing often gives evidence (but not proof) that code is glued together appropriately [refactoring even small projects without tests in Haskell is a joy: the compiler essentially tells you what to change]. However, it doesn't give evidence that the high level behaviour is what it needs to be. So I think higher level tests are still needed. I think maybe changing the first question from... > Am I confident the feature I wrote today works because of the tests and not because I loaded up the application? to... > Am I confident the feature I wrote today works because of the tests and type checking, and not because I loaded up the application? will probably help you to answer the question about how much static types allow you to forgo tests. My instinct is that in most cases, high level tests are still worthwhile.
- yawaramin 9y ago
  I'm thinking much the same, though I do think of tests as looking after the dynamic behaviour and types after the static behaviour. Which seems obvious in retrospect, but once you wrap your head around it you can build neat abstractions like lightweight static capabilities: https://github.com/yawaramin/lightweght-static-capabilities https://github.com/yawaramin/lightweght-static-capabilities

lambdabit 9y ago

Here's one more. Is maintaining this test more time and work than testing manually?

securingsincity 9y ago

I'll add a couple more Can I run these tests more than once? Will they ever go stale? Can I run these tests and they'll clean themselves up? Having to update tests because they didn't take into account the date changing (Happy birthday Joe Test!) or manually cleaning up data is a huge time suck.

sethammons 9y ago

> Mocking introduces assumptions, which introduce risk. This boils down something I've had on my mind a lot of late. Though, with a different spin. I write a lot of Go. I prefer testing interfaces while some others what to use mock generators. This quote captures part of my reasoning behind avoiding mocks. I plan to write a detailed post at some point full of examples. I think this quote will work its way in there.

michalc 9y ago
I would be interested in reading it... if / when, if you can send me a link, that would be great!

dvh 9y ago

1. Am I paid to write tests?

jyriand 9y ago
Yes. You are paid to write working software.

conatus 9y ago

The most important thing here seems to be "For every part of my code change, if I break the code, will a test fail?". Or as I'd put it "if you break the code deliberately does the test actually fail?". Seen quite a few tests in my time that don't capture the functionality they think they are. They pass but wouldn't be able to tell if the underlying functionality they capture is genuinely broken. This is why I guess the standard practice is to go test red before you go test green.

falsedan 9y ago

Here are some more questions: How much business value is this test adding? That is, if this test failed and we ignored it, how much would the business suffer? Is the code easy to test? That is, does the design have lots of self-contained components with well-described input/outputs & conditions/assumptions? Do the docs clearly communicate that? Will the test still work if we change the implementation? How much work to update/remove the tests if the behaviour has to change to follow new business requirements?

nexfitter 9y ago

> Have I just made something public in order to test it? > > If yes, consider instead writing higher level tests that each each test more of the code. This is one that I struggle with in JS with React.js components. If you have a little helper component in a file that isn't exported but used in the same file by a component that is exported, it is sometimes difficult to test that non-exported component. Because of how enzyme shallow rendering works you don't get the full tree so that component, if sufficiently nested, might not ever be touched. This forces me to export the component just to test it. example: https://imgur.com/a/gqgcS https://imgur.com/a/gqgcS

reledi 9y ago
I run into the same problem with React and it bugs me from a testing standpoint. Extracting code from a big component to helper functions and extracting those functions from the component can lead to cleaner code, and it makes it much easier to test the behaviour of the helpers directly than having to render the component with enzyme. A good example of this is moving state changes to pure functions [1] which makes them much easier to test, but you'll need to export those functions to test them. 1: https://twitter.com/dan_abramov/status/824310320399319040 https://twitter.com/dan_abramov/status/824310320399319040
- nexfitter 9y ago
  that is a really cool technique, thanks for the link!

reledi 9y ago

Couple more which haven't been mentioned yet: - Can I run the tests in random order? - Are the tests optimised for readability? - Are the tests unnecessarily testing third-party code? - Do individual tests contain the whole story? (Or do you have to look in many places to understand each test?)

baristaGeek 9y ago
Being able to run tests in a random order is indeed pretty important. It is not only an indicator that the features were written with high quality, but that the tests are high quality as well. Things such as the mocks mutating happen and the developer writing the tests should avoid those kinds of things happening.