3 ms·

I think the true advantage you're referring to is automated regression testing, which isn't necessarily just unit testing (or at all)

mikekchar 8y ago
The interesting thing is (and without meaning to denigrate your point of view) I actually think that automated regression testing is not particularly valuable when you have unit testing. I actually want unit testing for the ability to help you reason about the code. One of the things that I would dearly like to discover is whether or not my view holds any water. It certainly feels that way from my perspective, but I'm biased ;-) However, I think you are probably correct in that I think it is far more likely that Chrome has a good automated regression test suite than it has a good unit test suite.
- jacobush 8y ago
  Good regression testing is a nice way to keep unit tests honest. If you never catch anything with your unit tests but boat loads in regression testing - you are inclined to pause and ponder.
  - mikekchar 8y ago
    I actually test my unit tests by changing the behaviour and measuring if the tests fail :-). A great metric is fuzzing the behaviour and measure the number of unit tests that fail. If it's a lot, then you can improve your unit test game :-)
    - kmonsen 8y ago
      The problem with unit tests is that they test things you know are problematic. The larger issues is then ones you were never aware could be a problem in the first place.
      - Retric 8y ago
        A larger problem is they make you feel better about bad code.
      - mikekchar 8y ago
        I highly recommend reading Michael Feather's Working Effectively with Legacy Code. He has the best description of unit tests that I've seen. Briefly, he describes it like a clamp. When you are working on a wood working project, you clamp part of your project so that it doesn't move. Then you work on the bit you are interested in. Later you clamp that part and work on another part. The purpose of unit tests is not to test the behaviour (unfortunate nomenclature aside) -- it's to immobilise it. This allows you to work on another part of the the system and be alerted if you've caused something to slip. Acceptance tests are incredibly important. They tell you if the system is working. No amount of unit tests are going to help you with that. Once you have accepted the behaviour, what you're really interested in is whether or not the behaviour has changed. You do not need your acceptance tests for that -- your unit tests will tell you. I'll write it a bit more concisely because I think it is important: acceptance tests tell you whether or not the code is working correctly. Unit tests tell you whether or not the code is doing the same thing it was doing the last time you ran the tests. The reason I don't favour a large suite of acceptance tests is because they are unwieldy. It's fine for a few months, but once you get a few tens of thousands of lines of code, you will end up with a lot of acceptance tests. These acceptance tests are extremely hard to refactor. It's extremely hard to remove duplication. Over time, they get more and more problematic until you are spending more time trying to figure out how to make your acceptance tests pass than you are trying to figure out how to make your production changes. Unit tests, when written in specific ways, have less problem with this. Some people think about a "unit" as being a class. But really a "unit" is anything that you might want to isolate in your clamp. It can be a function. It can be a class. It can be a module. Your unit tests should probe the behaviour in the unit (and by "probe" I mean, expose the internal state). Michael Feather's has a great analogy of a "seam" which runs through your code. You try to find (or make) that seam and you insert probes to show you the state in various circumstances. IMHO, you should write unit tests the exact same way you write any code. Your "circumstances" (or scenarios, I guess) consist of creating the data structures to give your initial state. Your "tests" consist of probing the state along the seams and comparing it to expected values. This is simple code. You should be able to maintain this code using the same tools you use to maintain any code. You should write functions. You should write classes. You should write modules. You should use all the tricks of your trade to reduce the complexity of your "test" code. Your goal is to create specificity when tests "fail" (the probe detects behaviour different than your expectation -- or the clamp detects that your wood has slipped). When behaviour changes, only a few tests (ideally one) should "fail". It should report the "failure" in a way that immediately describes the difference between the state you expected and the state that you probed. It should be easy to change the probe when the behaviour is intentionally changed (ideally changing only one place). It should be easy to probe new behaviour (just build your data and add an expectation). Finally, it should be easy to reason about the behaviour of the code by reading the "tests". Refactoring your tests and removing duplication is very important here. As for acceptance tests, like I said, they are incredibly important. What I don't find particularly useful is a large suite of regression acceptance tests. The unit tests already tell me when the behaviour has changed. When written well, they even tell me exactly where in the code the behaviour "slipped". I often write manual acceptance tests. Once I have tested it, it is not necessary to test it again (as long as I have a good unit test suite). My personal opinion as to why people find automated acceptance tests suites important is because they have never worked with a good unit test suite before. There is a general lack of experience in the industry with these concepts. Quite a few people's experience with well tested code is with green field projects. Often these people leave after a year or so. It's not until you have a lot of experience working with the legacy of various testing techniques that you can understand the advantages and disadvantages. I think this is why Michael Feathers is so respected -- as far as I can tell he specialises in legacy code. Having said all that (and I'll be surprised if you make it to the bottom :-) ), I do value a small automated acceptance test suite. It's my canary in the mine. If it ever fails, then I know I've really stuffed something up and I launch an immediate investigation. Also, there are some things that can't be unit tested effectively (for example testing a web application across both client and server) -- you end up faking the boundaries, which leads to the possibility of skewing. Again, in those cases, I try to find a few end to end tests that will hit the majority possible problems. I hope you found that interesting. I've typed up essentially this same message in at least 10 different threads of the past couple of years. I think it's slowly getting better, but I think I still haven't managed to explain the concepts as well as they need to be explained. If you've made it this far, thanks for reading :-)
    - spockz 8y ago
      You can use mutation testing[1] to automate this. E.g. pitest[2] for Java, or golang[3]. [1]: https://en.wikipedia.org/wiki/Mutation_testing https://en.wikipedia.org/wiki/Mutation_testing [2]: http://pitest.org/ http://pitest.org/ [3]: https://github.com/zimmski/go-mutesting https://github.com/zimmski/go-mutesting
- sudhirj 8y ago
  Libraries benefit very heavily from unit tests, built applications or frameworks derive more benefit from regression tests. For a small unit of code, or a library, the unit tests effectively prove that the code/library does what it says on the box. For a continuously worked on application, regression tests holds the guarantee that the application continues to work correctly for whatever thousands or millions of use cases built up over its lifetime - even when the implementations, algorithms and libraries used change underneath.

mikekchar 8y ago

The interesting thing is (and without meaning to denigrate your point of view) I actually think that automated regression testing is not particularly valuable when you have unit testing. I actually want unit testing for the ability to help you reason about the code. One of the things that I would dearly like to discover is whether or not my view holds any water. It certainly feels that way from my perspective, but I'm biased ;-) However, I think you are probably correct in that I think it is far more likely that Chrome has a good automated regression test suite than it has a good unit test suite.

jacobush 8y ago
Good regression testing is a nice way to keep unit tests honest. If you never catch anything with your unit tests but boat loads in regression testing - you are inclined to pause and ponder.
- mikekchar 8y ago
  I actually test my unit tests by changing the behaviour and measuring if the tests fail :-). A great metric is fuzzing the behaviour and measure the number of unit tests that fail. If it's a lot, then you can improve your unit test game :-)
  - kmonsen 8y ago
    The problem with unit tests is that they test things you know are problematic. The larger issues is then ones you were never aware could be a problem in the first place.
    - Retric 8y ago
      A larger problem is they make you feel better about bad code.
    - mikekchar 8y ago
      I highly recommend reading Michael Feather's Working Effectively with Legacy Code. He has the best description of unit tests that I've seen. Briefly, he describes it like a clamp. When you are working on a wood working project, you clamp part of your project so that it doesn't move. Then you work on the bit you are interested in. Later you clamp that part and work on another part. The purpose of unit tests is not to test the behaviour (unfortunate nomenclature aside) -- it's to immobilise it. This allows you to work on another part of the the system and be alerted if you've caused something to slip. Acceptance tests are incredibly important. They tell you if the system is working. No amount of unit tests are going to help you with that. Once you have accepted the behaviour, what you're really interested in is whether or not the behaviour has changed. You do not need your acceptance tests for that -- your unit tests will tell you. I'll write it a bit more concisely because I think it is important: acceptance tests tell you whether or not the code is working correctly. Unit tests tell you whether or not the code is doing the same thing it was doing the last time you ran the tests. The reason I don't favour a large suite of acceptance tests is because they are unwieldy. It's fine for a few months, but once you get a few tens of thousands of lines of code, you will end up with a lot of acceptance tests. These acceptance tests are extremely hard to refactor. It's extremely hard to remove duplication. Over time, they get more and more problematic until you are spending more time trying to figure out how to make your acceptance tests pass than you are trying to figure out how to make your production changes. Unit tests, when written in specific ways, have less problem with this. Some people think about a "unit" as being a class. But really a "unit" is anything that you might want to isolate in your clamp. It can be a function. It can be a class. It can be a module. Your unit tests should probe the behaviour in the unit (and by "probe" I mean, expose the internal state). Michael Feather's has a great analogy of a "seam" which runs through your code. You try to find (or make) that seam and you insert probes to show you the state in various circumstances. IMHO, you should write unit tests the exact same way you write any code. Your "circumstances" (or scenarios, I guess) consist of creating the data structures to give your initial state. Your "tests" consist of probing the state along the seams and comparing it to expected values. This is simple code. You should be able to maintain this code using the same tools you use to maintain any code. You should write functions. You should write classes. You should write modules. You should use all the tricks of your trade to reduce the complexity of your "test" code. Your goal is to create specificity when tests "fail" (the probe detects behaviour different than your expectation -- or the clamp detects that your wood has slipped). When behaviour changes, only a few tests (ideally one) should "fail". It should report the "failure" in a way that immediately describes the difference between the state you expected and the state that you probed. It should be easy to change the probe when the behaviour is intentionally changed (ideally changing only one place). It should be easy to probe new behaviour (just build your data and add an expectation). Finally, it should be easy to reason about the behaviour of the code by reading the "tests". Refactoring your tests and removing duplication is very important here. As for acceptance tests, like I said, they are incredibly important. What I don't find particularly useful is a large suite of regression acceptance tests. The unit tests already tell me when the behaviour has changed. When written well, they even tell me exactly where in the code the behaviour "slipped". I often write manual acceptance tests. Once I have tested it, it is not necessary to test it again (as long as I have a good unit test suite). My personal opinion as to why people find automated acceptance tests suites important is because they have never worked with a good unit test suite before. There is a general lack of experience in the industry with these concepts. Quite a few people's experience with well tested code is with green field projects. Often these people leave after a year or so. It's not until you have a lot of experience working with the legacy of various testing techniques that you can understand the advantages and disadvantages. I think this is why Michael Feathers is so respected -- as far as I can tell he specialises in legacy code. Having said all that (and I'll be surprised if you make it to the bottom :-) ), I do value a small automated acceptance test suite. It's my canary in the mine. If it ever fails, then I know I've really stuffed something up and I launch an immediate investigation. Also, there are some things that can't be unit tested effectively (for example testing a web application across both client and server) -- you end up faking the boundaries, which leads to the possibility of skewing. Again, in those cases, I try to find a few end to end tests that will hit the majority possible problems. I hope you found that interesting. I've typed up essentially this same message in at least 10 different threads of the past couple of years. I think it's slowly getting better, but I think I still haven't managed to explain the concepts as well as they need to be explained. If you've made it this far, thanks for reading :-)
  - spockz 8y ago
    You can use mutation testing[1] to automate this. E.g. pitest[2] for Java, or golang[3]. [1]: https://en.wikipedia.org/wiki/Mutation_testing https://en.wikipedia.org/wiki/Mutation_testing [2]: http://pitest.org/ http://pitest.org/ [3]: https://github.com/zimmski/go-mutesting https://github.com/zimmski/go-mutesting
sudhirj 8y ago
Libraries benefit very heavily from unit tests, built applications or frameworks derive more benefit from regression tests. For a small unit of code, or a library, the unit tests effectively prove that the code/library does what it says on the box. For a continuously worked on application, regression tests holds the guarantee that the application continues to work correctly for whatever thousands or millions of use cases built up over its lifetime - even when the implementations, algorithms and libraries used change underneath.