7 ms·

Love this article. Very interesting, and I love how the results are able to be quantified. Coming from some projects that had a very heavy unit testing cultur

by cbanek 9y ago

Love this article. Very interesting, and I love how the results are able to be quantified. Coming from some projects that had a very heavy unit testing culture, I can agree that unit testing doesn't always necessarily have the intended results on "quality" and can really shape a codebase. I feel the root of the reason is that testing is not the same thing as development. Truly testing things involves a mentality that is different, it involves understanding how mulitple pieces fit together and operate in reality, not in some unit testing framework. A lot of developers look at unit tests the same way they look at development, once they work, it's done. But testing is proving they work in all the right conditions, not just getting it to work once. I've personally seen very simple code convoluted so that it could be unit tested, and how those convolutions can easily add bugs and additional layers of indirection that hurt performance (as well as add unnecessary code, which needs maintenance). Many times these tests could have been written with the existing code as long as it used some other component... but then it wouldn't be "unit testing" only to see these components not work together on the first runtime. (For some reason, places that rely heavily on unit testing seem to have fewer overall or integration tests, because they spend so much time and effort on unit testing) I'd love to see a study on how many lines of code change because of unit tests vs in non unit tested codebases. Both in terms of how many unit tests need to change and the size of the change itself.

pqh 9y ago
I'm curious what language and techniques you mean this applies to. In my experience, I've seen a lot of code that was written in a way that was not unit-testable, for what was the same reason that later turned out to be poorly-maintainable.
- loup-vaillant 9y ago
  Some seem to use (their personal idea of) unit testability as a proxy for maintainability or quality. That is often invalid. I've written some high-quality code and tested it to death¹, only to be told that it probably wasn't unit testable (because dependencies were hard coded instead of injected, IIRC). Making that code unit testable would have complicated it, thus lowering its maintainability. [1] Not a single bug has ever been found after my tests.
  - cbanek 9y ago
    Couldn't have said it better myself. I've had a lot of times where no bugs were found with huge amounts of unit testing over trivial code. It would have been easier and of more use to just do manual inspection (code review) with a bunch of people instead of writing the tests at that level, and instead writing tests at a higher level of complexity which involved more code/components. In a perfect world you'd test all the branches and possible values but out there in the real world, if you aren't finding bugs with your testing and there are still bugs in the code, you need to be spending your testing "dollars" (time/effort/talent) more efficiently.
    - loup-vaillant 9y ago
      I failed to mentioned that the tests themselves founds lots of bugs. I needed those tests. Also, those weren't mere individual tests, I ran a number of property-based tests as well, they tend to be much more thorough. I wrote those tests because I was writing foundational code. The outer layers hardly have any test, which is good enough for the proof of concept we were asked for. We'll need to be a bit more rigorous to polish this into production.
  - forkerenok 9y ago
    Out of curiosity, how do you come around assessing it as high-quality code? By what measure(s)?
    - loup-vaillant 9y ago
      I have developed some kind of instinct that is non-trivial to teach, and impossible to convey in a small HN comment. But my primary proxy is size. The less code the better. But even that have to give way to other concerns, such as style constraints, and straight up performance. While I can't show the code I was talking about above (it was proprietary), I can show my crypto library, Monocypher¹. I think it is a good example of what I mean by high-quality code. (Yes, I am boasting. I believe this is justified. Also, quality requirements for crypto code are kinda off the chart.) [1]: https://github.com/LoupVaillant/Monocypher https://github.com/LoupVaillant/Monocypher
ajmurmann 9y ago
It's crucial to keep in mind that unit tests are only one part of a healthy test and even TDD-diet. Especially if you write "true" unit tests that use doubles for test collaborators. The plus side of using doubles in your unit tests though is that the outcome insinuated by your last question should likely be different. I've found that changes typically only impact the unit test of the class or sometimes direct collaborators and a very small number of integration or acceptance tests. If you follow the open/closed principle those changes become even fewer.
- cbanek 9y ago
  Totally and completely agree. But everywhere I've worked in the past 10 years has always asked me to unit test everything, and unit testing is specifically called out all the time. Other types of tests are rarely even discussed, because many developers don't do a lot of testing beyond unit testing. Sometimes there are other organizations that pick up this slack, like a dedicated test organization, but many times there are not. Perhaps it's just my experience, but I've found unit tests often give a false sense of security when most of the issues I have to fix (which are frankly the hard issues, not simple logic bugs) heavily involve how the code interacts with its environment. I'm talking OS level, user level, privilege escalation (such as UAC in Windows), networking, load, perf and stress. It's not that I think that unit tests aren't good, they are, but I think people focus on them too much, to the detriment of other testing. Integration testing should catch all the things the unit tests would catch as well, the only difference is you have two copies of the test, and the unit test will give you more granularity into what is failing when you look at the test suite results. But as soon as you have a call stack or a logging facility (especially around error conditions), which is best practice, most of this is obvious as soon as you look at the result of the test.
  - ajmurmann 9y ago
    I totally agree that unit tests for bug prevention in new code are overrated. I mainly like them because they 1. Encourage decoupled code and 2. Make future refactoring easier especially in dynamically typed languages. Not having other tests is a huge mistake. I think I'm spoiled because I've spent my entire career at places that cared deeply about testing and proper TDD.

by cbanek 9y ago

pqh 9y ago

I'm curious what language and techniques you mean this applies to. In my experience, I've seen a lot of code that was written in a way that was not unit-testable, for what was the same reason that later turned out to be poorly-maintainable.

loup-vaillant 9y ago
Some seem to use (their personal idea of) unit testability as a proxy for maintainability or quality. That is often invalid. I've written some high-quality code and tested it to death¹, only to be told that it probably wasn't unit testable (because dependencies were hard coded instead of injected, IIRC). Making that code unit testable would have complicated it, thus lowering its maintainability. [1] Not a single bug has ever been found after my tests.
- cbanek 9y ago
  Couldn't have said it better myself. I've had a lot of times where no bugs were found with huge amounts of unit testing over trivial code. It would have been easier and of more use to just do manual inspection (code review) with a bunch of people instead of writing the tests at that level, and instead writing tests at a higher level of complexity which involved more code/components. In a perfect world you'd test all the branches and possible values but out there in the real world, if you aren't finding bugs with your testing and there are still bugs in the code, you need to be spending your testing "dollars" (time/effort/talent) more efficiently.
  - loup-vaillant 9y ago
    I failed to mentioned that the tests themselves founds lots of bugs. I needed those tests. Also, those weren't mere individual tests, I ran a number of property-based tests as well, they tend to be much more thorough. I wrote those tests because I was writing foundational code. The outer layers hardly have any test, which is good enough for the proof of concept we were asked for. We'll need to be a bit more rigorous to polish this into production.
- forkerenok 9y ago
  Out of curiosity, how do you come around assessing it as high-quality code? By what measure(s)?
  - loup-vaillant 9y ago
    I have developed some kind of instinct that is non-trivial to teach, and impossible to convey in a small HN comment. But my primary proxy is size. The less code the better. But even that have to give way to other concerns, such as style constraints, and straight up performance. While I can't show the code I was talking about above (it was proprietary), I can show my crypto library, Monocypher¹. I think it is a good example of what I mean by high-quality code. (Yes, I am boasting. I believe this is justified. Also, quality requirements for crypto code are kinda off the chart.) [1]: https://github.com/LoupVaillant/Monocypher https://github.com/LoupVaillant/Monocypher

ajmurmann 9y ago

It's crucial to keep in mind that unit tests are only one part of a healthy test and even TDD-diet. Especially if you write "true" unit tests that use doubles for test collaborators. The plus side of using doubles in your unit tests though is that the outcome insinuated by your last question should likely be different. I've found that changes typically only impact the unit test of the class or sometimes direct collaborators and a very small number of integration or acceptance tests. If you follow the open/closed principle those changes become even fewer.

cbanek 9y ago
Totally and completely agree. But everywhere I've worked in the past 10 years has always asked me to unit test everything, and unit testing is specifically called out all the time. Other types of tests are rarely even discussed, because many developers don't do a lot of testing beyond unit testing. Sometimes there are other organizations that pick up this slack, like a dedicated test organization, but many times there are not. Perhaps it's just my experience, but I've found unit tests often give a false sense of security when most of the issues I have to fix (which are frankly the hard issues, not simple logic bugs) heavily involve how the code interacts with its environment. I'm talking OS level, user level, privilege escalation (such as UAC in Windows), networking, load, perf and stress. It's not that I think that unit tests aren't good, they are, but I think people focus on them too much, to the detriment of other testing. Integration testing should catch all the things the unit tests would catch as well, the only difference is you have two copies of the test, and the unit test will give you more granularity into what is failing when you look at the test suite results. But as soon as you have a call stack or a logging facility (especially around error conditions), which is best practice, most of this is obvious as soon as you look at the result of the test.
- ajmurmann 9y ago
  I totally agree that unit tests for bug prevention in new code are overrated. I mainly like them because they 1. Encourage decoupled code and 2. Make future refactoring easier especially in dynamically typed languages. Not having other tests is a huge mistake. I think I'm spoiled because I've spent my entire career at places that cared deeply about testing and proper TDD.