7 ms·

I'm curious what language and techniques you mean this applies to. In my experience, I've seen a lot of code that was written in a way that was not unit-testab

by pqh 9y ago

I'm curious what language and techniques you mean this applies to. In my experience, I've seen a lot of code that was written in a way that was not unit-testable, for what was the same reason that later turned out to be poorly-maintainable.

loup-vaillant 9y ago
Some seem to use (their personal idea of) unit testability as a proxy for maintainability or quality. That is often invalid. I've written some high-quality code and tested it to death¹, only to be told that it probably wasn't unit testable (because dependencies were hard coded instead of injected, IIRC). Making that code unit testable would have complicated it, thus lowering its maintainability. [1] Not a single bug has ever been found after my tests.
- cbanek 9y ago
  Couldn't have said it better myself. I've had a lot of times where no bugs were found with huge amounts of unit testing over trivial code. It would have been easier and of more use to just do manual inspection (code review) with a bunch of people instead of writing the tests at that level, and instead writing tests at a higher level of complexity which involved more code/components. In a perfect world you'd test all the branches and possible values but out there in the real world, if you aren't finding bugs with your testing and there are still bugs in the code, you need to be spending your testing "dollars" (time/effort/talent) more efficiently.
  - loup-vaillant 9y ago
    I failed to mentioned that the tests themselves founds lots of bugs. I needed those tests. Also, those weren't mere individual tests, I ran a number of property-based tests as well, they tend to be much more thorough. I wrote those tests because I was writing foundational code. The outer layers hardly have any test, which is good enough for the proof of concept we were asked for. We'll need to be a bit more rigorous to polish this into production.
- forkerenok 9y ago
  Out of curiosity, how do you come around assessing it as high-quality code? By what measure(s)?
  - loup-vaillant 9y ago
    I have developed some kind of instinct that is non-trivial to teach, and impossible to convey in a small HN comment. But my primary proxy is size. The less code the better. But even that have to give way to other concerns, such as style constraints, and straight up performance. While I can't show the code I was talking about above (it was proprietary), I can show my crypto library, Monocypher¹. I think it is a good example of what I mean by high-quality code. (Yes, I am boasting. I believe this is justified. Also, quality requirements for crypto code are kinda off the chart.) [1]: https://github.com/LoupVaillant/Monocypher https://github.com/LoupVaillant/Monocypher
    - joshuamorton 9y ago
      I think crypto code is a bad example. This[1] would not be considered "idiomatic" anywhere, except in crypto. And for example, if one was optimizing for code readability over performance, I expect that would look very different (although you could do much of the same with macros, but I'm not sure if that's any better). [1]: https://github.com/LoupVaillant/Monocypher/blob/master/src/monocypher.c#L1070 https://github.com/LoupVaillant/Monocypher/blob/master/src/m...
      - loup-vaillant 9y ago
        Each domain has its own quirks. I'd definitely use a different style (even a different language) for other domains. I thought of showing some of my OCaml code as well, but I didn't put nearly as much effort, so I'm not so sure about its quality.
    - capitalsigma 9y ago
      #define FOR(i, start, end) for (size_t (i) = (start); (i) < (end); (i)++) yuck EDIT: Also, what the hell is this macro defined in the middle of a function and then only used twice: https://github.com/LoupVaillant/Monocypher/blob/master/src/monocypher.c#L1091 https://github.com/LoupVaillant/Monocypher/blob/master/src/m... Why do you need this function for "constant time comparison to 0": https://github.com/LoupVaillant/Monocypher/blob/master/src/monocypher.c#L68 https://github.com/LoupVaillant/Monocypher/blob/master/src/m... static int neq0(u64 diff) { // constant time comparison to zero // return diff != 0 ? -1 : 0 u64 half = (diff >> 32) | ((u32)diff); return (1 & ((half - 1) >> 32)) - 1; } I am incredibly skeptical that you've done better than the compiler with this bit twiddling stuff. You realize that on x86_64, it probably compiles down to 1-2 instructions to write `diff != 0`, right?
- lochlan 9y ago
  That strict definition of “unit” testing (advocating for isolationism and banning collaboration with dependencies) is too narrow for my taste. DI can make a lot of sense, but I don’t think that means you have to mock everything (inviting a maintenance cost) for the sake of purity. Here’s Fowler’s take: https://martinfowler.com/bliki/UnitTest.html https://martinfowler.com/bliki/UnitTest.html
- grogenaut 9y ago
  [1] can easily mean that no one used the software so it doesn't really tell us much. Also, unless you had actually re-written the code as unit testable how do you know it's more complicated that way or less maintainable? You're responding to a study with actual data, on a generally "preference" topic, with an anecdote so yes I do feel justified in pointing this out.
  - loup-vaillant 9y ago
    > [1] can easily mean that no one used the software so it doesn't really tell us much. This was foundational code, used for inter-module communication all over the place. > unless you had actually re-written the code as unit testable how do you know it's more complicated that way or less maintainable? Because the code was small enough to allow me to envision the necessary modifications for dependency injections. I had 3 classes, with an A->B->C dependency chain, and no reason to inject anything if it weren't for some cargo cult about unit tests. Testing the hell out of C, then B, then A, proved quite sufficient without mocking or injecting anything. > You're responding to a study with actual data, on a generally "preference" topic, with an anecdote so yes I do feel justified in pointing this out. Whatever evidence the study actually has is weak. Small sample size, and the failure to analyse actual outcomes (bugs, speed of development…) mean we cannot possibly get much out of it. It's a good starting point. I don't believe I have contradicted this study's evidence. But even if I did, my personal experience gives me way more evidence than such a study, so I feel perfectly justified in contradicting it on that basis. (More solidly settled science, that's another story.) Problem is, you do not have a privileged access to my personal experience. You only know what I just wrote. And my written report of my personal experience means little, next to that study. You'd better believe the study before you believe my report. Then you have your own personal experience.
    - grogenaut 9y ago
      Well stated and agreed with in the final paragraphs but why drop the half written anecdote? Also I think you may be conflating DI with testability. Items can be u nit testable and easy to di while not being built for it. I write code that is testable and it generally happens to be di-able as well but that's not the goal. From my own experience.
- msangi 9y ago
  What really makes the difference in terms of maintainability is how easy is to run the tests and how much they're automated. What's important is to have test coverage and to keep exercising it when the code changes. The best way to achieve this depends case by case. Sometimes unit tests are better, sometimes integration test is the way to go.
- pqh 9y ago
  I think good code usually results in the test suite being useless the vast majority of the time. That is, only let's say one run in a thousand of any given test should fail after adding or refactoring code, if that. Even then, the failures are usually commensurate with the changed expectations of the changed code and not indicative of a bug.
- curun1r 9y ago
  What makes unit tested code maintainable has nothing to do with how application code is structured and is entirely about the ability to know in an automated fashion whether the previous expectations for how the code should function still hold after a refactoring. Business realities change over time and there are lessons learned that get woven into the code. If we're not careful, refactoring code can lead to losing those lessons. Encoding those lessons as executable tests allows us to better preserve that knowledge as a piece of code evolves. Where unit tests have an advantage and where you might have been criticized is their ability to test permutations. If you write three tests for one function and three tests for a dependency of that function, you've essentially covered nine cases with six tests. In a complex application with a large dependency graph, that ability to multiply the value of each test can be difficult to overcome when testing at a higher level. Integration tests rarely cover edge cases. Additionally, unit tests are often quite simple to read and have limited setup which means that a programmer coming to the code base without prior knowledge of the code (I always pictured that being me six months in the future so that I'd be as kind as possible :-) can easily see exactly what is expected of the code. But we shouldn't be dogmatic about the kind of testing we do and lose sight of the goal of that testing in the first place. Any test that furthers a future programmer's ability to rip the application code apart, put it back together and quickly know whether all the previous expectations of the code are still met is a good test. There's no hard and fast rule and judgment and experience are still necessary to know what to do in a specific situation. Too many people focus on the details of automated testing and lose sight of the goal.
  - pqh 9y ago
    I agree with the latter parts of your comment, but not that "What makes unit tested code maintainable has nothing to do with how application code is structured". My original point was that code that's hard to test is often hard to refactor.
    - curun1r 9y ago
      > My original point was that code that's hard to test is often hard to refactor I agree with that. The poster I was replying to said somewhat the opposite. He said: > Making that code unit testable would have complicated it, thus lowering its maintainability My basic point was that less complicated doesn't mean maintainable. There's no perfect design in the face of future changes in requirements. So the most important quality of the code we write today is the ability to refactor it and know that it still satisfies all the initial requirements that haven't changed. Unit tests give you that property, lack of complexity doesn't. Simple code can be better than complex, but it's a secondary concern to a comprehensive test suite. I'll take a full test suite over code quality any day because it allows me to go in and add the quality later without fear of breaking things. I guess I misspoke when I said "how the application is structured" since it you've interpreted it differently than I meant it. It was a reference to the line I quoted above...that complicating code lowers it's maintainability. I just meant that complexity and tested are separate concerns and that tests more so than simplicity make code maintainable. So I think we agree more than we disagree.

by pqh 9y ago

loup-vaillant 9y ago

Some seem to use (their personal idea of) unit testability as a proxy for maintainability or quality. That is often invalid. I've written some high-quality code and tested it to death¹, only to be told that it probably wasn't unit testable (because dependencies were hard coded instead of injected, IIRC). Making that code unit testable would have complicated it, thus lowering its maintainability. [1] Not a single bug has ever been found after my tests.

cbanek 9y ago
Couldn't have said it better myself. I've had a lot of times where no bugs were found with huge amounts of unit testing over trivial code. It would have been easier and of more use to just do manual inspection (code review) with a bunch of people instead of writing the tests at that level, and instead writing tests at a higher level of complexity which involved more code/components. In a perfect world you'd test all the branches and possible values but out there in the real world, if you aren't finding bugs with your testing and there are still bugs in the code, you need to be spending your testing "dollars" (time/effort/talent) more efficiently.
- loup-vaillant 9y ago
  I failed to mentioned that the tests themselves founds lots of bugs. I needed those tests. Also, those weren't mere individual tests, I ran a number of property-based tests as well, they tend to be much more thorough. I wrote those tests because I was writing foundational code. The outer layers hardly have any test, which is good enough for the proof of concept we were asked for. We'll need to be a bit more rigorous to polish this into production.
forkerenok 9y ago
Out of curiosity, how do you come around assessing it as high-quality code? By what measure(s)?
- loup-vaillant 9y ago
  I have developed some kind of instinct that is non-trivial to teach, and impossible to convey in a small HN comment. But my primary proxy is size. The less code the better. But even that have to give way to other concerns, such as style constraints, and straight up performance. While I can't show the code I was talking about above (it was proprietary), I can show my crypto library, Monocypher¹. I think it is a good example of what I mean by high-quality code. (Yes, I am boasting. I believe this is justified. Also, quality requirements for crypto code are kinda off the chart.) [1]: https://github.com/LoupVaillant/Monocypher https://github.com/LoupVaillant/Monocypher
  - joshuamorton 9y ago
    I think crypto code is a bad example. This[1] would not be considered "idiomatic" anywhere, except in crypto. And for example, if one was optimizing for code readability over performance, I expect that would look very different (although you could do much of the same with macros, but I'm not sure if that's any better). [1]: https://github.com/LoupVaillant/Monocypher/blob/master/src/monocypher.c#L1070 https://github.com/LoupVaillant/Monocypher/blob/master/src/m...
    - loup-vaillant 9y ago
      Each domain has its own quirks. I'd definitely use a different style (even a different language) for other domains. I thought of showing some of my OCaml code as well, but I didn't put nearly as much effort, so I'm not so sure about its quality.
  - capitalsigma 9y ago
    #define FOR(i, start, end) for (size_t (i) = (start); (i) < (end); (i)++) yuck EDIT: Also, what the hell is this macro defined in the middle of a function and then only used twice: https://github.com/LoupVaillant/Monocypher/blob/master/src/monocypher.c#L1091 https://github.com/LoupVaillant/Monocypher/blob/master/src/m... Why do you need this function for "constant time comparison to 0": https://github.com/LoupVaillant/Monocypher/blob/master/src/monocypher.c#L68 https://github.com/LoupVaillant/Monocypher/blob/master/src/m... static int neq0(u64 diff) { // constant time comparison to zero // return diff != 0 ? -1 : 0 u64 half = (diff >> 32) | ((u32)diff); return (1 & ((half - 1) >> 32)) - 1; } I am incredibly skeptical that you've done better than the compiler with this bit twiddling stuff. You realize that on x86_64, it probably compiles down to 1-2 instructions to write `diff != 0`, right?
lochlan 9y ago
That strict definition of “unit” testing (advocating for isolationism and banning collaboration with dependencies) is too narrow for my taste. DI can make a lot of sense, but I don’t think that means you have to mock everything (inviting a maintenance cost) for the sake of purity. Here’s Fowler’s take: https://martinfowler.com/bliki/UnitTest.html https://martinfowler.com/bliki/UnitTest.html
grogenaut 9y ago
[1] can easily mean that no one used the software so it doesn't really tell us much. Also, unless you had actually re-written the code as unit testable how do you know it's more complicated that way or less maintainable? You're responding to a study with actual data, on a generally "preference" topic, with an anecdote so yes I do feel justified in pointing this out.
- loup-vaillant 9y ago
  > [1] can easily mean that no one used the software so it doesn't really tell us much. This was foundational code, used for inter-module communication all over the place. > unless you had actually re-written the code as unit testable how do you know it's more complicated that way or less maintainable? Because the code was small enough to allow me to envision the necessary modifications for dependency injections. I had 3 classes, with an A->B->C dependency chain, and no reason to inject anything if it weren't for some cargo cult about unit tests. Testing the hell out of C, then B, then A, proved quite sufficient without mocking or injecting anything. > You're responding to a study with actual data, on a generally "preference" topic, with an anecdote so yes I do feel justified in pointing this out. Whatever evidence the study actually has is weak. Small sample size, and the failure to analyse actual outcomes (bugs, speed of development…) mean we cannot possibly get much out of it. It's a good starting point. I don't believe I have contradicted this study's evidence. But even if I did, my personal experience gives me way more evidence than such a study, so I feel perfectly justified in contradicting it on that basis. (More solidly settled science, that's another story.) Problem is, you do not have a privileged access to my personal experience. You only know what I just wrote. And my written report of my personal experience means little, next to that study. You'd better believe the study before you believe my report. Then you have your own personal experience.
  - grogenaut 9y ago
    Well stated and agreed with in the final paragraphs but why drop the half written anecdote? Also I think you may be conflating DI with testability. Items can be u nit testable and easy to di while not being built for it. I write code that is testable and it generally happens to be di-able as well but that's not the goal. From my own experience.
msangi 9y ago
What really makes the difference in terms of maintainability is how easy is to run the tests and how much they're automated. What's important is to have test coverage and to keep exercising it when the code changes. The best way to achieve this depends case by case. Sometimes unit tests are better, sometimes integration test is the way to go.
pqh 9y ago
I think good code usually results in the test suite being useless the vast majority of the time. That is, only let's say one run in a thousand of any given test should fail after adding or refactoring code, if that. Even then, the failures are usually commensurate with the changed expectations of the changed code and not indicative of a bug.
curun1r 9y ago
What makes unit tested code maintainable has nothing to do with how application code is structured and is entirely about the ability to know in an automated fashion whether the previous expectations for how the code should function still hold after a refactoring. Business realities change over time and there are lessons learned that get woven into the code. If we're not careful, refactoring code can lead to losing those lessons. Encoding those lessons as executable tests allows us to better preserve that knowledge as a piece of code evolves. Where unit tests have an advantage and where you might have been criticized is their ability to test permutations. If you write three tests for one function and three tests for a dependency of that function, you've essentially covered nine cases with six tests. In a complex application with a large dependency graph, that ability to multiply the value of each test can be difficult to overcome when testing at a higher level. Integration tests rarely cover edge cases. Additionally, unit tests are often quite simple to read and have limited setup which means that a programmer coming to the code base without prior knowledge of the code (I always pictured that being me six months in the future so that I'd be as kind as possible :-) can easily see exactly what is expected of the code. But we shouldn't be dogmatic about the kind of testing we do and lose sight of the goal of that testing in the first place. Any test that furthers a future programmer's ability to rip the application code apart, put it back together and quickly know whether all the previous expectations of the code are still met is a good test. There's no hard and fast rule and judgment and experience are still necessary to know what to do in a specific situation. Too many people focus on the details of automated testing and lose sight of the goal.
- pqh 9y ago
  I agree with the latter parts of your comment, but not that "What makes unit tested code maintainable has nothing to do with how application code is structured". My original point was that code that's hard to test is often hard to refactor.
  - curun1r 9y ago
    > My original point was that code that's hard to test is often hard to refactor I agree with that. The poster I was replying to said somewhat the opposite. He said: > Making that code unit testable would have complicated it, thus lowering its maintainability My basic point was that less complicated doesn't mean maintainable. There's no perfect design in the face of future changes in requirements. So the most important quality of the code we write today is the ability to refactor it and know that it still satisfies all the initial requirements that haven't changed. Unit tests give you that property, lack of complexity doesn't. Simple code can be better than complex, but it's a secondary concern to a comprehensive test suite. I'll take a full test suite over code quality any day because it allows me to go in and add the quality later without fear of breaking things. I guess I misspoke when I said "how the application is structured" since it you've interpreted it differently than I meant it. It was a reference to the line I quoted above...that complicating code lowers it's maintainability. I just meant that complexity and tested are separate concerns and that tests more so than simplicity make code maintainable. So I think we agree more than we disagree.