7 ms·

I used to have the same attitude toward unit tests for my solo projects… But a few years ago I was working on library that had to do a lot of vector math, and I

I used to have the same attitude toward unit tests for my solo projects… But a few years ago I was working on library that had to do a lot of vector math, and I was doing SO much manual checking after every change. I wrote a bunch of tests so that I could see what was breaking when. What I didn't expect is that the tests gave me so much confidence to improve, extend and refactor the code. Before it felt so fragile, like the whole thing could crumble if I don't quadruple check my work. With the tests in place, I changed the entire underlying data type from a struct to a native simd vector in an hour. Made the changes, fixed a couple of bugs, merged the branch, boom. I don't strive for anything close to 100% coverage, but the handful of tests that cover my critical code have saved me so much time.

Jtsummers 4y ago
"Confidence" is a good word for it. When teams don't have good test suites (meaning mostly automated, if not entirely, and reasonably comprehensive), then you can watch as over time you get either increasing error rates or a massive slow down in team performance. Even for small changes, having no tests or primarily manual end-to-end/integration style tests means that you have little chance, unless it's a particularly good code base, of determining the actual effects statically (by analysis) or dynamically (because your tests will take too long and are likely mostly ad hoc so you will miss edge cases). Teams that don't care (or are pressured into not caring) will deliver crappier and crappier software over time. Teams that do care will eventually stop delivering software at all, or at least no more meaningful updates. All for lack of confidence.
- deleted 4y ago
  [deleted]
- hbrn 4y ago
  Unit tests can never give you "confidence". > you can watch as over time you get either increasing error rates or a massive slow down in team performance I hear this story repeated over and over and yet I consistently witness the opposite of what you're describing: teams start investing in unit testing and they noticeably slow down and ship more bugs at the same time. What can give you confidence is simplicity. Being able to fit the entire system in your head. Being able to say what are the side effects of your changes. If you feel like you're losing those, it's time to take a step back and rethink your architecture. But of course, nobody has the balls to take a step back and think, to admit that you failed at something. It's so much easier to just slap a bunch of unit tests on your big ball of mud and hope for the best.
  - Jtsummers 4y ago
    I didn't mention unit tests, so maybe you meant to respond to someone else? Or you're just shouting into the wind about something that wasn't said for fun? I was writing about tests generally, in case you were wondering.
    - hbrn 4y ago
      The comment you responded to says unit tests give confidence, you seem to agree. "Tests generally" and unit tests are pretty much the opposite camps in testing world.
      - Jtsummers 4y ago
        I agreed about the word "confidence" and the importance of testing for confidence. You read something into my comment that I did not write nor did I intend. Good for you, I guess.
  - turnsout 4y ago
    If you can fit the entire system in your head, it's probably a relatively simple codebase. I've written a Photoshop file format parser twice, and trust me—between byte alignment changing constantly throughout the spec, at least three different string encodings, sub-formats and errors in the spec—it does not "fit in your head." Certainly not enough to confidently add features without worrying that you've broken the blend mode parser. And of course, if you can and do fit a complex system in your head when no one else on the team can, congratulations: you've made yourself into a bottleneck and flight risk.
    - hbrn 4y ago
      > If you can fit the entire system in your head, it's probably a relatively simple codebase. Or it has good boundaries and I never have the need to load the whole system in my head. You're not loading the whole OS kernel in your head every time you call fopen(), do you? > Photoshop file format parser Would you say that PSD format is an example of good design? Is complexity inherent to the problem domain, or is it accidental? You are forced to replicate complexity created by other engineers. Woe is you. My bet is that Adobe also has a lot of unit tests, and contrary to what unit-testing advocates claim, it resulted in a terrible design you are dealing with.
CharlieDigital 4y ago
Not having tests for complex things make such constructs feel fragile. If you're building simple things, tests can be overkill. If you're building complex things, tests pay for themselves rather quickly.
- hbrn 4y ago
  If you're building complex things... most likely you should be building simple things. Most software in the world implements business rules defined by people who are way less smart than your average engineer. And yet our software is more complicated than ever.
  - CharlieDigital 4y ago
    There's a difference between accidental and necessary complexity. Not everything can be simple. Mathematical algorithms are an example where there's necessary complexity and unit tests backstop defects when you refactor code.
    - turnsout 4y ago
      ^ THIS, 100%. Some SW engineers truly believe that complexity is the enemy, and that it can always be eliminated. Sometimes it can, but other times, you can watch developers push complexity around on their plate to avoid eating it. They move logic to a different library, break it into 10 libraries, introduce layers of abstraction, try increasingly complicated View/Model architectures… And ultimately the code is no easier to understand or manage than when they started.
      - hbrn 4y ago
        But complexity is the enemy. Where it can't be eliminated, it can be contained. That's what software engineering is about. It is nearly impossible to find a complex domain that doesn't naturally break into smaller subdomains. That's because "domain" is human concept. The ones that are too complex and don't have natural boundaries within them just don't stick. There are two main reasons software becomes complicated: 1. It is over-engineered at the beginning of it's lifecycle. 2. It is under-engineered in the middle of it's lifecycle. Interestingly, unit tests contribute to both of these. For an engineer it's always hard to admit that your system is complex because you failed at engineering. It's much easier to claim that you're working in a complex domain: you get to shift blame and boost your ego at the same time.
gavinhoward 4y ago
I agree with the sibling comment that says confidence is a good word, because it is. I've done three or four major refactors on a FOSS project of mine. The refactors went without any hitch because of an almost excessive test suite.
ccll 4y ago
This is great example of unit tests, but I don’t think it’s comparable with OP’s scenario. Math laws are ‘fixed’ and well known, you can just write fixed tests to find the discrepancy between the laws and your implentation. Where OP’s point is about ‘startup business logics’, the ‘laws’ here are constantly changing all the time, which means you will constantly throw out most of your tests along the way.

Jtsummers 4y ago

"Confidence" is a good word for it. When teams don't have good test suites (meaning mostly automated, if not entirely, and reasonably comprehensive), then you can watch as over time you get either increasing error rates or a massive slow down in team performance. Even for small changes, having no tests or primarily manual end-to-end/integration style tests means that you have little chance, unless it's a particularly good code base, of determining the actual effects statically (by analysis) or dynamically (because your tests will take too long and are likely mostly ad hoc so you will miss edge cases). Teams that don't care (or are pressured into not caring) will deliver crappier and crappier software over time. Teams that do care will eventually stop delivering software at all, or at least no more meaningful updates. All for lack of confidence.

deleted 4y ago
[deleted]
hbrn 4y ago
Unit tests can never give you "confidence". > you can watch as over time you get either increasing error rates or a massive slow down in team performance I hear this story repeated over and over and yet I consistently witness the opposite of what you're describing: teams start investing in unit testing and they noticeably slow down and ship more bugs at the same time. What can give you confidence is simplicity. Being able to fit the entire system in your head. Being able to say what are the side effects of your changes. If you feel like you're losing those, it's time to take a step back and rethink your architecture. But of course, nobody has the balls to take a step back and think, to admit that you failed at something. It's so much easier to just slap a bunch of unit tests on your big ball of mud and hope for the best.
- Jtsummers 4y ago
  I didn't mention unit tests, so maybe you meant to respond to someone else? Or you're just shouting into the wind about something that wasn't said for fun? I was writing about tests generally, in case you were wondering.
  - hbrn 4y ago
    The comment you responded to says unit tests give confidence, you seem to agree. "Tests generally" and unit tests are pretty much the opposite camps in testing world.
    - Jtsummers 4y ago
      I agreed about the word "confidence" and the importance of testing for confidence. You read something into my comment that I did not write nor did I intend. Good for you, I guess.
- turnsout 4y ago
  If you can fit the entire system in your head, it's probably a relatively simple codebase. I've written a Photoshop file format parser twice, and trust me—between byte alignment changing constantly throughout the spec, at least three different string encodings, sub-formats and errors in the spec—it does not "fit in your head." Certainly not enough to confidently add features without worrying that you've broken the blend mode parser. And of course, if you can and do fit a complex system in your head when no one else on the team can, congratulations: you've made yourself into a bottleneck and flight risk.
  - hbrn 4y ago
    > If you can fit the entire system in your head, it's probably a relatively simple codebase. Or it has good boundaries and I never have the need to load the whole system in my head. You're not loading the whole OS kernel in your head every time you call fopen(), do you? > Photoshop file format parser Would you say that PSD format is an example of good design? Is complexity inherent to the problem domain, or is it accidental? You are forced to replicate complexity created by other engineers. Woe is you. My bet is that Adobe also has a lot of unit tests, and contrary to what unit-testing advocates claim, it resulted in a terrible design you are dealing with.

CharlieDigital 4y ago

Not having tests for complex things make such constructs feel fragile. If you're building simple things, tests can be overkill. If you're building complex things, tests pay for themselves rather quickly.

hbrn 4y ago
If you're building complex things... most likely you should be building simple things. Most software in the world implements business rules defined by people who are way less smart than your average engineer. And yet our software is more complicated than ever.
- CharlieDigital 4y ago
  There's a difference between accidental and necessary complexity. Not everything can be simple. Mathematical algorithms are an example where there's necessary complexity and unit tests backstop defects when you refactor code.
  - turnsout 4y ago
    ^ THIS, 100%. Some SW engineers truly believe that complexity is the enemy, and that it can always be eliminated. Sometimes it can, but other times, you can watch developers push complexity around on their plate to avoid eating it. They move logic to a different library, break it into 10 libraries, introduce layers of abstraction, try increasingly complicated View/Model architectures… And ultimately the code is no easier to understand or manage than when they started.
    - hbrn 4y ago
      But complexity is the enemy. Where it can't be eliminated, it can be contained. That's what software engineering is about. It is nearly impossible to find a complex domain that doesn't naturally break into smaller subdomains. That's because "domain" is human concept. The ones that are too complex and don't have natural boundaries within them just don't stick. There are two main reasons software becomes complicated: 1. It is over-engineered at the beginning of it's lifecycle. 2. It is under-engineered in the middle of it's lifecycle. Interestingly, unit tests contribute to both of these. For an engineer it's always hard to admit that your system is complex because you failed at engineering. It's much easier to claim that you're working in a complex domain: you get to shift blame and boost your ego at the same time.

gavinhoward 4y ago

I agree with the sibling comment that says confidence is a good word, because it is. I've done three or four major refactors on a FOSS project of mine. The refactors went without any hitch because of an almost excessive test suite.

ccll 4y ago

This is great example of unit tests, but I don’t think it’s comparable with OP’s scenario. Math laws are ‘fixed’ and well known, you can just write fixed tests to find the discrepancy between the laws and your implentation. Where OP’s point is about ‘startup business logics’, the ‘laws’ here are constantly changing all the time, which means you will constantly throw out most of your tests along the way.