17 ms·

Warming Up to Unit Testing

51 points by yumaikas 7y ago

barrowclift 7y ago
Bit of a disappointing read for me. Given the title, I was expecting more "meat" to the topic, like more details about why and how the author is now warming up to unit tests, anything to get the article beyond a surface-level summary of events. Right now, it simply reads "I didn't previously unit test, then I read some books, and now I do". Seems to me writing like this is best served with a few book recommendation tweets and doesn't really make sense in this format.
- yumaikas 7y ago
  (author here) Why: Mostly, I think it's down to having a better understanding of where to try to break dependencies, specifically trying to abstract away from the outside world (akin to how they did it in the Moonpig billing system). The how is still in progress. Perhaps the most notable instance I can think of it is that recently, due to work on breaking dependencies, I was able to stub out a RabbitMQ connection for an in-memory class, so that I was able to easily test how two components interacted without having to stand up an external service. For me, looking back, I think it was the Moonpig article that's had the most influence on how I think about things. Do you have any other questions?
crimsonalucard 7y ago
When you first hear about unit tests it seems like the greatest idea ever. Automation of a process you already do manually makes that process repeated-able and quick. When I was a junior programmer I was all over it. Over the years I have come to reject some things that are seemingly fundamental to software engineering in the industry. One of them is unit testing. I don't believe that unit testing is harmful in itself, but excessive use of it is a sign of weakness in another area. - if your code relies on unit testing for correctness than your coding methodology (or language) is error prone. A good programming methodology where the domain is well understood usually does not need a single unit test. - If your code base utilizes dependency injection to make the code more "testable" with mocks, you are making the code 10x worse. There are other ways to make your code more modular without injecting new scope, logic and apis into an object. This logic sort of applies integration tests which is basically testing the same thing as unit tests but touching functions that have IO. However it is a little different. If your program does Not need unit tests but needs integration tests it means things like: - your type checker does not extend across systems - the programming methodologies become more error prone as you move to other systems and you have no control over it. - you lack understanding of the foreign system. The thing with integration tests is that all of the above is largely inevitable and integration testing really becomes the only way to correlate (not verify) the entire system with correctness when you lack the integrated approach of a single programming language. Therefore because of this I am not against integration testing. I recently green fielded a small micro-service at my company and I did not write a single unit test. It had one logic error and that was largely due to a flaw in the type checker. I can imagine many engineers being aghast at the whole concept. So essentially I've had the opposite conclusion. Unit testing to me is a symptom of bad engineering practices. If you rely on it, something is wrong.
- MattPalmer1086 7y ago
  Speaking as someone who has fielded many reliable bits of software at least a decade or more before unit testing became a thing, I can't agree unit testing is a sign of bad engineering. Tests add real value for me. They are now usually the first "user" of my code. I often find design issues when writing tests. They also catch bugs (rarely but often enough to be useful) and make it easier to refactor with confidence. YMMV!
  - crimsonalucard 7y ago
    It's a sign that your programming methodology is so error prone that you need unit tests to catch errors. Most of the code I write doesn't need unit tests (automated or manual) because the way I programmed it reduces the complexity of the program to the point where my intuition and the type checker alone will cover correctness. Basically another way to put is... the code doesn't have any errors in the first place so any form of unit testing isn't really needed. Note that I am talking only about unit tests or tests that cover pure internal logic. Anything that touches IO or anything external becomes an integration test and That is not what I'm talking about.
    - bobm_kite9 7y ago
      Can you point us to some code that demonstrates how you do this?
      - crimsonalucard 7y ago
        Haskell code. I would try coding some solution in haskell with minimal IO. You should only need the type checker to verify correctness.
- war1025 7y ago
  > If your code base utilizes dependency injection to make the code more "testable" with mocks, you are making the code 10x worse. I don't know that I'd word it exactly like that, but one thing I've found over the years is any tests we have that rely on mocks basically just don't test anything at all. If your whole test is "This gets called, then that, then that", you aren't actually writing a test. You're repeating what the code already says.
kiwicopple 7y ago
We started using doctests to do our unit testing: https://github.com/supabase/doctest-js/blob/master/README.md https://github.com/supabase/doctest-js/blob/master/README.md This was a good way to get our coverage up. Something about writing the ‘test’ in the context of the function makes it seem easier. Also it helps that you get tests + docs + intellisense all in one fell swoop.
aazaa 7y ago
The article doesn't mention the single most import reason for unit testing: to enable you to refactor your code. Without a suite of unit tests, there's no way to know that your refactor hasn't introduced bugs of some kind. A suite of tests won't necessarily catch every bug you'll introduce, but it will catch many of them. I've seen this repeatedly. I've had many conversations with developers who claim they don't need unit tests. When I ask "How do you refactor?" the answers is usually some variant of "I don't."
- hota_mazi 7y ago
  You need "tests" to be able to refactor. They don't have to be unit tests.
- granshaw 7y ago
  There’s more nuance to this - poorly written unit tests (eg excessive mocking and things like testing private methods) actually make it harder to refactor code, and using strongly typed languages vastly reduces the importance of unit testing for refractors (to the point where the IDE can even do things like method extraction for you)
  - VBprogrammer 7y ago
    I propose "Granshaw's law". Every time the value of unit tests is discussed the debate over static vs dynamic typing must be looming eerily in the background. That said, as someone who has spent the majority of his career writing dynamically typed code I can definitely recall far more instances where statically typed code has run and worked first time vs times when I've written some code with unit tests and it worked first time.
  - Ididntdothis 7y ago
    When I was contractor there was a project where the tests were so extensive and complex that most change was impossible because of the impact on the tests. Testing private methods is especially evil.
    - 7y ago
squirrel 7y ago
Still my favourite source on unit testing and mocking: http://www.growing-object-oriented-software.com/ http://www.growing-object-oriented-software.com/
justanotherc 7y ago
I admit I still haven't really latched onto unit testing. I gave it an honest effort about a year ago, but the amount of time I was spending to mock out the inputs and then actually writing the test was absurd and I didn't feel provided enough benefits for the time out took me away from "productive" coding. Maybe I'm doing it wrong, who knows...
- notacoward 7y ago
  Maybe, but there are a couple of more likely explanations. (1) You're working on a system that is poorly structured for testing. (2) You're trying to make a unit test do an integration test's job. Both are incredibly common. In particular, if you find that you're spending a lot of time creating mocks/stubs to support unit tests you'd probably be better off finding a way to inject errors in an integration test. Error injection can be done at any level, works even when the code structure makes mocking difficult, and exercises code paths that cross multiple components in their "final" close-to-production form. IMX most of the people who go crazy about unit tests just haven't thought deeply enough about the problem to realize that error injection is strictly better (and a lot easier) most of the time.
  - JoeAltmaier 7y ago
    There's quite a bit of "no true Scotsman" in there? If few get 'unit testing' right, then I believe there may actually be something wrong with it, not just a failure to 'do it right'. Mocking is for exercising all inputs to a method, not just error injection. In fact that's fundamental to unit testing, and a necessary part. And it can grow to become a nightmare.
    - notacoward 7y ago
      > There's quite a bit of "no true Scotsman" in there? Only in your head. I'm not trying to claim unit tests aren't useful, or that any particular unit tests aren't good. The only fallacy in sight here is strawman. > Mocking is for exercising all inputs to a method How useful is that if they're values that the real version of the mocked component can never return? As I said right at the start, this kind of validation has value when changing code later, but that's a lower priority than validating that the whole system as it is today does what it's supposed to. Doing unit tests right is valuable, but is much more of an art than most people realize. A good unit test should cover inputs and paths well, but not constrain implementations. Most unit tests I've actually seen - quite a lot BTW - are the exact opposite. People need to stop thinking in terms of depth (adding mock after mock and implementation-specific check after check for a few obvious scenarios) and think in terms of breadth (lighter weight validation of more scenarios) instead. P.S. If your "inputs" are results from functions that have to be mocked for testing, it's often a sign that the code has a backwards control flow. When that happens, it's better to fix the control flow than to add more mocks.

51 points by yumaikas 7y ago

Bit of a disappointing read for me. Given the title, I was expecting more "meat" to the topic, like more details about why and how the author is now warming up to unit tests, anything to get the article beyond a surface-level summary of events. Right now, it simply reads "I didn't previously unit test, then I read some books, and now I do". Seems to me writing like this is best served with a few book recommendation tweets and doesn't really make sense in this format.

yumaikas 7y ago
(author here) Why: Mostly, I think it's down to having a better understanding of where to try to break dependencies, specifically trying to abstract away from the outside world (akin to how they did it in the Moonpig billing system). The how is still in progress. Perhaps the most notable instance I can think of it is that recently, due to work on breaking dependencies, I was able to stub out a RabbitMQ connection for an in-memory class, so that I was able to easily test how two components interacted without having to stand up an external service. For me, looking back, I think it was the Moonpig article that's had the most influence on how I think about things. Do you have any other questions?

crimsonalucard 7y ago

When you first hear about unit tests it seems like the greatest idea ever. Automation of a process you already do manually makes that process repeated-able and quick. When I was a junior programmer I was all over it. Over the years I have come to reject some things that are seemingly fundamental to software engineering in the industry. One of them is unit testing. I don't believe that unit testing is harmful in itself, but excessive use of it is a sign of weakness in another area. - if your code relies on unit testing for correctness than your coding methodology (or language) is error prone. A good programming methodology where the domain is well understood usually does not need a single unit test. - If your code base utilizes dependency injection to make the code more "testable" with mocks, you are making the code 10x worse. There are other ways to make your code more modular without injecting new scope, logic and apis into an object. This logic sort of applies integration tests which is basically testing the same thing as unit tests but touching functions that have IO. However it is a little different. If your program does Not need unit tests but needs integration tests it means things like: - your type checker does not extend across systems - the programming methodologies become more error prone as you move to other systems and you have no control over it. - you lack understanding of the foreign system. The thing with integration tests is that all of the above is largely inevitable and integration testing really becomes the only way to correlate (not verify) the entire system with correctness when you lack the integrated approach of a single programming language. Therefore because of this I am not against integration testing. I recently green fielded a small micro-service at my company and I did not write a single unit test. It had one logic error and that was largely due to a flaw in the type checker. I can imagine many engineers being aghast at the whole concept. So essentially I've had the opposite conclusion. Unit testing to me is a symptom of bad engineering practices. If you rely on it, something is wrong.

MattPalmer1086 7y ago
Speaking as someone who has fielded many reliable bits of software at least a decade or more before unit testing became a thing, I can't agree unit testing is a sign of bad engineering. Tests add real value for me. They are now usually the first "user" of my code. I often find design issues when writing tests. They also catch bugs (rarely but often enough to be useful) and make it easier to refactor with confidence. YMMV!
- crimsonalucard 7y ago
  It's a sign that your programming methodology is so error prone that you need unit tests to catch errors. Most of the code I write doesn't need unit tests (automated or manual) because the way I programmed it reduces the complexity of the program to the point where my intuition and the type checker alone will cover correctness. Basically another way to put is... the code doesn't have any errors in the first place so any form of unit testing isn't really needed. Note that I am talking only about unit tests or tests that cover pure internal logic. Anything that touches IO or anything external becomes an integration test and That is not what I'm talking about.
  - bobm_kite9 7y ago
    Can you point us to some code that demonstrates how you do this?
    - crimsonalucard 7y ago
      Haskell code. I would try coding some solution in haskell with minimal IO. You should only need the type checker to verify correctness.
war1025 7y ago
> If your code base utilizes dependency injection to make the code more "testable" with mocks, you are making the code 10x worse. I don't know that I'd word it exactly like that, but one thing I've found over the years is any tests we have that rely on mocks basically just don't test anything at all. If your whole test is "This gets called, then that, then that", you aren't actually writing a test. You're repeating what the code already says.

kiwicopple 7y ago

We started using doctests to do our unit testing: https://github.com/supabase/doctest-js/blob/master/README.md https://github.com/supabase/doctest-js/blob/master/README.md This was a good way to get our coverage up. Something about writing the ‘test’ in the context of the function makes it seem easier. Also it helps that you get tests + docs + intellisense all in one fell swoop.

aazaa 7y ago

The article doesn't mention the single most import reason for unit testing: to enable you to refactor your code. Without a suite of unit tests, there's no way to know that your refactor hasn't introduced bugs of some kind. A suite of tests won't necessarily catch every bug you'll introduce, but it will catch many of them. I've seen this repeatedly. I've had many conversations with developers who claim they don't need unit tests. When I ask "How do you refactor?" the answers is usually some variant of "I don't."

hota_mazi 7y ago
You need "tests" to be able to refactor. They don't have to be unit tests.
granshaw 7y ago
There’s more nuance to this - poorly written unit tests (eg excessive mocking and things like testing private methods) actually make it harder to refactor code, and using strongly typed languages vastly reduces the importance of unit testing for refractors (to the point where the IDE can even do things like method extraction for you)
- VBprogrammer 7y ago
  I propose "Granshaw's law". Every time the value of unit tests is discussed the debate over static vs dynamic typing must be looming eerily in the background. That said, as someone who has spent the majority of his career writing dynamically typed code I can definitely recall far more instances where statically typed code has run and worked first time vs times when I've written some code with unit tests and it worked first time.
- Ididntdothis 7y ago
  When I was contractor there was a project where the tests were so extensive and complex that most change was impossible because of the impact on the tests. Testing private methods is especially evil.
  - 7y ago

squirrel 7y ago

Still my favourite source on unit testing and mocking: http://www.growing-object-oriented-software.com/ http://www.growing-object-oriented-software.com/

justanotherc 7y ago

I admit I still haven't really latched onto unit testing. I gave it an honest effort about a year ago, but the amount of time I was spending to mock out the inputs and then actually writing the test was absurd and I didn't feel provided enough benefits for the time out took me away from "productive" coding. Maybe I'm doing it wrong, who knows...

notacoward 7y ago
Maybe, but there are a couple of more likely explanations. (1) You're working on a system that is poorly structured for testing. (2) You're trying to make a unit test do an integration test's job. Both are incredibly common. In particular, if you find that you're spending a lot of time creating mocks/stubs to support unit tests you'd probably be better off finding a way to inject errors in an integration test. Error injection can be done at any level, works even when the code structure makes mocking difficult, and exercises code paths that cross multiple components in their "final" close-to-production form. IMX most of the people who go crazy about unit tests just haven't thought deeply enough about the problem to realize that error injection is strictly better (and a lot easier) most of the time.
- JoeAltmaier 7y ago
  There's quite a bit of "no true Scotsman" in there? If few get 'unit testing' right, then I believe there may actually be something wrong with it, not just a failure to 'do it right'. Mocking is for exercising all inputs to a method, not just error injection. In fact that's fundamental to unit testing, and a necessary part. And it can grow to become a nightmare.
  - notacoward 7y ago
    > There's quite a bit of "no true Scotsman" in there? Only in your head. I'm not trying to claim unit tests aren't useful, or that any particular unit tests aren't good. The only fallacy in sight here is strawman. > Mocking is for exercising all inputs to a method How useful is that if they're values that the real version of the mocked component can never return? As I said right at the start, this kind of validation has value when changing code later, but that's a lower priority than validating that the whole system as it is today does what it's supposed to. Doing unit tests right is valuable, but is much more of an art than most people realize. A good unit test should cover inputs and paths well, but not constrain implementations. Most unit tests I've actually seen - quite a lot BTW - are the exact opposite. People need to stop thinking in terms of depth (adding mock after mock and implementation-specific check after check for a few obvious scenarios) and think in terms of breadth (lighter weight validation of more scenarios) instead. P.S. If your "inputs" are results from functions that have to be mocked for testing, it's often a sign that the code has a backwards control flow. When that happens, it's better to fix the control flow than to add more mocks.