3 ms·

unfortunately legitimate use cases for unit tests (like this) are pretty rare in corporate codebases, overwhelmingly, unit tests are just mocked tests that enf

by bedobi 3y ago

unfortunately legitimate use cases for unit tests (like this) are pretty rare in corporate codebases, overwhelmingly, unit tests are just mocked tests that enforce a certain implementation at the class or even individual method/function level and pretend that it works, making it impossible to refactor anything or even fix bugs without breaking tests such tests are not just useless, they're positively harmful https://gist.github.com/androidfred/501d276c7dc26a5db09e893bb7c9f3ee https://gist.github.com/androidfred/501d276c7dc26a5db09e893b...

bobthepanda 3y ago
The point of unit tests is not to CYA during refactors, but to confirm that the implementation is consistent between small changes without weird side effects. A coworker once thought unit tests were dumb, and ended up writing code that repeated the call to an application 10x for the same info. This didn’t result in a changed UI because it was a read, but it’s not good to just suddenly 10x your reads for no good reason. TFA also describes discovering weird side effect race conditions as a result of unit tests.
- domano 3y ago
  Using mockserver etc. you can cover for these things in component-test cases even more easily through your whole application while being more flexible with bigger code changes than unit tests allow.
  - bobthepanda 3y ago
    A lot of places on the internet treat component testing and unit testing as synonyms. I’ve never heard of the former and it basically sounds like the unit tests as we write.
- bedobi 3y ago
  > confirm that the implementation is consistent between small changes without weird side effects not sure what this is referring to, but I'll give an example say you have a requirement that says if you call POST /user with a non-existing user, a user should be created and you should get a 2xx response with some basic details back you could test this by actually hitting the endpoint with randomly generated user data known to not already exist, check that you get the expected 2xx response in the expected format, and then use the user id you got back to call the GET /user/userId endpoint and check that it's the same user that was just created this is a great test! it enforces actual business logic while still allowing you to change literally everything about the implementation - you could change the codebase from Java Spring Boot to Python Flask if you wanted to, you could change the persistence tech from MySQL to MariaDB or Redis etc etc - the test would still pass when the endpoint behaves as expected and fail when it doesn't, and it's a single test that is cheap to write, maintain and run OR you could write dozens of the typical corporate style unit test i'm referring to, where you create instances of each individual layer class, mocks of every class it interacts with, mocked database calls etc etc which 1) literally enforce every single aspect of the implementation, so now you can't change anything without breaking the tests 2) pretend that things work when they actually don't (eg, it could be that the CreateUserDAO actually breaks because someone stuffs up a db call, but guess what, the CreateUserResource and CreateUserService unit tests will still pass, because they just pretend (through mocks) that CreateUserDao.createUser returns a created user
  - bobthepanda 3y ago
    I literally gave examples in my example. As a general example, a unit test is great for things like: Your ExampleFactory calls your ExampleSerivce exactly once, not more, not less, so you can check that side effects don’t result in unnecessary extra calls in more load. This is particularly relevant in a language like Java; modern Java style is functional, but the old style relied heavily on side effects, and they’re still possible to write unintentionally.
    - Verdex 3y ago
      I believe the kind of scenario that bedobi is referring to is something like this (using your example): Unit test exists, ExampleFactory only calls ExampleService once. Hmm, it turns out that ExampleUIButton calls ExamplePoxyNavigator more than one time if the text in ExampleUIButton happens to be wider than the default width for the button. What does ExampleProxyNavigator do? Oh, it calls ExampleFactory which calls ExampleService. But only when the whole system is wired up for deployment. The unit tests indicate that the system should be functioning okay, but when you put everything together you find out that the system does not function okay.
    - bedobi 3y ago
      I understand your example and I agree that in that particular case, maybe a unit test to ensure a given service calls a dao once and only once or whatever is justified but I don't think the hypothetical risk of someone needlessly calling the db ten times is a good reason to justify adding that style unit tests to everything by default - if it happens, sure, add one for that particular call and call it a day
  - Verdex 3y ago
    To be fair to unit tests, I really like them for making sure that complicated code gets tested thoroughly. However, very often the complicated code isn't isolated to a single unit. It instead lives distributed amongst multiple objects that are reused for several competing aspects and all have their own undocumented assumptions about how the world works. Now maybe this implies that we need a wide scale change in coding methodology such that the complicated parts are all isolated to units. But pending that, I'm not sure that the answer is a bunch of static mocks pretending to be dynamic objects with yet another set of undocumented assumptions of how the world works. The unit tests that have made me happiest has been unit tests on top of a very complicated library that had a very simple api. And on the other hand, the tests that make me most believe that the projects I'm working on are correct have been integration tests incorporating a significant part of the application AND the QA team's test very thorough test plan.
nabbe23 3y ago
Well-written breaking tests represent something changing in a code base. You can be intentional about breaking a test, but then at least you can be very explicit about what you are changing. Have seen all to many times I've broken a unit test in a code base that I did not intend to break, just to have an aha moment that I would have introduced a bug had that test not been present. Unit tests are a trade off between development speed and stability (putting aside other factors, such as integration tests, etc). In large corporate settings, that stability could mean millions of dollars saved per bug. That example you provided is a poor one and not really consistent with your point that unit tests are useless - the point is being made that that specific test of UserResource is useless, which I also agree with. Testing at the Resource level via integration test and Service level via unit test is probably sufficient.
- Joel_Mckay 3y ago
  Especially true if you get emergent side-effects from non-obvious shared state dependencies in large projects. Nightmares... =)
  - nabbe23 3y ago
    Yes sir :) And pragmatically - this always happens at some point. Something something about deadlines and need to get this out yesterday.
    - deleted 3y ago
      [deleted]
Joel_Mckay 3y ago
If maintained right, unit tests at edge conditions can quickly diagnose system and runtime state health. If you work with malicious on incompetent staff at times (over 100 there is always at least 1)... it is the only way to enforce actual accountability after a dozen people touch the same files over years. "The sculpture is already complete within the marble block, before I start my work. It is already there, I just have to chisel away the superfluous material." ( Michelangelo ) Admittedly, in-house automated testing for cosmetic things like GUI or 3D rendering pipelines is still nontrivial. Best of luck =)
Verdex 3y ago
I've been on several projects where we had a significant number of unit tests that would only fail when requirements would change and we had to change the code. "Look, if it only fails with requirement changes, then maybe we're better off not having them." This only made people uncomfortable. They just don't like walking down the mental path that leads them to the conclusion that their high coverage unit tests are not worth the tradeoff. Or even that there is a tradeoff present at all. Meanwhile, PRs constantly ask for more coverage. Not very often, but sometimes someone will mention: "but unit tests are the specification of the code" test ShouldReturnOutput { _mock1.setup( /* complicated setup code returning mock2 */ ); _mock2.setup( /* even more complicated setup code */ ); let output = _obj.Method( 23.5 ); Assert( output == 0.7543213 ); } /* hundreds of lines above this test case */ setup { if ( _boolean ) { _obj = new obj(_mock1); } else { _obj = new obj(new mock()); } } I'm just not sure I can get there.

by bedobi 3y ago

bobthepanda 3y ago

The point of unit tests is not to CYA during refactors, but to confirm that the implementation is consistent between small changes without weird side effects. A coworker once thought unit tests were dumb, and ended up writing code that repeated the call to an application 10x for the same info. This didn’t result in a changed UI because it was a read, but it’s not good to just suddenly 10x your reads for no good reason. TFA also describes discovering weird side effect race conditions as a result of unit tests.

domano 3y ago
Using mockserver etc. you can cover for these things in component-test cases even more easily through your whole application while being more flexible with bigger code changes than unit tests allow.
- bobthepanda 3y ago
  A lot of places on the internet treat component testing and unit testing as synonyms. I’ve never heard of the former and it basically sounds like the unit tests as we write.
bedobi 3y ago
> confirm that the implementation is consistent between small changes without weird side effects not sure what this is referring to, but I'll give an example say you have a requirement that says if you call POST /user with a non-existing user, a user should be created and you should get a 2xx response with some basic details back you could test this by actually hitting the endpoint with randomly generated user data known to not already exist, check that you get the expected 2xx response in the expected format, and then use the user id you got back to call the GET /user/userId endpoint and check that it's the same user that was just created this is a great test! it enforces actual business logic while still allowing you to change literally everything about the implementation - you could change the codebase from Java Spring Boot to Python Flask if you wanted to, you could change the persistence tech from MySQL to MariaDB or Redis etc etc - the test would still pass when the endpoint behaves as expected and fail when it doesn't, and it's a single test that is cheap to write, maintain and run OR you could write dozens of the typical corporate style unit test i'm referring to, where you create instances of each individual layer class, mocks of every class it interacts with, mocked database calls etc etc which 1) literally enforce every single aspect of the implementation, so now you can't change anything without breaking the tests 2) pretend that things work when they actually don't (eg, it could be that the CreateUserDAO actually breaks because someone stuffs up a db call, but guess what, the CreateUserResource and CreateUserService unit tests will still pass, because they just pretend (through mocks) that CreateUserDao.createUser returns a created user
- bobthepanda 3y ago
  I literally gave examples in my example. As a general example, a unit test is great for things like: Your ExampleFactory calls your ExampleSerivce exactly once, not more, not less, so you can check that side effects don’t result in unnecessary extra calls in more load. This is particularly relevant in a language like Java; modern Java style is functional, but the old style relied heavily on side effects, and they’re still possible to write unintentionally.
  - Verdex 3y ago
    I believe the kind of scenario that bedobi is referring to is something like this (using your example): Unit test exists, ExampleFactory only calls ExampleService once. Hmm, it turns out that ExampleUIButton calls ExamplePoxyNavigator more than one time if the text in ExampleUIButton happens to be wider than the default width for the button. What does ExampleProxyNavigator do? Oh, it calls ExampleFactory which calls ExampleService. But only when the whole system is wired up for deployment. The unit tests indicate that the system should be functioning okay, but when you put everything together you find out that the system does not function okay.
  - bedobi 3y ago
    I understand your example and I agree that in that particular case, maybe a unit test to ensure a given service calls a dao once and only once or whatever is justified but I don't think the hypothetical risk of someone needlessly calling the db ten times is a good reason to justify adding that style unit tests to everything by default - if it happens, sure, add one for that particular call and call it a day
- Verdex 3y ago
  To be fair to unit tests, I really like them for making sure that complicated code gets tested thoroughly. However, very often the complicated code isn't isolated to a single unit. It instead lives distributed amongst multiple objects that are reused for several competing aspects and all have their own undocumented assumptions about how the world works. Now maybe this implies that we need a wide scale change in coding methodology such that the complicated parts are all isolated to units. But pending that, I'm not sure that the answer is a bunch of static mocks pretending to be dynamic objects with yet another set of undocumented assumptions of how the world works. The unit tests that have made me happiest has been unit tests on top of a very complicated library that had a very simple api. And on the other hand, the tests that make me most believe that the projects I'm working on are correct have been integration tests incorporating a significant part of the application AND the QA team's test very thorough test plan.

nabbe23 3y ago

Well-written breaking tests represent something changing in a code base. You can be intentional about breaking a test, but then at least you can be very explicit about what you are changing. Have seen all to many times I've broken a unit test in a code base that I did not intend to break, just to have an aha moment that I would have introduced a bug had that test not been present. Unit tests are a trade off between development speed and stability (putting aside other factors, such as integration tests, etc). In large corporate settings, that stability could mean millions of dollars saved per bug. That example you provided is a poor one and not really consistent with your point that unit tests are useless - the point is being made that that specific test of UserResource is useless, which I also agree with. Testing at the Resource level via integration test and Service level via unit test is probably sufficient.

Joel_Mckay 3y ago
Especially true if you get emergent side-effects from non-obvious shared state dependencies in large projects. Nightmares... =)
- nabbe23 3y ago
  Yes sir :) And pragmatically - this always happens at some point. Something something about deadlines and need to get this out yesterday.
  - deleted 3y ago
    [deleted]

Joel_Mckay 3y ago

If maintained right, unit tests at edge conditions can quickly diagnose system and runtime state health. If you work with malicious on incompetent staff at times (over 100 there is always at least 1)... it is the only way to enforce actual accountability after a dozen people touch the same files over years. "The sculpture is already complete within the marble block, before I start my work. It is already there, I just have to chisel away the superfluous material." ( Michelangelo ) Admittedly, in-house automated testing for cosmetic things like GUI or 3D rendering pipelines is still nontrivial. Best of luck =)

Verdex 3y ago

I've been on several projects where we had a significant number of unit tests that would only fail when requirements would change and we had to change the code. "Look, if it only fails with requirement changes, then maybe we're better off not having them." This only made people uncomfortable. They just don't like walking down the mental path that leads them to the conclusion that their high coverage unit tests are not worth the tradeoff. Or even that there is a tradeoff present at all. Meanwhile, PRs constantly ask for more coverage. Not very often, but sometimes someone will mention: "but unit tests are the specification of the code" test ShouldReturnOutput { _mock1.setup( /* complicated setup code returning mock2 */ ); _mock2.setup( /* even more complicated setup code */ ); let output = _obj.Method( 23.5 ); Assert( output == 0.7543213 ); } /* hundreds of lines above this test case */ setup { if ( _boolean ) { _obj = new obj(_mock1); } else { _obj = new obj(new mock()); } } I'm just not sure I can get there.