7 ms·

Testing. Unit testing to me seemed akin to drinking 8 glasses of water every day. A lot of people talk about how important it is for your health, but it reall

Testing. Unit testing to me seemed akin to drinking 8 glasses of water every day. A lot of people talk about how important it is for your health, but it really tends to get in the way, and it doesn't seem to really be necessary. Too frequently, code would change and mocks would need to change with it, removing a good chunk of the benefit of having the code under test. Then I started writing integration testing while working on converting a bunch of code recently, and it has been eye-opening. Instead of testing individual models and functions, I was testing the API response and DB changes, and who really cares what the code in the middle does and how it interfaces with other internal code? So long as the API and DB are in the expected state, you can go muck about with the guts of your code all you want, while having the assurance that callers of your code are getting exactly out of it what you promise. Unit test suites would break all the time for silly reasons, like someone optimizing a function would mean a spy wouldn't get called with the same intermediary data, and you'd have to stop and go fix the test code that was now broken, even though the actual code worked as intended. Integration tests (mainly) only break when the code itself is broken and incorrect results are getting spit out. This has prevented all kinds of issues from actually reaching customers during the conversion process, and isn't nearly so brittle as our unit tests were.

arduinomancer 8y ago
Just adding to this, for me I think the benefits of testing became really clear in situations where you end up fixing the same bug twice. Knowing about that bug is really valuable knowledge and not adding a test for it is basically like throwing away that knowledge instead of sharing it for future people working on the project.
Bahamut 8y ago
My experience has been very much the opposite - integration tests are often very brittle, and finding the root cause is often almost a fools errand since stack traces are often no good for figuring out what went wrong, at least for web UI. While yes, unit tests do have a maintenance burden, they are often reproducible, less flaky, give you targeted debug information, and run extremely fast. There are heavy costs to integration and e2e testing that often gets dismissed by developers who often have not experienced the fast feedback loop a good fat unit suite gives you.
- yashap 8y ago
  Although I agree with you in general, integration tests that are really focused on the contract between two services can be pretty easy to debug, ESPECIALLY if you use request ids that are passed between services, and included in every log. It’s often a matter of simply searching for the request id in your logs and quickly seeing what happened. E2E UI tests are definitely hard to debug though, in most cases. They still have value, but I’m a fan of the “testing pyramid” here. A small number of E2E UI tests, a decently large number of integration tests that each try to test a single contract between 2 services, and a tonne of unit tests to cover sad paths, detailed behaviour, etc.
  - Bahamut 8y ago
    Totally agreed - the problem is when integration tests are used in lieu of what unit tests are for, which has been an increasing trend in the frontend web world I’ve seen, in large part because of UI frameworks not providing the tools to do testing well.
    - yashap 8y ago
      Definitely. E2E UI tests are almost always brittle, flakey and hard to debug, you want a very small number of them just to test really core workflows. Drives me crazy too when I see frontend devs using them like UI unit tests, while writing no actual unit tests.
- danmaz74 8y ago
  Ideally, you should have both unit and integration tests. See "Test Pyramid".
discreteevent 8y ago
You need to write a book and then get on the conference circuit with this message. You would be rowing against the tide but there is always hope!
- bradleyjg 8y ago
  https://kentcdodds.com/blog/write-tests https://kentcdodds.com/blog/write-tests Write tests. Not too many. Mostly integration.
buzzy_hacker 8y ago
> Unit test suites would break all the time for silly reasons, like someone optimizing a function would mean a spy wouldn't get called with the same intermediary data, and you'd have to stop and go fix the test code that was now broken, even though the actual code worked as intended. Can you or others speak more about this? I was taught that verifying function calls for spies/mocks was good practice. But, I encountered this problem just the other day when I refactored some Java code for a personal project. Everything still worked perfectly, but, exactly as you said, the intermediate function calls changed so the tests would fail due to spies/mocks calling different "unexpected" functions. I'm an intermediate programmer so can someone with more experience fill me in with what's best practice here and why? Do I update the test code to reflect the new intermediate function calls? But this whole approach now seems silly since a refactoring that doesn't affect the ultimate behavior of the function that is under test will break the test and that seems wrong. So do I instead not verify function calls when using spies/mocks? In that case, what is the use case for verifying spies/mocks?
- wakamoleguy 8y ago
  If it is important to the operation of the function, from an outsider perspective, the intermediate call should be tested. Oftentimes this isn't the case, though. Most often, I see intermediate calls spied/mocked when they have side effects to be avoided. This is actually a sign of tight coupling between modules, and patterns like dependency injection can help make it easier to test. The trick for me is focusing on what the unit does from a consumers perspective. Avoid testing implementation details (unless they are important side effects), and test the behavior that does not change. If you do that, then refactoring becomes easier, because tests will only break when the contract of the unit changes.
- ht85 8y ago
  What you want to spy on are side-effects that are part of the function's contract. If you have a function that fetches data, you shouldn't test that its hitting the data layer, only that the correct data is returned. This way when you improve the function to not hit the data layer at all under some conditions, your tests will keep passing. On the other hand, if that function is supposed to log metrics or details about its execution, you should test that ,as it is part of its contract and can't be inferred from the return value.
wakamoleguy 8y ago
The integration tests that you are describing behave much like I would expect unit tests to behave. Give an input, assert the results (return value or state change). The example of optimizations breaking the unit test could be either a poor test (If the function doesn't need to call that intermediate with that data, why is it being tested?). Oftentimes the spy is added to the test because there is unnecessary tight coupling between two units. This isn't to say that integration tests aren't great, too. They all tell different stories. I like unit tests, as they force me to think about modules as units. If a unit is hard to test or brittle, it may need more attention to its design.
- stevesmithwick 8y ago
  A good use of mocks/spies is if you need to test a side-effect. For instance: did your code call that service with the right parameters?
jondubois 8y ago
I have the same experience. Integration tests are the best. They test only what really matters and allow you to keep flexibility over implementation details. When your TDD approach revolves around integration tests, you have complete freedom to add, remove and shift around internal components. Having the flexibility to keep moving around the guts of a system to bring it closer to its intended behavior is what software engineering is all about. This is also how evolution works; the guts and organs of every living creature were never independently tested by nature. Nature only cares about very high-level functionality; can this specimen survive long enough to reproduce? Yes. Ok, then it works! It doesn't matter that this creature has ended up with intestines which are 1.5 meters long; that's an implementation detail. The specimen is good because it works within all imposed external/environmental constraints. That's why there are so many different species in the world; when a system has well defined external requirements, it's possible to find many solutions (of varying complexity) which can perfectly meet those requirements.
- overkalix 8y ago
  Fyi, I'm definitely stealing your analogy.
- deleted 8y ago
  [deleted]
- shittydeveloper 8y ago
  I'll would like to watch you running around and trying out everything when integration tests fails and you don't know which part of the code base caused the failure, while I just run all the specs and figure out the exact unit of code that is causing the problem.
  - jondubois 8y ago
    That would not happen because each of my integration test cases are properly isolated form each other. If any specific test case starts failing, I just disable all other test cases and start debugging the affected code path; it usually only takes me a few minutes to identify even the most complex problems. Also, because I use a TDD approach, I usually have a pretty good idea about what changes I might have made to the code which could have introduced the issue. Unit tests on the other hand are useless at identifying complex issues like race conditions; they're only good at detecting issues that are already obvious to a skilled developer. * I restart the system from scratch between each test case but if that's too time-consuming (I.e. more than a few milliseconds), I just clear the entire system state between each test case. If the system is truly massive and clearing the mock state between each test case takes too long, then I break it up into smaller microservices; each with their own integration tests.
JesseAldridge 8y ago
I think most would agree that integration tests are better. The problem is they tend to be slower. Having to initialize the system appropriately for every test (e.g. writing to the database) tends to limit the number of tests you can have. Unit tests scale a lot better. That's why most generally use a pyramid structure: lots of unit tests, a moderate amount integration tests, and a few end-to-end tests.
- humanrebar 8y ago
  Unit tests also tend to have fewer dependencies and are therefore more portable and robust. If your integration tests can be reasonably be set up with a couple containers, great, but not every system is that flexible. And not every data store is that simple to provision.
- cellularmitosis 8y ago
  An approach of “almost” integration tests can be much faster. I.e. rather than spinning up a full-blown web server or database, use fake objects for those. Uncle bob describes refactoring the architecture of a project to make integration tests faster by decoupling the web server. Mentally, something like clojure’s ring framework make this easier to grasp: the abstraction it provides is dictionary-in, dictionary-out. Once you have something like this, there’s no need to spin up a web server to do integration testing: you just shove a bunch of dictionaries and and make sure the output dictionaries are what you expected. A good approach is to use each technique where you get the most bang-for-buck: use unit tests only for pure functions, and decouple your system in a way that integration tests can be reduced to simple data-in data-out (which also makes them very fast)
  - cellularmitosis 8y ago
    Ah I forgot the link https://youtu.be/WpkDN78P884 https://youtu.be/WpkDN78P884
- MarkMc 8y ago
  With an in-memory database like Derby I find my test cases run fast enough
u801e 8y ago
> Too frequently, code would change and mocks would need to change with it, removing a good chunk of the benefit of having the code under test. A lot of people overuse mocks when testing. In fact, using mocks enforces coupling between different methods because a lot of people use them to assert that a method with a specific name was called with specific parameters or they create one that asserts a method with a certain name returns a certain value. So when one wants to refactor, they not only have change the code; they need to update all the mocks that reference it as well. I've found that a better way to structure code is to take the result of an external dependency and pass it in as a parameter to a method that will process it. Then when I unit test that method, I just pass in what I expect from that dependency and assert on the return value of that method. I don't try to unit test the outer method that calls the dependency by creating a mock call for it. > Then I started writing integration testing while working on converting a bunch of code recently, and it has been eye-opening. Instead of testing individual models and functions, I was testing the API response and DB changes, and who really cares what the code in the middle does and how it interfaces with other internal code? It makes it easier to isolate the cause of the error rather than having to search to the entire call chain to find it (especially if it's a logic error that doesn't result in an exception). Plus, integration test suites take a lot longer to run and can have timing issues due to caching or other reasons which can result in sporadic failures.
alexval 8y ago
My first job I wrote c++ for a win32 desktop app. I hated unit testing. My workflow was write the code, compile it, trigger the scenario, step through the code, write some unit tests after I knew it worked. It was the expectation on my team to write UTs so I did it. Fast forward to a different team I learned from a co-worker how UTs can help you influence your design. If you find yourself doing a ton of frustrating work to set up your UT scenarios there's probably a way of fixing the design to be less coupled that would allow you to test the code better. Now I think of UTs more as a way to help me understand how the design of my code is working and I get some extra validation out of it as well.
- pjmlp 8y ago
  My trick question to TDD advocates is how to develop native apps following those principles, after all no GUI change is allowed without a test.
  - AnimalMuppet 8y ago
    In my book, the UI isn't part of unit testing - at least, the views aren't. The models and controllers (to use the MVC paradigm) might be, though.
  - magduf 8y ago
    GUIs don't lend themselves to unit testing at all, because the requirements aren't mathematical, but are instead based on human factors. For GUIs, the proper approach is to unit-test the functionality underneath, not the GUI itself.
    - pjmlp 8y ago
      Which leads to convoluted architectures just to be able to follow "only write code which there is a failing test for".
      - romwell 8y ago
        How are these convoluted? After all, any program out there does something. If you know what it's doing, you know what it should do, and what it shouldn't. That means you can test it.
danmaz74 8y ago
I almost only use mocks for external calls (usually external APIs). Very rarely, I use them for some internal code that is unusually time consuming or difficult to set up. A few months ago, I did a contracting job and the team I worked with used the "mock everything" approach, without even one integration test, which to me seemed crazy (especially for a component which was calle "integration layer"). I tried hard to find the advantages in this approach, studying what the rationale and the best practices were, and questioned my previous assumptions. In the end, I had to confirm those assumptions: even if there were hundreds of tests and they all passed, many logical errors weren't caught. Even worse, they gave a false sense of confidence to the team, and made refactoring super-slow. But the team leader was super-convinced it was the best idea since sliced bread.
- taurath 8y ago
  People I’ve worked with have had that idea, and I’ve found they’re most frequently from the games industry or from contract shops. In both of those, the maintenance of the code over time is far less important than that it works on the ship date. Additionally, people with large state machines with too-complex sets of possible states (games, big frontends without a top level state management system) tend to only unit test because it’s frequently too much of a pain to set up an integration runtime environment. Places with lots and lots of manual QA testers.
  - danmaz74 8y ago
    Yes, it's for sure a quicker way to have 100% test coverage...
chasingthewind 8y ago
I will probably be the lone voice here defending mock testing but probably not for the reasons you'd expect. Does mock testing end up being brittle? Yes. Do you have to refactor the tests immediately after making small changes? Yes. Is there a cost to this? Yes. Mock based testing however is the only thing I've ever encountered that forces me to think very, very clearly about what my code is doing. It makes me inspect the code and think about what dependencies are being called, how they're being called, and why they're being called. I have found that this process is extremely valuable for creating code that is more elegant and more correct. I value mock tests not for the tests that I end up with at the end, but for the better production code that I wrote because of them.
- mdgrech23 8y ago
  What's stopping you from thinking clearly about the code while/after you write it? As I'm writing this sentence I'm thinking clearly about it and I'll probably reread it after I hit save. There's no reason you can't do the same w/ code.
  - nirse 8y ago
    It's not that it can't be done, it's just that sometimes it is nice to have a tool that helps you think about the code. Unit tested code isn't better in any way, it's just a style of writing code that can produce good quality code, but not the only coding style that can do that. But I'd like to add that there have been quite a few refactors that I have done that would have been way harder if it hadn't been for the unit tests. It might not work for everyone, but I'd recommend you give it a serious try, it might be useful.
    - AnimalMuppet 8y ago
      The tests don't just help you think about the code, they force you to do so. For those who are tempted to think "I can just do this quick change this one time; nothing will go wrong", tests force you to actually think. Or they don't. If the tests just continue to work, then you ask yourself whether the change should have broken them. If not, you go on your way with a fair amount of confidence. But if the change should have broken the tests, then you have to look at why it didn't...
yougotrioted 8y ago
I completely agree. If you have limited time to write tests, then I've found that integration tests provide the most value by far. If you're happy with your suite of integration tests, then it's also great to have a set of unit tests.
h0l0cube 8y ago
You might like this article: https://rbcs-us.com/documents/Why-Most-Unit-Testing-is-Waste.pdf https://rbcs-us.com/documents/Why-Most-Unit-Testing-is-Waste...
jorblumesea 8y ago
I would disagree. Unit tests make sure your code is clean and composable. It's hard to write unit tests if you have hundred line methods etc. It usually forces you to write smaller chunks of code. Integration tests make sure it works in test/prod. You can have a complete spaghetti mess and still pass your integration tests. You can also have good unit test coverage but break prod. They serve two different purposes.
virtualmic 8y ago
Both unit and integration tests have their place. This talk was excellent in explaining it to me (with functional cores, imperative shells). Unit test functional cores, use integration tests for imperative shells: https://www.destroyallsoftware.com/talks/boundaries https://www.destroyallsoftware.com/talks/boundaries I am fan of Gary Bernhardt ever since viewing this talk.
jaabe 8y ago
We buy software from a lot of different companies, sometimes we own the codebase but hire software companies to build, expand and maintain it, we also benchmark performance. We have a lot of data that explicitly shows that automated unit-testing doesn’t work. One good example is one of our ESDH systems, which changed supplier in a bidding war, partly because we wanted higher code-quality. It’s a two million line project, we paid for 5000 hours worth of refactoring and let the new supplier spend two years getting familiar with it and setting up their advanced testing platforms. So far we’ve seen some nice performances increases thanks to the refactoring. It has more problems than it did before though, even though everything is tested by unit tests now and it wasn’t before. We have a lot of stories like this by the way. I don’t think we have a single story where unit-testing actually made the product better, but we do have tested systems where we couldn’t say because they always had unit-testing. Ironically we still do TDD ourselves on larger projects. Not because we can prove it works, but because everyone expects it.
- geezerjay 8y ago
  > We have a lot of data that explicitly shows that automated unit-testing doesn’t work. At best you have data that shows that a poor unit testing implementation failed to deliver. A bad experience doesn't prove a whole tech strategy doesn't work when the whole world shows otherwise.
kayoone 8y ago
you can also unit test in a more blackbox way where you test your interal APIs instead of the implementation. It makes sense to have both in many cases.
a_imho 8y ago
There was a comment on HN which I keep thinking about. What is the value of expensive testing when you are promptly alerted of critical problems and can deploy in under a minute? Obviously not applicable for every scenario, but at this point I think for most non critical systems a good rollback/plan B is at least as important as testing.
- geezerjay 8y ago
  > What is the value of expensive testing when you are promptly alerted of critical problems and can deploy in under a minute? Because your users should not be your testers and you must catch bugs before deploying them in production. You may be able to deploy in 3 minutes but it takes way more to debug and fix your bugs. Testing helps the project to catch bugs prior to deploying them and also provides the infrastructure to avoid regressions.
jMyles 8y ago
I had this experience as well, but then I began to realize that, as the codebase got bigger (let's say, for your metaphor, I was exercising more) that I really actually did want unit tests (that I needed more water). As the codebase gets larger and more complex (and interesting!), I want unit tests to fail because of small changes. That's actually useful feedback, whereas the simple, brutal failure of an integration test is just not granular enough to quickly help me understand the details of the change.
- jwdunne 8y ago
  This has been my experience too. On a project I'm working on now, I went majority integration tests. I have about 200 of them and they now take 2 minutes per run. VirtualBox and Jest doesn't help. To stay sane, I've had to run only relevant tests. They're great for a final sweep but they do slow down the feedback loop quite a lot. For future tests and projects, unit tests do have a place and I'll make better use of them. A good learning experience, none the less.
entity345 8y ago
As is the case for all software, unit tests should be well designed. If a unit test fails when the code under test hasn't functionally changed then it means that your unit tests are flaky.
imhoguy 8y ago
It is all about balance. Use system/integration/functional/unit tests wisely. For precise stateless stuff, like making sure your custom input format parser/regexp covers all edge cases, I prefer unit tests - no need to init/rollback database state.

arduinomancer 8y ago

Just adding to this, for me I think the benefits of testing became really clear in situations where you end up fixing the same bug twice. Knowing about that bug is really valuable knowledge and not adding a test for it is basically like throwing away that knowledge instead of sharing it for future people working on the project.

Bahamut 8y ago

My experience has been very much the opposite - integration tests are often very brittle, and finding the root cause is often almost a fools errand since stack traces are often no good for figuring out what went wrong, at least for web UI. While yes, unit tests do have a maintenance burden, they are often reproducible, less flaky, give you targeted debug information, and run extremely fast. There are heavy costs to integration and e2e testing that often gets dismissed by developers who often have not experienced the fast feedback loop a good fat unit suite gives you.

yashap 8y ago
Although I agree with you in general, integration tests that are really focused on the contract between two services can be pretty easy to debug, ESPECIALLY if you use request ids that are passed between services, and included in every log. It’s often a matter of simply searching for the request id in your logs and quickly seeing what happened. E2E UI tests are definitely hard to debug though, in most cases. They still have value, but I’m a fan of the “testing pyramid” here. A small number of E2E UI tests, a decently large number of integration tests that each try to test a single contract between 2 services, and a tonne of unit tests to cover sad paths, detailed behaviour, etc.
- Bahamut 8y ago
  Totally agreed - the problem is when integration tests are used in lieu of what unit tests are for, which has been an increasing trend in the frontend web world I’ve seen, in large part because of UI frameworks not providing the tools to do testing well.
  - yashap 8y ago
    Definitely. E2E UI tests are almost always brittle, flakey and hard to debug, you want a very small number of them just to test really core workflows. Drives me crazy too when I see frontend devs using them like UI unit tests, while writing no actual unit tests.
danmaz74 8y ago
Ideally, you should have both unit and integration tests. See "Test Pyramid".

discreteevent 8y ago

You need to write a book and then get on the conference circuit with this message. You would be rowing against the tide but there is always hope!

bradleyjg 8y ago
https://kentcdodds.com/blog/write-tests https://kentcdodds.com/blog/write-tests Write tests. Not too many. Mostly integration.

buzzy_hacker 8y ago

> Unit test suites would break all the time for silly reasons, like someone optimizing a function would mean a spy wouldn't get called with the same intermediary data, and you'd have to stop and go fix the test code that was now broken, even though the actual code worked as intended. Can you or others speak more about this? I was taught that verifying function calls for spies/mocks was good practice. But, I encountered this problem just the other day when I refactored some Java code for a personal project. Everything still worked perfectly, but, exactly as you said, the intermediate function calls changed so the tests would fail due to spies/mocks calling different "unexpected" functions. I'm an intermediate programmer so can someone with more experience fill me in with what's best practice here and why? Do I update the test code to reflect the new intermediate function calls? But this whole approach now seems silly since a refactoring that doesn't affect the ultimate behavior of the function that is under test will break the test and that seems wrong. So do I instead not verify function calls when using spies/mocks? In that case, what is the use case for verifying spies/mocks?

wakamoleguy 8y ago
If it is important to the operation of the function, from an outsider perspective, the intermediate call should be tested. Oftentimes this isn't the case, though. Most often, I see intermediate calls spied/mocked when they have side effects to be avoided. This is actually a sign of tight coupling between modules, and patterns like dependency injection can help make it easier to test. The trick for me is focusing on what the unit does from a consumers perspective. Avoid testing implementation details (unless they are important side effects), and test the behavior that does not change. If you do that, then refactoring becomes easier, because tests will only break when the contract of the unit changes.
ht85 8y ago
What you want to spy on are side-effects that are part of the function's contract. If you have a function that fetches data, you shouldn't test that its hitting the data layer, only that the correct data is returned. This way when you improve the function to not hit the data layer at all under some conditions, your tests will keep passing. On the other hand, if that function is supposed to log metrics or details about its execution, you should test that ,as it is part of its contract and can't be inferred from the return value.

wakamoleguy 8y ago

The integration tests that you are describing behave much like I would expect unit tests to behave. Give an input, assert the results (return value or state change). The example of optimizations breaking the unit test could be either a poor test (If the function doesn't need to call that intermediate with that data, why is it being tested?). Oftentimes the spy is added to the test because there is unnecessary tight coupling between two units. This isn't to say that integration tests aren't great, too. They all tell different stories. I like unit tests, as they force me to think about modules as units. If a unit is hard to test or brittle, it may need more attention to its design.

stevesmithwick 8y ago
A good use of mocks/spies is if you need to test a side-effect. For instance: did your code call that service with the right parameters?

jondubois 8y ago

I have the same experience. Integration tests are the best. They test only what really matters and allow you to keep flexibility over implementation details. When your TDD approach revolves around integration tests, you have complete freedom to add, remove and shift around internal components. Having the flexibility to keep moving around the guts of a system to bring it closer to its intended behavior is what software engineering is all about. This is also how evolution works; the guts and organs of every living creature were never independently tested by nature. Nature only cares about very high-level functionality; can this specimen survive long enough to reproduce? Yes. Ok, then it works! It doesn't matter that this creature has ended up with intestines which are 1.5 meters long; that's an implementation detail. The specimen is good because it works within all imposed external/environmental constraints. That's why there are so many different species in the world; when a system has well defined external requirements, it's possible to find many solutions (of varying complexity) which can perfectly meet those requirements.

overkalix 8y ago
Fyi, I'm definitely stealing your analogy.
deleted 8y ago
[deleted]
shittydeveloper 8y ago
I'll would like to watch you running around and trying out everything when integration tests fails and you don't know which part of the code base caused the failure, while I just run all the specs and figure out the exact unit of code that is causing the problem.
- jondubois 8y ago
  That would not happen because each of my integration test cases are properly isolated form each other. If any specific test case starts failing, I just disable all other test cases and start debugging the affected code path; it usually only takes me a few minutes to identify even the most complex problems. Also, because I use a TDD approach, I usually have a pretty good idea about what changes I might have made to the code which could have introduced the issue. Unit tests on the other hand are useless at identifying complex issues like race conditions; they're only good at detecting issues that are already obvious to a skilled developer. * I restart the system from scratch between each test case but if that's too time-consuming (I.e. more than a few milliseconds), I just clear the entire system state between each test case. If the system is truly massive and clearing the mock state between each test case takes too long, then I break it up into smaller microservices; each with their own integration tests.

JesseAldridge 8y ago

I think most would agree that integration tests are better. The problem is they tend to be slower. Having to initialize the system appropriately for every test (e.g. writing to the database) tends to limit the number of tests you can have. Unit tests scale a lot better. That's why most generally use a pyramid structure: lots of unit tests, a moderate amount integration tests, and a few end-to-end tests.

humanrebar 8y ago
Unit tests also tend to have fewer dependencies and are therefore more portable and robust. If your integration tests can be reasonably be set up with a couple containers, great, but not every system is that flexible. And not every data store is that simple to provision.
cellularmitosis 8y ago
An approach of “almost” integration tests can be much faster. I.e. rather than spinning up a full-blown web server or database, use fake objects for those. Uncle bob describes refactoring the architecture of a project to make integration tests faster by decoupling the web server. Mentally, something like clojure’s ring framework make this easier to grasp: the abstraction it provides is dictionary-in, dictionary-out. Once you have something like this, there’s no need to spin up a web server to do integration testing: you just shove a bunch of dictionaries and and make sure the output dictionaries are what you expected. A good approach is to use each technique where you get the most bang-for-buck: use unit tests only for pure functions, and decouple your system in a way that integration tests can be reduced to simple data-in data-out (which also makes them very fast)
- cellularmitosis 8y ago
  Ah I forgot the link https://youtu.be/WpkDN78P884 https://youtu.be/WpkDN78P884
MarkMc 8y ago
With an in-memory database like Derby I find my test cases run fast enough

u801e 8y ago

> Too frequently, code would change and mocks would need to change with it, removing a good chunk of the benefit of having the code under test. A lot of people overuse mocks when testing. In fact, using mocks enforces coupling between different methods because a lot of people use them to assert that a method with a specific name was called with specific parameters or they create one that asserts a method with a certain name returns a certain value. So when one wants to refactor, they not only have change the code; they need to update all the mocks that reference it as well. I've found that a better way to structure code is to take the result of an external dependency and pass it in as a parameter to a method that will process it. Then when I unit test that method, I just pass in what I expect from that dependency and assert on the return value of that method. I don't try to unit test the outer method that calls the dependency by creating a mock call for it. > Then I started writing integration testing while working on converting a bunch of code recently, and it has been eye-opening. Instead of testing individual models and functions, I was testing the API response and DB changes, and who really cares what the code in the middle does and how it interfaces with other internal code? It makes it easier to isolate the cause of the error rather than having to search to the entire call chain to find it (especially if it's a logic error that doesn't result in an exception). Plus, integration test suites take a lot longer to run and can have timing issues due to caching or other reasons which can result in sporadic failures.

alexval 8y ago

My first job I wrote c++ for a win32 desktop app. I hated unit testing. My workflow was write the code, compile it, trigger the scenario, step through the code, write some unit tests after I knew it worked. It was the expectation on my team to write UTs so I did it. Fast forward to a different team I learned from a co-worker how UTs can help you influence your design. If you find yourself doing a ton of frustrating work to set up your UT scenarios there's probably a way of fixing the design to be less coupled that would allow you to test the code better. Now I think of UTs more as a way to help me understand how the design of my code is working and I get some extra validation out of it as well.

pjmlp 8y ago
My trick question to TDD advocates is how to develop native apps following those principles, after all no GUI change is allowed without a test.
- AnimalMuppet 8y ago
  In my book, the UI isn't part of unit testing - at least, the views aren't. The models and controllers (to use the MVC paradigm) might be, though.
- magduf 8y ago
  GUIs don't lend themselves to unit testing at all, because the requirements aren't mathematical, but are instead based on human factors. For GUIs, the proper approach is to unit-test the functionality underneath, not the GUI itself.
  - pjmlp 8y ago
    Which leads to convoluted architectures just to be able to follow "only write code which there is a failing test for".
    - romwell 8y ago
      How are these convoluted? After all, any program out there does something. If you know what it's doing, you know what it should do, and what it shouldn't. That means you can test it.

danmaz74 8y ago

I almost only use mocks for external calls (usually external APIs). Very rarely, I use them for some internal code that is unusually time consuming or difficult to set up. A few months ago, I did a contracting job and the team I worked with used the "mock everything" approach, without even one integration test, which to me seemed crazy (especially for a component which was calle "integration layer"). I tried hard to find the advantages in this approach, studying what the rationale and the best practices were, and questioned my previous assumptions. In the end, I had to confirm those assumptions: even if there were hundreds of tests and they all passed, many logical errors weren't caught. Even worse, they gave a false sense of confidence to the team, and made refactoring super-slow. But the team leader was super-convinced it was the best idea since sliced bread.

taurath 8y ago
People I’ve worked with have had that idea, and I’ve found they’re most frequently from the games industry or from contract shops. In both of those, the maintenance of the code over time is far less important than that it works on the ship date. Additionally, people with large state machines with too-complex sets of possible states (games, big frontends without a top level state management system) tend to only unit test because it’s frequently too much of a pain to set up an integration runtime environment. Places with lots and lots of manual QA testers.
- danmaz74 8y ago
  Yes, it's for sure a quicker way to have 100% test coverage...

chasingthewind 8y ago

I will probably be the lone voice here defending mock testing but probably not for the reasons you'd expect. Does mock testing end up being brittle? Yes. Do you have to refactor the tests immediately after making small changes? Yes. Is there a cost to this? Yes. Mock based testing however is the only thing I've ever encountered that forces me to think very, very clearly about what my code is doing. It makes me inspect the code and think about what dependencies are being called, how they're being called, and why they're being called. I have found that this process is extremely valuable for creating code that is more elegant and more correct. I value mock tests not for the tests that I end up with at the end, but for the better production code that I wrote because of them.

mdgrech23 8y ago
What's stopping you from thinking clearly about the code while/after you write it? As I'm writing this sentence I'm thinking clearly about it and I'll probably reread it after I hit save. There's no reason you can't do the same w/ code.
- nirse 8y ago
  It's not that it can't be done, it's just that sometimes it is nice to have a tool that helps you think about the code. Unit tested code isn't better in any way, it's just a style of writing code that can produce good quality code, but not the only coding style that can do that. But I'd like to add that there have been quite a few refactors that I have done that would have been way harder if it hadn't been for the unit tests. It might not work for everyone, but I'd recommend you give it a serious try, it might be useful.
  - AnimalMuppet 8y ago
    The tests don't just help you think about the code, they force you to do so. For those who are tempted to think "I can just do this quick change this one time; nothing will go wrong", tests force you to actually think. Or they don't. If the tests just continue to work, then you ask yourself whether the change should have broken them. If not, you go on your way with a fair amount of confidence. But if the change should have broken the tests, then you have to look at why it didn't...

yougotrioted 8y ago

I completely agree. If you have limited time to write tests, then I've found that integration tests provide the most value by far. If you're happy with your suite of integration tests, then it's also great to have a set of unit tests.

h0l0cube 8y ago

You might like this article: https://rbcs-us.com/documents/Why-Most-Unit-Testing-is-Waste.pdf https://rbcs-us.com/documents/Why-Most-Unit-Testing-is-Waste...

jorblumesea 8y ago

I would disagree. Unit tests make sure your code is clean and composable. It's hard to write unit tests if you have hundred line methods etc. It usually forces you to write smaller chunks of code. Integration tests make sure it works in test/prod. You can have a complete spaghetti mess and still pass your integration tests. You can also have good unit test coverage but break prod. They serve two different purposes.

virtualmic 8y ago

Both unit and integration tests have their place. This talk was excellent in explaining it to me (with functional cores, imperative shells). Unit test functional cores, use integration tests for imperative shells: https://www.destroyallsoftware.com/talks/boundaries https://www.destroyallsoftware.com/talks/boundaries I am fan of Gary Bernhardt ever since viewing this talk.

jaabe 8y ago

We buy software from a lot of different companies, sometimes we own the codebase but hire software companies to build, expand and maintain it, we also benchmark performance. We have a lot of data that explicitly shows that automated unit-testing doesn’t work. One good example is one of our ESDH systems, which changed supplier in a bidding war, partly because we wanted higher code-quality. It’s a two million line project, we paid for 5000 hours worth of refactoring and let the new supplier spend two years getting familiar with it and setting up their advanced testing platforms. So far we’ve seen some nice performances increases thanks to the refactoring. It has more problems than it did before though, even though everything is tested by unit tests now and it wasn’t before. We have a lot of stories like this by the way. I don’t think we have a single story where unit-testing actually made the product better, but we do have tested systems where we couldn’t say because they always had unit-testing. Ironically we still do TDD ourselves on larger projects. Not because we can prove it works, but because everyone expects it.

geezerjay 8y ago
> We have a lot of data that explicitly shows that automated unit-testing doesn’t work. At best you have data that shows that a poor unit testing implementation failed to deliver. A bad experience doesn't prove a whole tech strategy doesn't work when the whole world shows otherwise.

kayoone 8y ago

you can also unit test in a more blackbox way where you test your interal APIs instead of the implementation. It makes sense to have both in many cases.

a_imho 8y ago

There was a comment on HN which I keep thinking about. What is the value of expensive testing when you are promptly alerted of critical problems and can deploy in under a minute? Obviously not applicable for every scenario, but at this point I think for most non critical systems a good rollback/plan B is at least as important as testing.

geezerjay 8y ago
> What is the value of expensive testing when you are promptly alerted of critical problems and can deploy in under a minute? Because your users should not be your testers and you must catch bugs before deploying them in production. You may be able to deploy in 3 minutes but it takes way more to debug and fix your bugs. Testing helps the project to catch bugs prior to deploying them and also provides the infrastructure to avoid regressions.

jMyles 8y ago

I had this experience as well, but then I began to realize that, as the codebase got bigger (let's say, for your metaphor, I was exercising more) that I really actually did want unit tests (that I needed more water). As the codebase gets larger and more complex (and interesting!), I want unit tests to fail because of small changes. That's actually useful feedback, whereas the simple, brutal failure of an integration test is just not granular enough to quickly help me understand the details of the change.

jwdunne 8y ago
This has been my experience too. On a project I'm working on now, I went majority integration tests. I have about 200 of them and they now take 2 minutes per run. VirtualBox and Jest doesn't help. To stay sane, I've had to run only relevant tests. They're great for a final sweep but they do slow down the feedback loop quite a lot. For future tests and projects, unit tests do have a place and I'll make better use of them. A good learning experience, none the less.

entity345 8y ago

As is the case for all software, unit tests should be well designed. If a unit test fails when the code under test hasn't functionally changed then it means that your unit tests are flaky.

imhoguy 8y ago

It is all about balance. Use system/integration/functional/unit tests wisely. For precise stateless stuff, like making sure your custom input format parser/regexp covers all edge cases, I prefer unit tests - no need to init/rollback database state.