4 ms·

My "unit tests" do hit the database and file system, and I have found and fixed many many problems during testing by doing so. I have found many other problems

My "unit tests" do hit the database and file system, and I have found and fixed many many problems during testing by doing so. I have found many other problems with those calls in production when I didn't do so. Yes, they make testing a lot slower. Our main app takes around 40 minutes to build which isn't good. I'd like it to be faster. But writing a bunch of separate integration tests to cover those functions would be a steep price. I can understand reasonable people choosing either approach.

berkes 3y ago
> My "unit tests" do hit the database and file system, and I have found and fixed many many problems during testing by doing so. I have found many other problems with those calls in production when I didn't do so. No-one said that integration tests can't also be very valuable. From the little context I get that you write integration tests, and that is fine. They are useful, valuable! But they are not unit-tests. edit: on re-reading, I get the feeling that for you "integration tests" are a synonym for "end to end tests". But -at least in most literature- end-to-end tests are a kind of integration-test. But not all integration tests are end-to-end tests. In my software, I'll often have integration tests that swap out some adapter (e.g. the postgres-users-repository, for the memory-users-repository, or fake-users-repository. Or the test-payment for the stripe-payment) but that still test a lot of stuff stacked on top of each-other. Integration tests, just not integration tests that test the entire integration.
- lokar 3y ago
  And integration tests can also be fast
  - alemanek 3y ago
    Test containers really help with this. Should still have the big system tests that run overnight but a set of integration tests using Test Containers to stand in for the infrastructure dependencies is awesome. My team has a ton of those and they run inside a reasonable time frame (5min or so) but we still allow for excluding those from test runs so you can run just the unit tests.
    - darkteflon 3y ago
      I hadn’t heard of Test Containers[1], but it looks really useful - thanks for the rec. [1] https://testcontainers.com/ https://testcontainers.com/
  - berkes 3y ago
    Indeed! It's one of the reasons I like the adapter pattern (aka hexagonal architecture) so much. Data flowing through some 100 classes and 300 conditionals then into a `memory-payments` and back takes mere milliseconds. "Memory payments" is then some silly wrapper around a hashmap with the same API as the full-blown production payments-adapter that calls stripe over HTTP. Or the same api as the "production adapter" that wraps some RDBMS running on the other end of the data-center.
- MoreQARespect 3y ago
  >No-one said that integration tests can't also be very valuable. Integration tests are a better kind of default test because they bring value under pretty much all circumstances. Nobody said that unit tests cant also be valuable and under just the right circumstances i.e. - complex stateless code behind a stable API. Unit tests shine in that environment - theyre not impeded by their crippling lack of realism because that stable abstraction walls off the rest of reality. And theyre very fast. Most code isnt parsers, calculation engines, complex string manipulation, etc. - but when it is unit tests really do kick ass. They just suck so badly at testing code that doesnt fit that mold. Which, to be fair, is most code. I dont write a lot of parsers at work. My job involes moving data into databases, calling APIs, linking up message queues, etc.
  - berkes 3y ago
    > Integration tests are a better kind of default test because they bring value under pretty much all circumstances. I respectfully disagree. Not with the last part, that is true: they do bring value under pretty much all circumstances. But the first. Because integration tests come with (extremely) high costs. They are expensive to run. They are much harder (costlier) to write. They are even harder (costlier) to maintain. The common pushback against tests -but they slow down our team a lot- applies to integration tests much more than to unit tests - factors more. And so on. As with everything software-engineering, choosing what tests to write is a tradeoff. And taking all into consideration, e2e or integration tests are often not worth their investment¹. The testing pyramid fixes this, because testing always (well- it depends) is worth the investment. But when you skew the testing pyramid, or worse, make it an testing-ice-cream-cone, that ROI can and will often quickly become negative. ¹Edit: I meant to say that many of these e2e tests are not worth their investment. Testing edge-cases for example: if you need man-hours to write a regression test e2e style and then man-weeks to maintain and run that over coming years, it's often better ROI to just let that regression re-appear and have customers report it. Whereas a unit-test that captures this edge-case costs maybe an hour to write, milliseconds to run and hardly any time to maintain.
    - MoreQARespect 3y ago
- kcartlidge 3y ago
  I find the easiest way (for me) to identify what type of test is running is by looking at responsibility. - A unit test is single responsibility. It tests just that one bit of code with all dependencies stubbed, abstracted, mocked, or removed from consideration in some way. - An integration test is multiple responsibility. It tests just one bit of functionality as a vertical slice through the stack (including [only] relevant dependencies) with all other aspects of the code base eliminated from consideration. - An end to end test is full responsibility. It tests a complete path through all the functionality necessary to complete a 'journey' as a user/consumer of the app/tool. So for example VAT calculation is unit tested as isolated code, invoicing is integration tested as a vertical slice including database etc, and order processing is end-to-end tested from placing the order through to its completion. That's a simplified example and not always accurate depending upon the system and the team perspectives/opinions, but the principle of looking at responsibilities is a very useful rule of thumb.
fesc 3y ago
What would "integration tests" (that you don't write) look then in your opinion? I ask because in my team we also a long time made the destinction between unit/integration based on a stupid technicality in the framework we are using. We stopped doing that and now we mostly write integration tests (which in reality we did for a long time). Of course this all arguing over definitions and kind of stupid but I do agree with the definition of the parent commenter.
- atticora 3y ago
  > What would "integration tests" (that you don't write) look then in your opinion? In our local lingo, an integration test is one that also exercises the front-end, while hitting a fully functional back-end. So you could think of our "unit tests" as small back-end integration tests. If you think that way, we don't write very many pure unit tests, mostly just two flavors of integration tests. That works well for our shop. I'm not concerned about the impurity.
  - ebiester 3y ago
    The "impurity" isn't the problem. The problem is that such integration tests take a longer time to run and in aggregate, it takes minutes to run your test suite. This changes how often you run your tests and slows down your feedback loop. That's why you separate them: not because the integration test isn't valuable, but because it takes longer.
gbacon 3y ago
No one suggests discarding integration tests. The end of the quoted excerpt from Feathers above explicitly supports them. Tests that do these things aren’t bad. Often they are worth writing, and you generally will write them in unit test harnesses. You hinted at the value of separating unit tests and integration tests with your observation about 40-minute unit test runs being way too slow. The process friction it creates means people will check in “obviously correct” changes without running the tests first. Feathers continues: However, it is important to be able to separate them from true unit tests so that you can keep a set of tests that you can run fast whenever you make changes. You want your unit tests to be an easy habit for a quick sanity check. For the situation you described, I’d suggest moving the integration tests to a separate suite that run at least once a day. Ripping that coverage out of your CI may make you uncomfortable. That’s solid engineering intuition. Let your healthy respect for the likelihood of errors creeping in drive you to add at least one fast (less than one-tenth of a second to run is the rule of thumb from Feathers, p. 13) test in the general area of the slower integration tests. The first one may be challenging to write. From here forward, it will never be easier than today. Putting it off is how your team got to the situation now of having to wait 40 minutes for the green bar. One test is better than no tests. Your first case with the fixture and mocks you create will make adding more fast unit tests easier down the road. Yes, just as it’s possible to make mistakes in production code, it’s certainly possible to make mistakes in test code. Unit tests are sometimes brittle and over-constrain. Refactoring them is fair game too and far better than throwing them away.

berkes 3y ago

> My "unit tests" do hit the database and file system, and I have found and fixed many many problems during testing by doing so. I have found many other problems with those calls in production when I didn't do so. No-one said that integration tests can't also be very valuable. From the little context I get that you write integration tests, and that is fine. They are useful, valuable! But they are not unit-tests. edit: on re-reading, I get the feeling that for you "integration tests" are a synonym for "end to end tests". But -at least in most literature- end-to-end tests are a kind of integration-test. But not all integration tests are end-to-end tests. In my software, I'll often have integration tests that swap out some adapter (e.g. the postgres-users-repository, for the memory-users-repository, or fake-users-repository. Or the test-payment for the stripe-payment) but that still test a lot of stuff stacked on top of each-other. Integration tests, just not integration tests that test the entire integration.

lokar 3y ago
And integration tests can also be fast
- alemanek 3y ago
  Test containers really help with this. Should still have the big system tests that run overnight but a set of integration tests using Test Containers to stand in for the infrastructure dependencies is awesome. My team has a ton of those and they run inside a reasonable time frame (5min or so) but we still allow for excluding those from test runs so you can run just the unit tests.
  - darkteflon 3y ago
    I hadn’t heard of Test Containers[1], but it looks really useful - thanks for the rec. [1] https://testcontainers.com/ https://testcontainers.com/
- berkes 3y ago
  Indeed! It's one of the reasons I like the adapter pattern (aka hexagonal architecture) so much. Data flowing through some 100 classes and 300 conditionals then into a `memory-payments` and back takes mere milliseconds. "Memory payments" is then some silly wrapper around a hashmap with the same API as the full-blown production payments-adapter that calls stripe over HTTP. Or the same api as the "production adapter" that wraps some RDBMS running on the other end of the data-center.
MoreQARespect 3y ago
>No-one said that integration tests can't also be very valuable. Integration tests are a better kind of default test because they bring value under pretty much all circumstances. Nobody said that unit tests cant also be valuable and under just the right circumstances i.e. - complex stateless code behind a stable API. Unit tests shine in that environment - theyre not impeded by their crippling lack of realism because that stable abstraction walls off the rest of reality. And theyre very fast. Most code isnt parsers, calculation engines, complex string manipulation, etc. - but when it is unit tests really do kick ass. They just suck so badly at testing code that doesnt fit that mold. Which, to be fair, is most code. I dont write a lot of parsers at work. My job involes moving data into databases, calling APIs, linking up message queues, etc.
- berkes 3y ago
  > Integration tests are a better kind of default test because they bring value under pretty much all circumstances. I respectfully disagree. Not with the last part, that is true: they do bring value under pretty much all circumstances. But the first. Because integration tests come with (extremely) high costs. They are expensive to run. They are much harder (costlier) to write. They are even harder (costlier) to maintain. The common pushback against tests -but they slow down our team a lot- applies to integration tests much more than to unit tests - factors more. And so on. As with everything software-engineering, choosing what tests to write is a tradeoff. And taking all into consideration, e2e or integration tests are often not worth their investment¹. The testing pyramid fixes this, because testing always (well- it depends) is worth the investment. But when you skew the testing pyramid, or worse, make it an testing-ice-cream-cone, that ROI can and will often quickly become negative. ¹Edit: I meant to say that many of these e2e tests are not worth their investment. Testing edge-cases for example: if you need man-hours to write a regression test e2e style and then man-weeks to maintain and run that over coming years, it's often better ROI to just let that regression re-appear and have customers report it. Whereas a unit-test that captures this edge-case costs maybe an hour to write, milliseconds to run and hardly any time to maintain.
  - MoreQARespect 3y ago
kcartlidge 3y ago
I find the easiest way (for me) to identify what type of test is running is by looking at responsibility. - A unit test is single responsibility. It tests just that one bit of code with all dependencies stubbed, abstracted, mocked, or removed from consideration in some way. - An integration test is multiple responsibility. It tests just one bit of functionality as a vertical slice through the stack (including [only] relevant dependencies) with all other aspects of the code base eliminated from consideration. - An end to end test is full responsibility. It tests a complete path through all the functionality necessary to complete a 'journey' as a user/consumer of the app/tool. So for example VAT calculation is unit tested as isolated code, invoicing is integration tested as a vertical slice including database etc, and order processing is end-to-end tested from placing the order through to its completion. That's a simplified example and not always accurate depending upon the system and the team perspectives/opinions, but the principle of looking at responsibilities is a very useful rule of thumb.

fesc 3y ago

What would "integration tests" (that you don't write) look then in your opinion? I ask because in my team we also a long time made the destinction between unit/integration based on a stupid technicality in the framework we are using. We stopped doing that and now we mostly write integration tests (which in reality we did for a long time). Of course this all arguing over definitions and kind of stupid but I do agree with the definition of the parent commenter.

atticora 3y ago
> What would "integration tests" (that you don't write) look then in your opinion? In our local lingo, an integration test is one that also exercises the front-end, while hitting a fully functional back-end. So you could think of our "unit tests" as small back-end integration tests. If you think that way, we don't write very many pure unit tests, mostly just two flavors of integration tests. That works well for our shop. I'm not concerned about the impurity.
- ebiester 3y ago
  The "impurity" isn't the problem. The problem is that such integration tests take a longer time to run and in aggregate, it takes minutes to run your test suite. This changes how often you run your tests and slows down your feedback loop. That's why you separate them: not because the integration test isn't valuable, but because it takes longer.

gbacon 3y ago

No one suggests discarding integration tests. The end of the quoted excerpt from Feathers above explicitly supports them. Tests that do these things aren’t bad. Often they are worth writing, and you generally will write them in unit test harnesses. You hinted at the value of separating unit tests and integration tests with your observation about 40-minute unit test runs being way too slow. The process friction it creates means people will check in “obviously correct” changes without running the tests first. Feathers continues: However, it is important to be able to separate them from true unit tests so that you can keep a set of tests that you can run fast whenever you make changes. You want your unit tests to be an easy habit for a quick sanity check. For the situation you described, I’d suggest moving the integration tests to a separate suite that run at least once a day. Ripping that coverage out of your CI may make you uncomfortable. That’s solid engineering intuition. Let your healthy respect for the likelihood of errors creeping in drive you to add at least one fast (less than one-tenth of a second to run is the rule of thumb from Feathers, p. 13) test in the general area of the slower integration tests. The first one may be challenging to write. From here forward, it will never be easier than today. Putting it off is how your team got to the situation now of having to wait 40 minutes for the green bar. One test is better than no tests. Your first case with the fixture and mocks you create will make adding more fast unit tests easier down the road. Yes, just as it’s possible to make mistakes in production code, it’s certainly possible to make mistakes in test code. Unit tests are sometimes brittle and over-constrain. Refactoring them is fair game too and far better than throwing them away.