6 ms·

I've never really found this to be the case in practice. When I look at well-written unit tests and well-written integration tests, they're usually doing exactl

by MrJohz 1y ago

I've never really found this to be the case in practice. When I look at well-written unit tests and well-written integration tests, they're usually doing exactly the same sort of thing and have very similar concerns in terms of code organisation and test structure. For example, in both cases, the tests work best if I test the subject under test as a black box (i.e. interact only with its public interface) but use my knowledge of its internals to identify the weaknesses that will most require testing. In both cases, I want to structure the code so that the subject under test is as isolated as possible - i.e. no complex interactions with global state, no mocking of unrelated modules, and no complex mechanism to reset anything after the test is done. In both cases, I want the test to run fast, ideally instantaneously, so I get immediate results. The biggest difference is that it's usually harder to write good integration tests because they're interacting with external systems that are generally slower and stateful, so I've got to put extra work into getting the tests themselves to be fast and stateless. But when that works, there's really not much difference at all between a test that tests a single function, and a test that tests a service class with a database dependency.

RHSeeger 1y ago
I'll go with a bank account, because that was one of the initial examples for automated testing. I would write integration/system (different, but similar, imo) to test that the black box integrations with the system work as expected. Generally closer to the "user story" end of things. I would write integration tests for smaller, targeted thing. Like making sure the sort method works in various cases, etc. Individual methods, especially ones that don't interact with data outside what is passed into them (functional methods), are good for unit testing.
- 9rx 1y ago
  > to test that the black box integrations with the system work as expected. Generally closer to the "user story" end of things. This is what unit testing was originally described as. Which confirms my belief that unit testing and integration testing has always been the very same thing. > Individual methods, especially ones that don't interact with data outside what is passed into them (functional methods), are good for unit testing. Perhaps unit testing has come to mean this, but these kinds of tests are rarely ever worth writing, so it is questionable if it even needs a name. Sometimes it can be helpful to isolate a function like that for the sake of pinning down complex logic or edge cases, but is likely you'll want to delete this kind of test once you're done. This is where testing brittleness is born.
  - mrugge 1y ago
    In test-driven development, fast unit tests are a must-have. Integration tests are too slow. If you are not doing test-driven development, can go heavier into integration tests. I find the developer experience is not as fun without good unit tests, and even if velocity metrics are the same, that factor alone is a good reason to focus on writing more fast unit tests.
    - MrJohz 1y ago
      In general, fast tests are a must-have, but I find that means figuring out how to write fast integration tests as well so that they can also be run as part of a TDD-like cycle. In my experience, integration tests can generally be written to be very quick, but maybe my definition of an integration test is different from yours? For me, heavy tests implies end-to-end tests, because at that point you're interacting with the whole system including potentially a browser, and that's just going to be slow whichever way you look at it. But just accessing a database, or parsing and sending http requests doesn't have to be particularly slow, at least not compared to the speed at which I develop. I'd expect to be able to run hundreds of those sorts of tests in less than a second, which is fast enough for me.
      - mrugge 1y ago
        I inherited a django project which has mostly 'unit' tests that flex the ORM and the db, so they are really integration tests and are painfully slow. There is some important logic that happens in the ORM layer and that needs to be tested. At some point I want to find the time to mock the database so that they can be faster, but in some cases I worry about missing important interactions. Domain is highly specialized so not very easy to just know how to untangle the mess.
        9rx 1y ago
        > I worry about missing important interactions. If you are concerned that the ORM won't behave as it claims to, you can write tests targeted at it directly. You can then run the same tests against your mock implementation to show that it conforms to the same contract. But an ORM of any decent quality will already be well tested and shouldn't do unexpected things, so perhaps the worry is for not?
- barrkel 1y ago
  What if, instead of a bank account, it's FooSystemFrobnicationPreparer? Something which is necessary today, but probably should be refactored within the next year or two? Maybe FooSystem will be redesigned to take different inputs,maybe the upstream will change to provide different outputs, maybe responsibility will shift around due to changes in the number of dependencies and it makes sense to vertically integrate some prep to upstream to share it. Unit tests in these circumstances - and they're the majority of unit tests, IME - can act as a drag on the quality of the system. It's better to test things like this at a component level instead of units.
  - MrJohz 1y ago
    I mean, you get to decide what the unit is. I think this is one of the biggest issues with Java and some similar languages, in that it puts so much emphasis on classes (each class gets its own file and is the unit of import) that people used to Java think of classes as _the_ unit boundary, as opposed to being one type of boundary that can sometimes be useful. So `BankAccount` as a class is probably a useful unit boundary: once you've designed the class, you're probably not going to change the interface much, except for possibly adding new methods occasionally. You have a stable boundary there, where in theory you could completely rewrite the internals of the class but the external boundary will stay the same. `FooSystemFrobnicatorPreparer` sounds much more like an internal detail of some other system, I agree, and its interface could easily be rewritten or the class removed entirely if we decide to prepare our frobnication in a different way. But in that case, maybe the `foo.system.frobnicator` package is the unit we want to test as a whole, rather than one specific internal class inside that package. I think a lot of good test and system design is finding these natural fault lines where it's possible to create a relatively stable interface that can hide internal implementation details.
rkomorn 1y ago
I've found that well-written unit tests help me narrow down problems faster during development (eg one unit test failing for a function would show that a change or refactor missed an edge case). I've found that well-written integration tests help me catch workflow-level issues (eg something changed in a dependency that might be mocked in unit tests). So while I think good integration tests are the best way to make sure things should ship, I see a lot of value in good unit tests for day-to-day velocity, particularly in code that's being maintained or updated instead of new code.
soanvig 1y ago
I think this discussion has to be open with what is a "unit" in unit tests. "Integration" consists of many units working together. But my unit can be a function or entire module. That's what people ignore in most discussions about test types.
CuriouslyC 1y ago
Unit tests are good for testing isolated units of code, integration tests test integration. If you wait until you have enough code to test integration, when you actually write the tests you're going to find you've checked in a bunch of almost-working code.
globular-toast 1y ago
It depends what you are doing. Let's say your module implements a way to declare rules and then run some validation function to check objects against those rules. You can't just test every possible set of rules and object that you want to check, even though this is, of course, all that matters. You have to unit test the implementation of the module to be at all confident that it's doing the right thing. So ultimately we write tests at a lower level to deal with the combinatorial explosion of possible inputs at the edge. You should push your tests as far to the edge as possible but no further. If a test at the edge duplicates a test in the middle, delete the test in the middle. But if a test at the edge can't possibly account for everything you're going to bed a test in the middle.
- MrJohz 1y ago
  Yeah, that's similar to how I'd look for the correct place to put my tests. But at that point, a unit test is just the innermost layer of tests, which doesn't feel like a useful distinction. In your example, I might have a set of tests checking how the rules are parsed and interpreted (say), and then a set of tests one level up checking that the validation engine was a whole works, and then another set of tests a level up testing a module that uses the validation engine. The tests for the validation engine won't retest parsing, and the tests for the module using the validation engine won't test validation per se, but there's multiple layers there where each layer contains unit tests focusing on that layer's code specifically.
- troupo 1y ago
  > You can't just test every possible set of rules and object that you want to check, even though this is, of course, all that matters. If it matters, why can't you check? Will your product/app/system not run into these possible sets eventually? > So ultimately we write tests at a lower level to deal with the combinatorial explosion of possible inputs at the edge. Don't you have to write the combinatorial explosion of inputs for the unit tests, too, to test "every possible combination"? If not, and you're only testing a subset, then why not test the whole flow while you're at it?
  - globular-toast 1y ago
JackSlateur 1y ago
My integration tests test things that must work My unit tests test things that must not work
ffsm8 1y ago
Let's say you have a function thats being called to compute a state using hundreds of attributes spread across tens of different objects, and various different levels of nesting. Now, you could create hundreds of different integration tests for each branch of the computation..., most of which will assert the same final output state, but achieved through different transitions Or you can make some integration tests which make sure the logic itself is being called, and then only unittest the specific criteria in isolation. What you're talking about is likely founded in either frontend testing (component tests vs unittest) or backends which have generally pretty trivial logic complexity. In these cases, just doing an integration test gets it done for the most part, but as soon as you got multiple stakeholders giving you sperate requirements and the consumed inputs get bigger and multiply ... Testing via integration tests gets essentially impossible to do in practice
- chamomeal 1y ago
  I feel like that’s where property based testing comes in. Quickcheck style libraries. I only recently started looking into Quickcheck style libraries in the typescript world, and fast-check is fantastic. Like super high quality. Great support for shrinking in all sorts of cases, very well typed, etc. Hooking fast-check up to a real database/redis instance has been incredible for finding bugs. Pair it up with some regular ol case by case integration tests for some seriously robust typescript!

by MrJohz 1y ago

RHSeeger 1y ago

I'll go with a bank account, because that was one of the initial examples for automated testing. I would write integration/system (different, but similar, imo) to test that the black box integrations with the system work as expected. Generally closer to the "user story" end of things. I would write integration tests for smaller, targeted thing. Like making sure the sort method works in various cases, etc. Individual methods, especially ones that don't interact with data outside what is passed into them (functional methods), are good for unit testing.

9rx 1y ago
> to test that the black box integrations with the system work as expected. Generally closer to the "user story" end of things. This is what unit testing was originally described as. Which confirms my belief that unit testing and integration testing has always been the very same thing. > Individual methods, especially ones that don't interact with data outside what is passed into them (functional methods), are good for unit testing. Perhaps unit testing has come to mean this, but these kinds of tests are rarely ever worth writing, so it is questionable if it even needs a name. Sometimes it can be helpful to isolate a function like that for the sake of pinning down complex logic or edge cases, but is likely you'll want to delete this kind of test once you're done. This is where testing brittleness is born.
- mrugge 1y ago
  In test-driven development, fast unit tests are a must-have. Integration tests are too slow. If you are not doing test-driven development, can go heavier into integration tests. I find the developer experience is not as fun without good unit tests, and even if velocity metrics are the same, that factor alone is a good reason to focus on writing more fast unit tests.
  - MrJohz 1y ago
    In general, fast tests are a must-have, but I find that means figuring out how to write fast integration tests as well so that they can also be run as part of a TDD-like cycle. In my experience, integration tests can generally be written to be very quick, but maybe my definition of an integration test is different from yours? For me, heavy tests implies end-to-end tests, because at that point you're interacting with the whole system including potentially a browser, and that's just going to be slow whichever way you look at it. But just accessing a database, or parsing and sending http requests doesn't have to be particularly slow, at least not compared to the speed at which I develop. I'd expect to be able to run hundreds of those sorts of tests in less than a second, which is fast enough for me.
    - mrugge 1y ago
      I inherited a django project which has mostly 'unit' tests that flex the ORM and the db, so they are really integration tests and are painfully slow. There is some important logic that happens in the ORM layer and that needs to be tested. At some point I want to find the time to mock the database so that they can be faster, but in some cases I worry about missing important interactions. Domain is highly specialized so not very easy to just know how to untangle the mess.
      - 9rx 1y ago
        > I worry about missing important interactions. If you are concerned that the ORM won't behave as it claims to, you can write tests targeted at it directly. You can then run the same tests against your mock implementation to show that it conforms to the same contract. But an ORM of any decent quality will already be well tested and shouldn't do unexpected things, so perhaps the worry is for not?
barrkel 1y ago
What if, instead of a bank account, it's FooSystemFrobnicationPreparer? Something which is necessary today, but probably should be refactored within the next year or two? Maybe FooSystem will be redesigned to take different inputs,maybe the upstream will change to provide different outputs, maybe responsibility will shift around due to changes in the number of dependencies and it makes sense to vertically integrate some prep to upstream to share it. Unit tests in these circumstances - and they're the majority of unit tests, IME - can act as a drag on the quality of the system. It's better to test things like this at a component level instead of units.
- MrJohz 1y ago
  I mean, you get to decide what the unit is. I think this is one of the biggest issues with Java and some similar languages, in that it puts so much emphasis on classes (each class gets its own file and is the unit of import) that people used to Java think of classes as _the_ unit boundary, as opposed to being one type of boundary that can sometimes be useful. So `BankAccount` as a class is probably a useful unit boundary: once you've designed the class, you're probably not going to change the interface much, except for possibly adding new methods occasionally. You have a stable boundary there, where in theory you could completely rewrite the internals of the class but the external boundary will stay the same. `FooSystemFrobnicatorPreparer` sounds much more like an internal detail of some other system, I agree, and its interface could easily be rewritten or the class removed entirely if we decide to prepare our frobnication in a different way. But in that case, maybe the `foo.system.frobnicator` package is the unit we want to test as a whole, rather than one specific internal class inside that package. I think a lot of good test and system design is finding these natural fault lines where it's possible to create a relatively stable interface that can hide internal implementation details.

rkomorn 1y ago

I've found that well-written unit tests help me narrow down problems faster during development (eg one unit test failing for a function would show that a change or refactor missed an edge case). I've found that well-written integration tests help me catch workflow-level issues (eg something changed in a dependency that might be mocked in unit tests). So while I think good integration tests are the best way to make sure things should ship, I see a lot of value in good unit tests for day-to-day velocity, particularly in code that's being maintained or updated instead of new code.

soanvig 1y ago

I think this discussion has to be open with what is a "unit" in unit tests. "Integration" consists of many units working together. But my unit can be a function or entire module. That's what people ignore in most discussions about test types.

CuriouslyC 1y ago

Unit tests are good for testing isolated units of code, integration tests test integration. If you wait until you have enough code to test integration, when you actually write the tests you're going to find you've checked in a bunch of almost-working code.

globular-toast 1y ago

It depends what you are doing. Let's say your module implements a way to declare rules and then run some validation function to check objects against those rules. You can't just test every possible set of rules and object that you want to check, even though this is, of course, all that matters. You have to unit test the implementation of the module to be at all confident that it's doing the right thing. So ultimately we write tests at a lower level to deal with the combinatorial explosion of possible inputs at the edge. You should push your tests as far to the edge as possible but no further. If a test at the edge duplicates a test in the middle, delete the test in the middle. But if a test at the edge can't possibly account for everything you're going to bed a test in the middle.

MrJohz 1y ago
Yeah, that's similar to how I'd look for the correct place to put my tests. But at that point, a unit test is just the innermost layer of tests, which doesn't feel like a useful distinction. In your example, I might have a set of tests checking how the rules are parsed and interpreted (say), and then a set of tests one level up checking that the validation engine was a whole works, and then another set of tests a level up testing a module that uses the validation engine. The tests for the validation engine won't retest parsing, and the tests for the module using the validation engine won't test validation per se, but there's multiple layers there where each layer contains unit tests focusing on that layer's code specifically.
troupo 1y ago
> You can't just test every possible set of rules and object that you want to check, even though this is, of course, all that matters. If it matters, why can't you check? Will your product/app/system not run into these possible sets eventually? > So ultimately we write tests at a lower level to deal with the combinatorial explosion of possible inputs at the edge. Don't you have to write the combinatorial explosion of inputs for the unit tests, too, to test "every possible combination"? If not, and you're only testing a subset, then why not test the whole flow while you're at it?
- globular-toast 1y ago

JackSlateur 1y ago

My integration tests test things that must work My unit tests test things that must not work

ffsm8 1y ago

Let's say you have a function thats being called to compute a state using hundreds of attributes spread across tens of different objects, and various different levels of nesting. Now, you could create hundreds of different integration tests for each branch of the computation..., most of which will assert the same final output state, but achieved through different transitions Or you can make some integration tests which make sure the logic itself is being called, and then only unittest the specific criteria in isolation. What you're talking about is likely founded in either frontend testing (component tests vs unittest) or backends which have generally pretty trivial logic complexity. In these cases, just doing an integration test gets it done for the most part, but as soon as you got multiple stakeholders giving you sperate requirements and the consumed inputs get bigger and multiply ... Testing via integration tests gets essentially impossible to do in practice

chamomeal 1y ago
I feel like that’s where property based testing comes in. Quickcheck style libraries. I only recently started looking into Quickcheck style libraries in the typescript world, and fast-check is fantastic. Like super high quality. Great support for shrinking in all sorts of cases, very well typed, etc. Hooking fast-check up to a real database/redis instance has been incredible for finding bugs. Pair it up with some regular ol case by case integration tests for some seriously robust typescript!