6 ms·

Integration tests and Unit tests are different tools; and each has their place and purpose. Using one "instead" of the other is a mistake.

MrJohz 1y ago
I've never really found this to be the case in practice. When I look at well-written unit tests and well-written integration tests, they're usually doing exactly the same sort of thing and have very similar concerns in terms of code organisation and test structure. For example, in both cases, the tests work best if I test the subject under test as a black box (i.e. interact only with its public interface) but use my knowledge of its internals to identify the weaknesses that will most require testing. In both cases, I want to structure the code so that the subject under test is as isolated as possible - i.e. no complex interactions with global state, no mocking of unrelated modules, and no complex mechanism to reset anything after the test is done. In both cases, I want the test to run fast, ideally instantaneously, so I get immediate results. The biggest difference is that it's usually harder to write good integration tests because they're interacting with external systems that are generally slower and stateful, so I've got to put extra work into getting the tests themselves to be fast and stateless. But when that works, there's really not much difference at all between a test that tests a single function, and a test that tests a service class with a database dependency.
- RHSeeger 1y ago
  I'll go with a bank account, because that was one of the initial examples for automated testing. I would write integration/system (different, but similar, imo) to test that the black box integrations with the system work as expected. Generally closer to the "user story" end of things. I would write integration tests for smaller, targeted thing. Like making sure the sort method works in various cases, etc. Individual methods, especially ones that don't interact with data outside what is passed into them (functional methods), are good for unit testing.
  - 9rx 1y ago
    > to test that the black box integrations with the system work as expected. Generally closer to the "user story" end of things. This is what unit testing was originally described as. Which confirms my belief that unit testing and integration testing has always been the very same thing. > Individual methods, especially ones that don't interact with data outside what is passed into them (functional methods), are good for unit testing. Perhaps unit testing has come to mean this, but these kinds of tests are rarely ever worth writing, so it is questionable if it even needs a name. Sometimes it can be helpful to isolate a function like that for the sake of pinning down complex logic or edge cases, but is likely you'll want to delete this kind of test once you're done. This is where testing brittleness is born.
    - mrugge 1y ago
      In test-driven development, fast unit tests are a must-have. Integration tests are too slow. If you are not doing test-driven development, can go heavier into integration tests. I find the developer experience is not as fun without good unit tests, and even if velocity metrics are the same, that factor alone is a good reason to focus on writing more fast unit tests.
      - MrJohz 1y ago
        In general, fast tests are a must-have, but I find that means figuring out how to write fast integration tests as well so that they can also be run as part of a TDD-like cycle. In my experience, integration tests can generally be written to be very quick, but maybe my definition of an integration test is different from yours? For me, heavy tests implies end-to-end tests, because at that point you're interacting with the whole system including potentially a browser, and that's just going to be slow whichever way you look at it. But just accessing a database, or parsing and sending http requests doesn't have to be particularly slow, at least not compared to the speed at which I develop. I'd expect to be able to run hundreds of those sorts of tests in less than a second, which is fast enough for me.
        1y ago
- rkomorn 1y ago
  I've found that well-written unit tests help me narrow down problems faster during development (eg one unit test failing for a function would show that a change or refactor missed an edge case). I've found that well-written integration tests help me catch workflow-level issues (eg something changed in a dependency that might be mocked in unit tests). So while I think good integration tests are the best way to make sure things should ship, I see a lot of value in good unit tests for day-to-day velocity, particularly in code that's being maintained or updated instead of new code.
- soanvig 1y ago
  I think this discussion has to be open with what is a "unit" in unit tests. "Integration" consists of many units working together. But my unit can be a function or entire module. That's what people ignore in most discussions about test types.
- CuriouslyC 1y ago
  Unit tests are good for testing isolated units of code, integration tests test integration. If you wait until you have enough code to test integration, when you actually write the tests you're going to find you've checked in a bunch of almost-working code.
- globular-toast 1y ago
  It depends what you are doing. Let's say your module implements a way to declare rules and then run some validation function to check objects against those rules. You can't just test every possible set of rules and object that you want to check, even though this is, of course, all that matters. You have to unit test the implementation of the module to be at all confident that it's doing the right thing. So ultimately we write tests at a lower level to deal with the combinatorial explosion of possible inputs at the edge. You should push your tests as far to the edge as possible but no further. If a test at the edge duplicates a test in the middle, delete the test in the middle. But if a test at the edge can't possibly account for everything you're going to bed a test in the middle.
  - MrJohz 1y ago
    Yeah, that's similar to how I'd look for the correct place to put my tests. But at that point, a unit test is just the innermost layer of tests, which doesn't feel like a useful distinction. In your example, I might have a set of tests checking how the rules are parsed and interpreted (say), and then a set of tests one level up checking that the validation engine was a whole works, and then another set of tests a level up testing a module that uses the validation engine. The tests for the validation engine won't retest parsing, and the tests for the module using the validation engine won't test validation per se, but there's multiple layers there where each layer contains unit tests focusing on that layer's code specifically.
  - troupo 1y ago
    > You can't just test every possible set of rules and object that you want to check, even though this is, of course, all that matters. If it matters, why can't you check? Will your product/app/system not run into these possible sets eventually? > So ultimately we write tests at a lower level to deal with the combinatorial explosion of possible inputs at the edge. Don't you have to write the combinatorial explosion of inputs for the unit tests, too, to test "every possible combination"? If not, and you're only testing a subset, then why not test the whole flow while you're at it?
    - globular-toast 1y ago
- JackSlateur 1y ago
  My integration tests test things that must work My unit tests test things that must not work
- ffsm8 1y ago
  Let's say you have a function thats being called to compute a state using hundreds of attributes spread across tens of different objects, and various different levels of nesting. Now, you could create hundreds of different integration tests for each branch of the computation..., most of which will assert the same final output state, but achieved through different transitions Or you can make some integration tests which make sure the logic itself is being called, and then only unittest the specific criteria in isolation. What you're talking about is likely founded in either frontend testing (component tests vs unittest) or backends which have generally pretty trivial logic complexity. In these cases, just doing an integration test gets it done for the most part, but as soon as you got multiple stakeholders giving you sperate requirements and the consumed inputs get bigger and multiply ... Testing via integration tests gets essentially impossible to do in practice
  - chamomeal 1y ago
    I feel like that’s where property based testing comes in. Quickcheck style libraries. I only recently started looking into Quickcheck style libraries in the typescript world, and fast-check is fantastic. Like super high quality. Great support for shrinking in all sorts of cases, very well typed, etc. Hooking fast-check up to a real database/redis instance has been incredible for finding bugs. Pair it up with some regular ol case by case integration tests for some seriously robust typescript!
simianwords 1y ago
Wow I hate this dogmatism. It is indeed better to use one instead of the other. Let’s stop pretending all are equally good and we need every type of test. Sometimes you just don’t need unit tests and it’s okay to admit it and work accordingly.
- RHSeeger 1y ago
  And sometimes you only need screws, instead of nails; or vice versa. But that doesn't invalidate the tool; it just means your use case doesn't need it.
  - imiric 1y ago
    You can't build a house without nails and screws, though. Sure, if you're only writing a small script, you might not need tests at all. But as soon as that program evolves into a system that interacts with other systems, you need to test each component in isolation, as well as how it interacts with other systems. So this idea that unit tests are not useful is coming from a place of laziness. Some developers see it as a chore that slows them down, instead of seeing it as insurance that makes their life easier in the long run, while also ensuring the system works as intended at all layers.
- imiric 1y ago
  You claim it's dogmatism, yet do the same thing in reverse. (: Unit and integration tests test different layers of the system, and one isn't inherently better or more useful than the other. They complement each other to cover behavior that is impossible to test otherwise. You can't test low-level functionality in integration tests, just as you can't test high-level functionality in unit tests. There's nothing dogmatic about that statement. If you disagree with it, that's your prerogative, but it's also my opinion that it is a mistake. It is a harmful mentality that makes code bases risky to change, and regressions more likely. So feel free to adopt it in your personal projects if you wish, but don't be surprised if you get push back on it when working in a team. Unless your teammates think the same, in which case, good luck to you all.
  - 1y ago
s_ting765 1y ago
Integration tests make unit tests absolutely redundant.
- integralid 1y ago
  Integration tests are as old as unit tests, and both predate their names. When exactly were unit tests made redundant? I don't see the point of your quip without a trace of actual argument. I feel like I don't write enough tests, and when I do they're usually integration tests, but some things - algorithms, complex but pure functions, data structures - absolutely deserve their unit tests that can't be reasonably replaced by integration/e2e tests.
  - s_ting765 1y ago
    Here's the argument backing up my claim. Unit tests don't matter when you have other types of testing like functional or integration testing that will tell you whether your code has the intended behavior and effect when run. In the above statement unit tests is also considered as code. That's where the redundancy comes from.
    - skydhash 1y ago
      Unit tests do matter, especially when the logic is somewhat complex or very defined (splitting money, parsing some message). So unless the specs change, you rarely have to modify the tests. So it helps more in a technical sense, catching developer mistakes. Just like qa tests on some small part of the car can spot defect early on. Integrated tests is more about ensuring what matters to Product. A car that refuses to start is worthless for most cases. But the engine light and a window that can’t open is not usually a dealbreaker. Unit tests can help pinpoint an issue or ensure that a specs is implemented. But that’s mostly relevant to the developer world. So for a proper DX, add unit tests to help pinpoint bugs faster, especially with code that doesn’t change as much and where knowledge can be lost.
- yakshaving_jgt 1y ago
  No they don’t. If you’re, for example, writing a web application, and you have an endpoint which parses some data from the request and then responds with the result of that computation, why the hell would you test the fine-grained behaviour of your parser by emulating HTTP requests against your server? Testing the parsing function in isolation is orders of magnitude cheaper.
JimDabell 1y ago
In my experience, a bug that causes a unit test to fail also causes an integration or E2E test to fail. Also, it’s relatively easy to determine the cause of the problem given a change and a failing integration/E2E test. Unit tests are usually much quicker to run, but you also need a lot more of them. I think when you combine these things, it’s easy to reach the conclusion that unit tests are redundant.
troupo 1y ago
In the absolute vast majority of cases unit tests are useless. Because your end result should be a working system, not isolated working units with everything useful mocked out of existence. Integration tests will cover every use case unit tests are supposed to cover if you actually tests the system behavior.
- EliRivers 1y ago
  At my last employer, some of our customers used our software to run systems of which the hardware value alone exceeded the entire market cap of my employer. They ran systems with a physical footprint many many times the area of my employer's leased offices and workspaces. The physical hardware devices that our software ran alone exceeded the value of my employer's market cap; just buying one of each would have been an enormous expense, ignoring that many of them were no longer available new and some of them were decades old, sitting alongside hardware that was made in the last six months. All the physical hardware working in synchronicity, linking up with similar sites in two other locations around the world, handing off to each other to follow the sun. It would be nice to fully test system behaviour, but to do so would have bankrupted the company long before even coming close.
  - troupo 1y ago
    How were you making sure that your system actually works? So you have unit tests testing things in isolation? Did you not test how they work together? Did you never run your system to see it actually works and behaves as expected? You just YOLO'd it over to customers and prayed?
    - EliRivers 1y ago
      I think the record uptime for a customer before they shut down one process to upgrade the software (leaving a dozen other such processes running, taking over the work in turn as each was upgraded) was on the order of six years. This was a set of 24 hour broadcast channels. "How were you making sure that your system actually works?" Good design and good software engineering goes a long way. When you know that you cannot test by simply doing everything the customer will do, you have to think about what tests you can do that will indicate how the system will operate under a load that's orders of magnitude greater than what you can do yourself, with hardware you've never even seen. You have to think about how to write software that is likely to be high quality even if you can't test it how you'd like to. For example, one can design the architecture in such a way that adding more load, more devices, will only linearly increase the demands on resources, and then from testing infer what the loads will be on actual customer sites. Any non-linearity in those regard were identified, if not at the design stage, in the unit-testing thereof. One can design the code in such a way that the internal mechanisms of how devices work are suitably abstracted away, leaving as best one can manage common interfaces, and then rather than have to test with the exact arrangements of hardware customers will have, test with devices that to the extent possible, simulate the interactions our software will see. In this regard, it turns out that many devices that purport to meet standard protocols actually meet "variations about the theme" of protocols. But this too can be mitigated and handled to a degree by careful design and thought in the software engineering. The learning from doing this with one set of devices and protocols spreads to significantly different devices and protocols; every subsequent fresh design for the next iteration or generation of hardware is better and more resilient. A software engineering organisation that learns and retains knowledge and experience can go long way. One can recognise that running live on customers sites is itself an opportunity. Some customers would never say a thing, for years on end. Some would want to be involved and would regularly talk about things they'd seen, unexpected things that happened, loads and events and so on; one can ensure that all that information is gathered by the sales people, the support reps, anyone and everyone who talks to the customers, and passed back effectively to have the results of that testing applied. For doomsday scenarios, such as crashes, resource exhaustion, pathological behaviours and so on; good logging and live measurements and dump catching etc can at least feed back so that this situation (which we would never be able to truly test ourselves) is not wasted, and gets fixed, and the lessons of it applied forwards into design and development. Harsh for the customer who finds an issue, but great for the hundred customers who will never hit it because it was tested by that unlucky customer. We'd be fools not to gather as much information as we could from poor customer experiences. One can get hold of cheap, twenty year old devices that in theory match the same protocol, and go to town on them (some customers will actually be using that exact device and contemporaries - some customers will have brand new hardware that costs a tenth of my employer's market cap). From that, get an idea of how the software performs. Get another cheap device from ebay that is a decade old, and test it; see where it fails, but don't just fix those failures. From them, and similar repeats of the process, learn at a more fundamental level how devices differ and develop more general solutions that will either then be resilient to some new piece of hardware that hasn't even been made yet, or at least will not go wrong in such a way that the whole system is taken out and the poorly-supported brand new hardware is clearly seen by the software and reported on. There's more. There's so much more, but once you have no choice but to come up with cheap, fast testing that nonetheless give a good indication of how the system will work when someone spends tens of millions on the hardware, software engineers can really come up with some smart, reliable ideas. It can also be really fun and satisfying to work on this. "You just YOLO'd it over to customers and prayed?" Absolutely not. It was all tested, repeatedly, over and over, and over the course of about fifteen year became remarkably resilient, adaptable, resource light, and so on. All the good things one would hope for. In a pinch, a small system could be run from someone's laptop; at the top end, banks and banks of servers with their fans banshee wailing 24 hours a day, with dozens of the principal processes (i.e. the main executable that runs) all running, all talking to each other across countries and time zones, handling their own redundancy against individual processes turning off. Again, when you begin knowing that the software has to deliver on such a range of systems, where one customer is two college kids in a basement and one customer is valued in the tens of billions (although doing a lot more, of course, than just what our software let them do), design and good software engineering goes a very long way.
barrkel 1y ago
I don't agree. When you implement some new code and want to cover it in testing, you have a choice where you call into the system from your tests. Calling at the lowest level is not always the right choice, but it's the way the testing pyramid will bias you. Instead - instead, there really is an instead here - you can call at a higher level which is less brittle to refactoring, has less complex setup, doesn't involve mocks that may not behave like the real thing, but still runs quickly due to fakes stubbing out expensive dependencies.
queenkjuul 1y ago
I do agree, but for this to work everyone has to be on the same page about what is a "unit" I think most people are largely on the same page, but idk. Anyway what i mean is that as long as your unit tests are very narrowly scoped and testing specific chunks of logic, they really shouldn't break very often. I feel like my current project has done a decent job of this. Our unit tests rarely break, but to be fair, our integration tests fail too often. I'm a UI developer, I've seen a lot of "unit" tests in PRs that go too far beyond testing specific logic and end up being brittle and not very useful.
9rx 1y ago
Integration tests and unit tests are the exact same thing. If you can somehow find a discernible difference between them, you're doing something horribly wrong.

MrJohz 1y ago

I've never really found this to be the case in practice. When I look at well-written unit tests and well-written integration tests, they're usually doing exactly the same sort of thing and have very similar concerns in terms of code organisation and test structure. For example, in both cases, the tests work best if I test the subject under test as a black box (i.e. interact only with its public interface) but use my knowledge of its internals to identify the weaknesses that will most require testing. In both cases, I want to structure the code so that the subject under test is as isolated as possible - i.e. no complex interactions with global state, no mocking of unrelated modules, and no complex mechanism to reset anything after the test is done. In both cases, I want the test to run fast, ideally instantaneously, so I get immediate results. The biggest difference is that it's usually harder to write good integration tests because they're interacting with external systems that are generally slower and stateful, so I've got to put extra work into getting the tests themselves to be fast and stateless. But when that works, there's really not much difference at all between a test that tests a single function, and a test that tests a service class with a database dependency.

RHSeeger 1y ago
I'll go with a bank account, because that was one of the initial examples for automated testing. I would write integration/system (different, but similar, imo) to test that the black box integrations with the system work as expected. Generally closer to the "user story" end of things. I would write integration tests for smaller, targeted thing. Like making sure the sort method works in various cases, etc. Individual methods, especially ones that don't interact with data outside what is passed into them (functional methods), are good for unit testing.
- 9rx 1y ago
  > to test that the black box integrations with the system work as expected. Generally closer to the "user story" end of things. This is what unit testing was originally described as. Which confirms my belief that unit testing and integration testing has always been the very same thing. > Individual methods, especially ones that don't interact with data outside what is passed into them (functional methods), are good for unit testing. Perhaps unit testing has come to mean this, but these kinds of tests are rarely ever worth writing, so it is questionable if it even needs a name. Sometimes it can be helpful to isolate a function like that for the sake of pinning down complex logic or edge cases, but is likely you'll want to delete this kind of test once you're done. This is where testing brittleness is born.
  - mrugge 1y ago
    In test-driven development, fast unit tests are a must-have. Integration tests are too slow. If you are not doing test-driven development, can go heavier into integration tests. I find the developer experience is not as fun without good unit tests, and even if velocity metrics are the same, that factor alone is a good reason to focus on writing more fast unit tests.
    - MrJohz 1y ago
      In general, fast tests are a must-have, but I find that means figuring out how to write fast integration tests as well so that they can also be run as part of a TDD-like cycle. In my experience, integration tests can generally be written to be very quick, but maybe my definition of an integration test is different from yours? For me, heavy tests implies end-to-end tests, because at that point you're interacting with the whole system including potentially a browser, and that's just going to be slow whichever way you look at it. But just accessing a database, or parsing and sending http requests doesn't have to be particularly slow, at least not compared to the speed at which I develop. I'd expect to be able to run hundreds of those sorts of tests in less than a second, which is fast enough for me.
      - 1y ago
rkomorn 1y ago
I've found that well-written unit tests help me narrow down problems faster during development (eg one unit test failing for a function would show that a change or refactor missed an edge case). I've found that well-written integration tests help me catch workflow-level issues (eg something changed in a dependency that might be mocked in unit tests). So while I think good integration tests are the best way to make sure things should ship, I see a lot of value in good unit tests for day-to-day velocity, particularly in code that's being maintained or updated instead of new code.
soanvig 1y ago
I think this discussion has to be open with what is a "unit" in unit tests. "Integration" consists of many units working together. But my unit can be a function or entire module. That's what people ignore in most discussions about test types.
CuriouslyC 1y ago
Unit tests are good for testing isolated units of code, integration tests test integration. If you wait until you have enough code to test integration, when you actually write the tests you're going to find you've checked in a bunch of almost-working code.
globular-toast 1y ago
It depends what you are doing. Let's say your module implements a way to declare rules and then run some validation function to check objects against those rules. You can't just test every possible set of rules and object that you want to check, even though this is, of course, all that matters. You have to unit test the implementation of the module to be at all confident that it's doing the right thing. So ultimately we write tests at a lower level to deal with the combinatorial explosion of possible inputs at the edge. You should push your tests as far to the edge as possible but no further. If a test at the edge duplicates a test in the middle, delete the test in the middle. But if a test at the edge can't possibly account for everything you're going to bed a test in the middle.
- MrJohz 1y ago
  Yeah, that's similar to how I'd look for the correct place to put my tests. But at that point, a unit test is just the innermost layer of tests, which doesn't feel like a useful distinction. In your example, I might have a set of tests checking how the rules are parsed and interpreted (say), and then a set of tests one level up checking that the validation engine was a whole works, and then another set of tests a level up testing a module that uses the validation engine. The tests for the validation engine won't retest parsing, and the tests for the module using the validation engine won't test validation per se, but there's multiple layers there where each layer contains unit tests focusing on that layer's code specifically.
- troupo 1y ago
  > You can't just test every possible set of rules and object that you want to check, even though this is, of course, all that matters. If it matters, why can't you check? Will your product/app/system not run into these possible sets eventually? > So ultimately we write tests at a lower level to deal with the combinatorial explosion of possible inputs at the edge. Don't you have to write the combinatorial explosion of inputs for the unit tests, too, to test "every possible combination"? If not, and you're only testing a subset, then why not test the whole flow while you're at it?
  - globular-toast 1y ago
JackSlateur 1y ago
My integration tests test things that must work My unit tests test things that must not work
ffsm8 1y ago
Let's say you have a function thats being called to compute a state using hundreds of attributes spread across tens of different objects, and various different levels of nesting. Now, you could create hundreds of different integration tests for each branch of the computation..., most of which will assert the same final output state, but achieved through different transitions Or you can make some integration tests which make sure the logic itself is being called, and then only unittest the specific criteria in isolation. What you're talking about is likely founded in either frontend testing (component tests vs unittest) or backends which have generally pretty trivial logic complexity. In these cases, just doing an integration test gets it done for the most part, but as soon as you got multiple stakeholders giving you sperate requirements and the consumed inputs get bigger and multiply ... Testing via integration tests gets essentially impossible to do in practice
- chamomeal 1y ago
  I feel like that’s where property based testing comes in. Quickcheck style libraries. I only recently started looking into Quickcheck style libraries in the typescript world, and fast-check is fantastic. Like super high quality. Great support for shrinking in all sorts of cases, very well typed, etc. Hooking fast-check up to a real database/redis instance has been incredible for finding bugs. Pair it up with some regular ol case by case integration tests for some seriously robust typescript!

simianwords 1y ago

Wow I hate this dogmatism. It is indeed better to use one instead of the other. Let’s stop pretending all are equally good and we need every type of test. Sometimes you just don’t need unit tests and it’s okay to admit it and work accordingly.

RHSeeger 1y ago
And sometimes you only need screws, instead of nails; or vice versa. But that doesn't invalidate the tool; it just means your use case doesn't need it.
- imiric 1y ago
  You can't build a house without nails and screws, though. Sure, if you're only writing a small script, you might not need tests at all. But as soon as that program evolves into a system that interacts with other systems, you need to test each component in isolation, as well as how it interacts with other systems. So this idea that unit tests are not useful is coming from a place of laziness. Some developers see it as a chore that slows them down, instead of seeing it as insurance that makes their life easier in the long run, while also ensuring the system works as intended at all layers.
imiric 1y ago
You claim it's dogmatism, yet do the same thing in reverse. (: Unit and integration tests test different layers of the system, and one isn't inherently better or more useful than the other. They complement each other to cover behavior that is impossible to test otherwise. You can't test low-level functionality in integration tests, just as you can't test high-level functionality in unit tests. There's nothing dogmatic about that statement. If you disagree with it, that's your prerogative, but it's also my opinion that it is a mistake. It is a harmful mentality that makes code bases risky to change, and regressions more likely. So feel free to adopt it in your personal projects if you wish, but don't be surprised if you get push back on it when working in a team. Unless your teammates think the same, in which case, good luck to you all.
- 1y ago

s_ting765 1y ago

Integration tests make unit tests absolutely redundant.

integralid 1y ago
Integration tests are as old as unit tests, and both predate their names. When exactly were unit tests made redundant? I don't see the point of your quip without a trace of actual argument. I feel like I don't write enough tests, and when I do they're usually integration tests, but some things - algorithms, complex but pure functions, data structures - absolutely deserve their unit tests that can't be reasonably replaced by integration/e2e tests.
- s_ting765 1y ago
  Here's the argument backing up my claim. Unit tests don't matter when you have other types of testing like functional or integration testing that will tell you whether your code has the intended behavior and effect when run. In the above statement unit tests is also considered as code. That's where the redundancy comes from.
  - skydhash 1y ago
    Unit tests do matter, especially when the logic is somewhat complex or very defined (splitting money, parsing some message). So unless the specs change, you rarely have to modify the tests. So it helps more in a technical sense, catching developer mistakes. Just like qa tests on some small part of the car can spot defect early on. Integrated tests is more about ensuring what matters to Product. A car that refuses to start is worthless for most cases. But the engine light and a window that can’t open is not usually a dealbreaker. Unit tests can help pinpoint an issue or ensure that a specs is implemented. But that’s mostly relevant to the developer world. So for a proper DX, add unit tests to help pinpoint bugs faster, especially with code that doesn’t change as much and where knowledge can be lost.
yakshaving_jgt 1y ago
No they don’t. If you’re, for example, writing a web application, and you have an endpoint which parses some data from the request and then responds with the result of that computation, why the hell would you test the fine-grained behaviour of your parser by emulating HTTP requests against your server? Testing the parsing function in isolation is orders of magnitude cheaper.

JimDabell 1y ago

In my experience, a bug that causes a unit test to fail also causes an integration or E2E test to fail. Also, it’s relatively easy to determine the cause of the problem given a change and a failing integration/E2E test. Unit tests are usually much quicker to run, but you also need a lot more of them. I think when you combine these things, it’s easy to reach the conclusion that unit tests are redundant.

troupo 1y ago

In the absolute vast majority of cases unit tests are useless. Because your end result should be a working system, not isolated working units with everything useful mocked out of existence. Integration tests will cover every use case unit tests are supposed to cover if you actually tests the system behavior.

EliRivers 1y ago
At my last employer, some of our customers used our software to run systems of which the hardware value alone exceeded the entire market cap of my employer. They ran systems with a physical footprint many many times the area of my employer's leased offices and workspaces. The physical hardware devices that our software ran alone exceeded the value of my employer's market cap; just buying one of each would have been an enormous expense, ignoring that many of them were no longer available new and some of them were decades old, sitting alongside hardware that was made in the last six months. All the physical hardware working in synchronicity, linking up with similar sites in two other locations around the world, handing off to each other to follow the sun. It would be nice to fully test system behaviour, but to do so would have bankrupted the company long before even coming close.
- troupo 1y ago
  How were you making sure that your system actually works? So you have unit tests testing things in isolation? Did you not test how they work together? Did you never run your system to see it actually works and behaves as expected? You just YOLO'd it over to customers and prayed?
  - EliRivers 1y ago
    I think the record uptime for a customer before they shut down one process to upgrade the software (leaving a dozen other such processes running, taking over the work in turn as each was upgraded) was on the order of six years. This was a set of 24 hour broadcast channels. "How were you making sure that your system actually works?" Good design and good software engineering goes a long way. When you know that you cannot test by simply doing everything the customer will do, you have to think about what tests you can do that will indicate how the system will operate under a load that's orders of magnitude greater than what you can do yourself, with hardware you've never even seen. You have to think about how to write software that is likely to be high quality even if you can't test it how you'd like to. For example, one can design the architecture in such a way that adding more load, more devices, will only linearly increase the demands on resources, and then from testing infer what the loads will be on actual customer sites. Any non-linearity in those regard were identified, if not at the design stage, in the unit-testing thereof. One can design the code in such a way that the internal mechanisms of how devices work are suitably abstracted away, leaving as best one can manage common interfaces, and then rather than have to test with the exact arrangements of hardware customers will have, test with devices that to the extent possible, simulate the interactions our software will see. In this regard, it turns out that many devices that purport to meet standard protocols actually meet "variations about the theme" of protocols. But this too can be mitigated and handled to a degree by careful design and thought in the software engineering. The learning from doing this with one set of devices and protocols spreads to significantly different devices and protocols; every subsequent fresh design for the next iteration or generation of hardware is better and more resilient. A software engineering organisation that learns and retains knowledge and experience can go long way. One can recognise that running live on customers sites is itself an opportunity. Some customers would never say a thing, for years on end. Some would want to be involved and would regularly talk about things they'd seen, unexpected things that happened, loads and events and so on; one can ensure that all that information is gathered by the sales people, the support reps, anyone and everyone who talks to the customers, and passed back effectively to have the results of that testing applied. For doomsday scenarios, such as crashes, resource exhaustion, pathological behaviours and so on; good logging and live measurements and dump catching etc can at least feed back so that this situation (which we would never be able to truly test ourselves) is not wasted, and gets fixed, and the lessons of it applied forwards into design and development. Harsh for the customer who finds an issue, but great for the hundred customers who will never hit it because it was tested by that unlucky customer. We'd be fools not to gather as much information as we could from poor customer experiences. One can get hold of cheap, twenty year old devices that in theory match the same protocol, and go to town on them (some customers will actually be using that exact device and contemporaries - some customers will have brand new hardware that costs a tenth of my employer's market cap). From that, get an idea of how the software performs. Get another cheap device from ebay that is a decade old, and test it; see where it fails, but don't just fix those failures. From them, and similar repeats of the process, learn at a more fundamental level how devices differ and develop more general solutions that will either then be resilient to some new piece of hardware that hasn't even been made yet, or at least will not go wrong in such a way that the whole system is taken out and the poorly-supported brand new hardware is clearly seen by the software and reported on. There's more. There's so much more, but once you have no choice but to come up with cheap, fast testing that nonetheless give a good indication of how the system will work when someone spends tens of millions on the hardware, software engineers can really come up with some smart, reliable ideas. It can also be really fun and satisfying to work on this. "You just YOLO'd it over to customers and prayed?" Absolutely not. It was all tested, repeatedly, over and over, and over the course of about fifteen year became remarkably resilient, adaptable, resource light, and so on. All the good things one would hope for. In a pinch, a small system could be run from someone's laptop; at the top end, banks and banks of servers with their fans banshee wailing 24 hours a day, with dozens of the principal processes (i.e. the main executable that runs) all running, all talking to each other across countries and time zones, handling their own redundancy against individual processes turning off. Again, when you begin knowing that the software has to deliver on such a range of systems, where one customer is two college kids in a basement and one customer is valued in the tens of billions (although doing a lot more, of course, than just what our software let them do), design and good software engineering goes a very long way.

barrkel 1y ago

I don't agree. When you implement some new code and want to cover it in testing, you have a choice where you call into the system from your tests. Calling at the lowest level is not always the right choice, but it's the way the testing pyramid will bias you. Instead - instead, there really is an instead here - you can call at a higher level which is less brittle to refactoring, has less complex setup, doesn't involve mocks that may not behave like the real thing, but still runs quickly due to fakes stubbing out expensive dependencies.

queenkjuul 1y ago

I do agree, but for this to work everyone has to be on the same page about what is a "unit" I think most people are largely on the same page, but idk. Anyway what i mean is that as long as your unit tests are very narrowly scoped and testing specific chunks of logic, they really shouldn't break very often. I feel like my current project has done a decent job of this. Our unit tests rarely break, but to be fair, our integration tests fail too often. I'm a UI developer, I've seen a lot of "unit" tests in PRs that go too far beyond testing specific logic and end up being brittle and not very useful.

9rx 1y ago

Integration tests and unit tests are the exact same thing. If you can somehow find a discernible difference between them, you're doing something horribly wrong.