3 ms·

The problem is that “unit” isn’t well defined. Anything from a method to a service can be considered the unit under test.

by Ma8ee 3y ago

necovek 3y ago
I think it's actually quite simple to come to a common language for unit and integration tests if you start from "integration" instead: if there is any sort of integration (two things interacting: your test setup can tell you that's happening), it's an integration test. If there are no integrations, it's a unit test. As in, it's all in the name. How does one fit a whole "service" (if we are talking something like a networking service like web API server there and not a DDD-style service) there really beats me, but I am happy to hear you out. I'd note that things like setting up a client and then configuring it to access a service already fails that test, even if you hide that complexity into fixtures or "external" test setup. Edit: for avoidance of any doubt, these are terms without clear, widely accepted definitions. I am mostly sharing how I am making what I think is a useful distinction to drive me to write properly scoped tests and structured code.
- Ma8ee 3y ago
  > if there is any sort of integration (two things interacting: your test setup can tell you that's happening), it's an integration test. Two things of what? There are people that claim that as soon as there are more than one method involved, it's an integration test, and more common, that as soon as there are more than one class involved, it is. This lead to absurd mocking of utility classes to make sure that only one thing is tested at the time. And then, of course, the tests become very fragile. And while there are situations when it is useful to only test one specific thing, what we really care about is the behaviour of whole system of many parts. We mock (fake or stub) to make the tests faster, repeatable and reliable. That is, what we don't want in our tests are slow calls, over slow networks or lengthy calculations. We want the calls to be repeatable, so we don't want to call, say, a database where the state changes between test runs. And we don't want to make calls to unreliable services; unreliable because they are under development or because they are over an unreliable network.
  - necovek 3y ago
    I would be in the camp that says two methods calling each other are an integration, but I avoid mocking as much as possible. But if you need to use mocking there heavily, your code is not structured well: look for functional style and acyclic directed graph structure of calls.
jart 3y ago
At Google they draw the distinction based on i/o. Anything that does i/o wouldn't be considered a unit test by them. Unit tests are usually for algorithms, math functions, data structure classes, and stuff like that. Most developers have never needed to write any of those things because they're abstracted by modern languages, which is why I suspect the terminology confuses some people.
- randomdata 3y ago
  Unit testing in the realm of software originated from the Smalltalk world, where a unit described the test suite for a class. That, with some liberty around the use of class in languages that are not class-centric, reflects the vast, vast, vast majority of tests I see out there in the wild. The reality is that unit testing swept the world and has become synonymous with testing. The days of yore when things like capture and replay testing were it are behind us. Today, it is uncommon for anyone to write anything other than unit tests (as originally defined). And that's why "unit testing" has become confusing. Its original meaning no longer serves a purpose. Today, we just call it "testing", leaving those who wish to continue to use "unit" for the sake of fashion to invent pet definitions; definitions not shared by anyone else.
- kerkeslager 3y ago
  That's a useful distinction for creating a shared language within Google, and it might be a useful distinction for your team too. But it's definitely not the only way to divide things up. The important thing is that the terminology your term uses is useful for your team. One project I'm working on has two kinds of tests: unit tests run within the code and test classes, state transitions, etc. by calling the code directly, while integration tests are run in Selenium in a headless browser from the perspective of the user. Some of the unit tests on models do hit the DB (i.e. they do I/O).
kerkeslager 3y ago
That's not a problem, that's a hangup you have. We don't need a global definition of what a unit is, we just need to know what unit is being tested by a given unit test. This is important because it's how we understand coverage: 1. We need to understand what's within the unit (covered by the test) so we don't write redundant tests, and... 2. ...we need to understand what's not within the unit (not covered by the test) so we can write separate tests to cover that unit. The unit can be anything, as long as we know what the unit we're testing when we read/write/run the test. An integration test also needs to know what it's testing, but now you're looking at it from an outside perspective. You no longer care which specific areas of code (units) are within the integration, you just care what features are within the integration. Sometimes a refactor will mean that feature A no longer causes class B to call class C: that probably means changes to B's unit tests, because the code unit changed, but it doesn't mean changes to the integration test because the feature didn't change. The important thing is to understand what's being tested by a given test. For example, a project I'm working on has integration tests which are written with Selenium and run in a headless browser. Most of these tests involve users, but most of these tests aren't testing user creation, so I don't care how the users get created. Instead of going through the signup interface using Selenium, I'm just injecting the users into the database directly because that's faster and easier. The one exception is I'm not injecting the users into the database in my signup/login integration tests, because whether the user gets created is part of the feature under test. Of course there are significant overlaps between these two categories, that is to say, there are some unit tests that are also integration tests.
- Ma8ee 3y ago
  It certainly doesn’t bother me, but considering the length of your rant, which I won’t bother to read after your first sentence, you certainly seem to have quite a few hangups.
  - kerkeslager 3y ago
    Not wanting to respond to things you haven't read is a hangup you should have, but don't, apparently. Nothing says "doesn't bother me" like having to tell everyone you aren't bothered.
voxic11 3y ago
The whole point of unit tests is that to write them you have to break the system up into units and that decision is an important one that depends on the problem being solved. If they are not well defined then unit tests force you to do define them.

The problem is that “unit” isn’t well defined. Anything from a method to a service can be considered the unit under test.

by Ma8ee 3y ago

The problem is that “unit” isn’t well defined. Anything from a method to a service can be considered the unit under test.

necovek 3y ago

I think it's actually quite simple to come to a common language for unit and integration tests if you start from "integration" instead: if there is any sort of integration (two things interacting: your test setup can tell you that's happening), it's an integration test. If there are no integrations, it's a unit test. As in, it's all in the name. How does one fit a whole "service" (if we are talking something like a networking service like web API server there and not a DDD-style service) there really beats me, but I am happy to hear you out. I'd note that things like setting up a client and then configuring it to access a service already fails that test, even if you hide that complexity into fixtures or "external" test setup. Edit: for avoidance of any doubt, these are terms without clear, widely accepted definitions. I am mostly sharing how I am making what I think is a useful distinction to drive me to write properly scoped tests and structured code.

Ma8ee 3y ago
> if there is any sort of integration (two things interacting: your test setup can tell you that's happening), it's an integration test. Two things of what? There are people that claim that as soon as there are more than one method involved, it's an integration test, and more common, that as soon as there are more than one class involved, it is. This lead to absurd mocking of utility classes to make sure that only one thing is tested at the time. And then, of course, the tests become very fragile. And while there are situations when it is useful to only test one specific thing, what we really care about is the behaviour of whole system of many parts. We mock (fake or stub) to make the tests faster, repeatable and reliable. That is, what we don't want in our tests are slow calls, over slow networks or lengthy calculations. We want the calls to be repeatable, so we don't want to call, say, a database where the state changes between test runs. And we don't want to make calls to unreliable services; unreliable because they are under development or because they are over an unreliable network.
- necovek 3y ago
  I would be in the camp that says two methods calling each other are an integration, but I avoid mocking as much as possible. But if you need to use mocking there heavily, your code is not structured well: look for functional style and acyclic directed graph structure of calls.

jart 3y ago

At Google they draw the distinction based on i/o. Anything that does i/o wouldn't be considered a unit test by them. Unit tests are usually for algorithms, math functions, data structure classes, and stuff like that. Most developers have never needed to write any of those things because they're abstracted by modern languages, which is why I suspect the terminology confuses some people.

randomdata 3y ago
Unit testing in the realm of software originated from the Smalltalk world, where a unit described the test suite for a class. That, with some liberty around the use of class in languages that are not class-centric, reflects the vast, vast, vast majority of tests I see out there in the wild. The reality is that unit testing swept the world and has become synonymous with testing. The days of yore when things like capture and replay testing were it are behind us. Today, it is uncommon for anyone to write anything other than unit tests (as originally defined). And that's why "unit testing" has become confusing. Its original meaning no longer serves a purpose. Today, we just call it "testing", leaving those who wish to continue to use "unit" for the sake of fashion to invent pet definitions; definitions not shared by anyone else.
kerkeslager 3y ago
That's a useful distinction for creating a shared language within Google, and it might be a useful distinction for your team too. But it's definitely not the only way to divide things up. The important thing is that the terminology your term uses is useful for your team. One project I'm working on has two kinds of tests: unit tests run within the code and test classes, state transitions, etc. by calling the code directly, while integration tests are run in Selenium in a headless browser from the perspective of the user. Some of the unit tests on models do hit the DB (i.e. they do I/O).

kerkeslager 3y ago

That's not a problem, that's a hangup you have. We don't need a global definition of what a unit is, we just need to know what unit is being tested by a given unit test. This is important because it's how we understand coverage: 1. We need to understand what's within the unit (covered by the test) so we don't write redundant tests, and... 2. ...we need to understand what's not within the unit (not covered by the test) so we can write separate tests to cover that unit. The unit can be anything, as long as we know what the unit we're testing when we read/write/run the test. An integration test also needs to know what it's testing, but now you're looking at it from an outside perspective. You no longer care which specific areas of code (units) are within the integration, you just care what features are within the integration. Sometimes a refactor will mean that feature A no longer causes class B to call class C: that probably means changes to B's unit tests, because the code unit changed, but it doesn't mean changes to the integration test because the feature didn't change. The important thing is to understand what's being tested by a given test. For example, a project I'm working on has integration tests which are written with Selenium and run in a headless browser. Most of these tests involve users, but most of these tests aren't testing user creation, so I don't care how the users get created. Instead of going through the signup interface using Selenium, I'm just injecting the users into the database directly because that's faster and easier. The one exception is I'm not injecting the users into the database in my signup/login integration tests, because whether the user gets created is part of the feature under test. Of course there are significant overlaps between these two categories, that is to say, there are some unit tests that are also integration tests.

Ma8ee 3y ago
It certainly doesn’t bother me, but considering the length of your rant, which I won’t bother to read after your first sentence, you certainly seem to have quite a few hangups.
- kerkeslager 3y ago
  Not wanting to respond to things you haven't read is a hangup you should have, but don't, apparently. Nothing says "doesn't bother me" like having to tell everyone you aren't bothered.

voxic11 3y ago

The whole point of unit tests is that to write them you have to break the system up into units and that decision is an important one that depends on the problem being solved. If they are not well defined then unit tests force you to do define them.