9 ms·

I still like one of the defining characteristics of Unit Tests (paraphrasing Michael Feathers from memory): they are fast and cheap to run. Sure, they might not

by teeray 3y ago

I still like one of the defining characteristics of Unit Tests (paraphrasing Michael Feathers from memory): they are fast and cheap to run. Sure, they might not perfectly simulate production like integration tests, but they also don’t take hours burning cash in cloud infrastructure while risking failure from unrelated races dealing with those dependencies. You can use Unit Tests to get to a place where you’re fairly confident that the integration tests will pass (making that whole expensive affair cheaper to run).

Cthulhu_ 3y ago
That's exactly it; QA is a layered / tiered / pyramid shaped process, the more you catch lower down, the less reliance there is on the upper layers, and the faster the development iterations.
gbacon 3y ago
From Working Effectively With Legacy Code by Feathers, p. 14[0]: Unit tests run fast. If they don’t run fast, they aren’t unit tests. Other kinds of tests often masquerade as unit tests. A test is not a unit test if: 1. It talks to a database. 2. It communicates across a network. 3. It touches the file system. 4. You have to do special things to your environment (such as editing configuration files) to run it. Tests that do these things aren’t bad. Often they are worth writing, and you generally will write them in unit test harnesses. However, it is important to be able to separate them from true unit tests so that you can keep a set of tests that you can run fast whenever you make changes. [0]: https://www.google.com/books/edition/Working_Effectively_with_Legacy_Code/fB6s_Z6g0gIC?hl=en&gbpv=1&pg=PA14 https://www.google.com/books/edition/Working_Effectively_wit...
- brightball 3y ago
  That's a great rule of thumb.
- atticora 3y ago
  My "unit tests" do hit the database and file system, and I have found and fixed many many problems during testing by doing so. I have found many other problems with those calls in production when I didn't do so. Yes, they make testing a lot slower. Our main app takes around 40 minutes to build which isn't good. I'd like it to be faster. But writing a bunch of separate integration tests to cover those functions would be a steep price. I can understand reasonable people choosing either approach.
  - berkes 3y ago
    > My "unit tests" do hit the database and file system, and I have found and fixed many many problems during testing by doing so. I have found many other problems with those calls in production when I didn't do so. No-one said that integration tests can't also be very valuable. From the little context I get that you write integration tests, and that is fine. They are useful, valuable! But they are not unit-tests. edit: on re-reading, I get the feeling that for you "integration tests" are a synonym for "end to end tests". But -at least in most literature- end-to-end tests are a kind of integration-test. But not all integration tests are end-to-end tests. In my software, I'll often have integration tests that swap out some adapter (e.g. the postgres-users-repository, for the memory-users-repository, or fake-users-repository. Or the test-payment for the stripe-payment) but that still test a lot of stuff stacked on top of each-other. Integration tests, just not integration tests that test the entire integration.
    - lokar 3y ago
      And integration tests can also be fast
      - alemanek 3y ago
        Test containers really help with this. Should still have the big system tests that run overnight but a set of integration tests using Test Containers to stand in for the infrastructure dependencies is awesome. My team has a ton of those and they run inside a reasonable time frame (5min or so) but we still allow for excluding those from test runs so you can run just the unit tests.
        darkteflon 3y ago
        I hadn’t heard of Test Containers[1], but it looks really useful - thanks for the rec. [1] https://testcontainers.com/ https://testcontainers.com/
      - berkes 3y ago
        Indeed! It's one of the reasons I like the adapter pattern (aka hexagonal architecture) so much. Data flowing through some 100 classes and 300 conditionals then into a `memory-payments` and back takes mere milliseconds. "Memory payments" is then some silly wrapper around a hashmap with the same API as the full-blown production payments-adapter that calls stripe over HTTP. Or the same api as the "production adapter" that wraps some RDBMS running on the other end of the data-center.
- deleted 3y ago
  [deleted]
- randomdata 3y ago
  In other words, unit tests – unless, perhaps, you count an empty test as being a unit test – do not exist. "Talking" to structured sets of data, i.e. a database, is fundamental to computing.
- jcranmer 3y ago
  I'd actually quibble a lot on this definition--if you want to unit test code that needs to do any of the first three things, well, you have to do them to test the code. I would say that a test for a network protocol that works by spinning up an echo server on a random port and having the code connect to that echo server is still a unit test for that network protocol. In my definition, it would still be a unit test if it is fast and it talks to a database, a network server, or the file system, so long as such communication is done entirely in a "mock" fashion (it only does such communication as the test sets up, and it's done in such a fashion that tests can be run in parallel with no issues).
- dllthomas 3y ago
  I've never liked the conflating of target size and test time constraints. I very much agree that there's benefit to considering the pace of feedback and where it falls in your workflow; immediate feedback is hugely valuable, but it can come from unit tests, other tests, or things which are not tests. Meanwhile, some tests of a single unit might have to take a long time. Exhaustive tests are rarely applicable, but when they are it's going to be for something small and it's likely to be slow to run. That should not be in your tightest loop, but it is probably clearer to evict it simply because it is slow, rather than because it is not a unit test for being slow.
- closeparen 3y ago
  It is pretty excruciating (and IMO useless) to write true, DB-isolated unit tests for DB access layer code.
- physicles 3y ago
  I haven't found the distinction between unit tests and non-unit tests to be that useful in practice. The important questions are: 1. Is it kinda slow? (The test suite for a single module should run in a few seconds; a large monorepo should finish in under a minute) 2. Is there network access involved? (Could the test randomly fail?) 3. Do I need to set up anything special, like a database? (How easy is it for a new developer to run it?) If the answer to any of those is Yes, then your test might fall in the liminal space between unit tests and integration tests -- they're unit-level tests, but they're more expensive to run. For example, data access layer tests that run against an actual database. On the other hand, even if a test touches the filesystem, then it's generally fast enough that you don't have to worry about it (and you did make the test self-cleaning, right?) -- calling that test "not a unit test" doesn't help you. Likewise, if the database you're touching is sqlite, then that still leaves you with No's to the three questions above.
- _dain_ 3y ago
  >1. It talks to a database. >3. It touches the file system. These are BS. Maybe they made sense in the beforetimes when when we didn't have Docker containers or SSDs, but nowadays there's no reason you can't stand up a mini test database as part of your unit test suite. It's way simpler than mocking.
  - icedchai 3y ago
    100% this. A guy I work with just rebuilt our CI/CD pipeline and we're spinning up a database and all dependent services in containers. There are no mocks and it works great. In previous lives, I worked on tests that mocked everything. We spent more time creating and maintaining mocks than writing the actual tests.
    - nerdponx 3y ago
      I think those are still considered "integration" tests in the traditional way of thinking. The problem is that a lot of applications don't do much other than interact with external resources, to the point where isolated unit tests are rare or only cover non-critical code, while just about any substantive test is an "integration" test.
      - icedchai 3y ago
        Yes, you are correct. I never cared much for the traditional way of thinking about tests. I've had more arguments with people claiming this is an integration test, not a unit test, write some mocks, etc. Outside of very specific pieces of code, you generally get more value from integration tests.
- munksbeer 3y ago
  I've dropped my archaic thinking on what constitutes a unit test or an integration test. I now seldom write what most people consider unit tests, in the "one class, one test" sense. Instead, I classify units of logical coherence and write unit tests for those. I write financial trading systems - not hft, but still latency sensitive. I will test an order through our pipeline as a unit of work. This will necessarily touch multiple classes. So as an example, a test will cover orders which are accepted and then filled, orders which are rejected, and so on. Many people would classify these as integration tests, and to be fair, I don't really care what you name them. To me these are much more valuable than the traditional "one class, one test" mechanism because it means I am free to refactor the internals of our pipeline as much as I want with very low impact on the test code. One of the whole points of test code, that I think has been lost, is that it should be there to give you confidence in the correctness of your application under change. Writing "one class, one test" is a bad way to achieve this.
BeetleB 3y ago
> they are fast and cheap to run. But expensive to write. Especially if you want them to be fast and cheap to run.
AtlasBarfed 3y ago
THey are fast and cheap WHEN YOU FIRST WRITE THE CODE. Or, if you are the original author of the code. The problem is, and there will be people that disagree with me, is that unit tests make refactoring of other people's code a lot harder. STAY WITH ME! If the unit tests were "good" and helped document what the code does, then they don't. You won't believe this, but in dogmatic high-breadth-coverage (low depth coverage), there are tons of test code that is SO TIED TO IMPLEMENTATION rather than interface than any monkeying of the presumed encapsulated logic breaks the unit tests, so you have double the things to fix. You'll never believe what happens next. Some developer in some Agile thing that got assigned 2 unicorn shits for the task panics because the unit tests are SERIOUSLY slowing down his "velocity". So what does he do? Delete tests, change tests to make them work at any costs.

by teeray 3y ago

Cthulhu_ 3y ago

That's exactly it; QA is a layered / tiered / pyramid shaped process, the more you catch lower down, the less reliance there is on the upper layers, and the faster the development iterations.

gbacon 3y ago

From Working Effectively With Legacy Code by Feathers, p. 14[0]: Unit tests run fast. If they don’t run fast, they aren’t unit tests. Other kinds of tests often masquerade as unit tests. A test is not a unit test if: 1. It talks to a database. 2. It communicates across a network. 3. It touches the file system. 4. You have to do special things to your environment (such as editing configuration files) to run it. Tests that do these things aren’t bad. Often they are worth writing, and you generally will write them in unit test harnesses. However, it is important to be able to separate them from true unit tests so that you can keep a set of tests that you can run fast whenever you make changes. [0]: https://www.google.com/books/edition/Working_Effectively_with_Legacy_Code/fB6s_Z6g0gIC?hl=en&gbpv=1&pg=PA14 https://www.google.com/books/edition/Working_Effectively_wit...

brightball 3y ago
That's a great rule of thumb.
atticora 3y ago
My "unit tests" do hit the database and file system, and I have found and fixed many many problems during testing by doing so. I have found many other problems with those calls in production when I didn't do so. Yes, they make testing a lot slower. Our main app takes around 40 minutes to build which isn't good. I'd like it to be faster. But writing a bunch of separate integration tests to cover those functions would be a steep price. I can understand reasonable people choosing either approach.
- berkes 3y ago
  > My "unit tests" do hit the database and file system, and I have found and fixed many many problems during testing by doing so. I have found many other problems with those calls in production when I didn't do so. No-one said that integration tests can't also be very valuable. From the little context I get that you write integration tests, and that is fine. They are useful, valuable! But they are not unit-tests. edit: on re-reading, I get the feeling that for you "integration tests" are a synonym for "end to end tests". But -at least in most literature- end-to-end tests are a kind of integration-test. But not all integration tests are end-to-end tests. In my software, I'll often have integration tests that swap out some adapter (e.g. the postgres-users-repository, for the memory-users-repository, or fake-users-repository. Or the test-payment for the stripe-payment) but that still test a lot of stuff stacked on top of each-other. Integration tests, just not integration tests that test the entire integration.
  - lokar 3y ago
    And integration tests can also be fast
    - alemanek 3y ago
      Test containers really help with this. Should still have the big system tests that run overnight but a set of integration tests using Test Containers to stand in for the infrastructure dependencies is awesome. My team has a ton of those and they run inside a reasonable time frame (5min or so) but we still allow for excluding those from test runs so you can run just the unit tests.
      - darkteflon 3y ago
        I hadn’t heard of Test Containers[1], but it looks really useful - thanks for the rec. [1] https://testcontainers.com/ https://testcontainers.com/
    - berkes 3y ago
      Indeed! It's one of the reasons I like the adapter pattern (aka hexagonal architecture) so much. Data flowing through some 100 classes and 300 conditionals then into a `memory-payments` and back takes mere milliseconds. "Memory payments" is then some silly wrapper around a hashmap with the same API as the full-blown production payments-adapter that calls stripe over HTTP. Or the same api as the "production adapter" that wraps some RDBMS running on the other end of the data-center.
deleted 3y ago
[deleted]
randomdata 3y ago
In other words, unit tests – unless, perhaps, you count an empty test as being a unit test – do not exist. "Talking" to structured sets of data, i.e. a database, is fundamental to computing.
jcranmer 3y ago
I'd actually quibble a lot on this definition--if you want to unit test code that needs to do any of the first three things, well, you have to do them to test the code. I would say that a test for a network protocol that works by spinning up an echo server on a random port and having the code connect to that echo server is still a unit test for that network protocol. In my definition, it would still be a unit test if it is fast and it talks to a database, a network server, or the file system, so long as such communication is done entirely in a "mock" fashion (it only does such communication as the test sets up, and it's done in such a fashion that tests can be run in parallel with no issues).
dllthomas 3y ago
I've never liked the conflating of target size and test time constraints. I very much agree that there's benefit to considering the pace of feedback and where it falls in your workflow; immediate feedback is hugely valuable, but it can come from unit tests, other tests, or things which are not tests. Meanwhile, some tests of a single unit might have to take a long time. Exhaustive tests are rarely applicable, but when they are it's going to be for something small and it's likely to be slow to run. That should not be in your tightest loop, but it is probably clearer to evict it simply because it is slow, rather than because it is not a unit test for being slow.
closeparen 3y ago
It is pretty excruciating (and IMO useless) to write true, DB-isolated unit tests for DB access layer code.
physicles 3y ago
I haven't found the distinction between unit tests and non-unit tests to be that useful in practice. The important questions are: 1. Is it kinda slow? (The test suite for a single module should run in a few seconds; a large monorepo should finish in under a minute) 2. Is there network access involved? (Could the test randomly fail?) 3. Do I need to set up anything special, like a database? (How easy is it for a new developer to run it?) If the answer to any of those is Yes, then your test might fall in the liminal space between unit tests and integration tests -- they're unit-level tests, but they're more expensive to run. For example, data access layer tests that run against an actual database. On the other hand, even if a test touches the filesystem, then it's generally fast enough that you don't have to worry about it (and you did make the test self-cleaning, right?) -- calling that test "not a unit test" doesn't help you. Likewise, if the database you're touching is sqlite, then that still leaves you with No's to the three questions above.
_dain_ 3y ago
>1. It talks to a database. >3. It touches the file system. These are BS. Maybe they made sense in the beforetimes when when we didn't have Docker containers or SSDs, but nowadays there's no reason you can't stand up a mini test database as part of your unit test suite. It's way simpler than mocking.
- icedchai 3y ago
  100% this. A guy I work with just rebuilt our CI/CD pipeline and we're spinning up a database and all dependent services in containers. There are no mocks and it works great. In previous lives, I worked on tests that mocked everything. We spent more time creating and maintaining mocks than writing the actual tests.
  - nerdponx 3y ago
    I think those are still considered "integration" tests in the traditional way of thinking. The problem is that a lot of applications don't do much other than interact with external resources, to the point where isolated unit tests are rare or only cover non-critical code, while just about any substantive test is an "integration" test.
    - icedchai 3y ago
      Yes, you are correct. I never cared much for the traditional way of thinking about tests. I've had more arguments with people claiming this is an integration test, not a unit test, write some mocks, etc. Outside of very specific pieces of code, you generally get more value from integration tests.
munksbeer 3y ago
I've dropped my archaic thinking on what constitutes a unit test or an integration test. I now seldom write what most people consider unit tests, in the "one class, one test" sense. Instead, I classify units of logical coherence and write unit tests for those. I write financial trading systems - not hft, but still latency sensitive. I will test an order through our pipeline as a unit of work. This will necessarily touch multiple classes. So as an example, a test will cover orders which are accepted and then filled, orders which are rejected, and so on. Many people would classify these as integration tests, and to be fair, I don't really care what you name them. To me these are much more valuable than the traditional "one class, one test" mechanism because it means I am free to refactor the internals of our pipeline as much as I want with very low impact on the test code. One of the whole points of test code, that I think has been lost, is that it should be there to give you confidence in the correctness of your application under change. Writing "one class, one test" is a bad way to achieve this.

BeetleB 3y ago

> they are fast and cheap to run. But expensive to write. Especially if you want them to be fast and cheap to run.

AtlasBarfed 3y ago

THey are fast and cheap WHEN YOU FIRST WRITE THE CODE. Or, if you are the original author of the code. The problem is, and there will be people that disagree with me, is that unit tests make refactoring of other people's code a lot harder. STAY WITH ME! If the unit tests were "good" and helped document what the code does, then they don't. You won't believe this, but in dogmatic high-breadth-coverage (low depth coverage), there are tons of test code that is SO TIED TO IMPLEMENTATION rather than interface than any monkeying of the presumed encapsulated logic breaks the unit tests, so you have double the things to fix. You'll never believe what happens next. Some developer in some Agile thing that got assigned 2 unicorn shits for the task panics because the unit tests are SERIOUSLY slowing down his "velocity". So what does he do? Delete tests, change tests to make them work at any costs.