14 ms·

The author isn't saying that long-running tests aren't valuable, though. They're asking the question: how much more valuable would they be if they ran in minute

by mjr00 4y ago

The author isn't saying that long-running tests aren't valuable, though. They're asking the question: how much more valuable would they be if they ran in minutes instead of days? Slow CI/CD and slow releases are a big problem because they change your entire workflow. Critical bugfix that needs to be shipped? If your CI/CD takes minutes, not a problem: fix it, test it, ship it and have a 5 whys. If it takes days, well, you either have a side-channel process where all your senior engineers have to gather around and speculate about the impact of the change, then you have them sign off, and get the VP to sign off to approve the hotfix, then hope for the best. Or you just wait for days while your customers complain on Twitter. Tests should, ideally, run really quickly. Not at the expense of correctness or completeness, of course, but in most cases I've seen, slow tests happen because nobody bothers optimizing test code, not because they're inherently slow. Sometimes CI/CD is slow for even dumber reasons. At one place I was at (1000+ SV-funded hot startup in the mid-10s), tests got relatively slow enough (~40 min) that VPs got involved and a "test speed task force" with a mandate of reducing CI/CD time to less than 20 minutes. This was announced at a company all-hands with a ton of fanfare. In the first week, one of the engineers did some quick glancing at the logs and noticed that the very old build scripts were still hardcoded to deploy to two servers which no longer existed... and the DNS resolution for those hosts was taking 10 minutes to time out. Needless to say, the task force accomplished its goal, though in a bit of a hollow victory.

bb88 4y ago
The bottom line is that developers experience higher cognitive load (and therefore more stress) when writing code in untested/undertested code bases. This is where burnout starts. Unit tests and testing suites ideally will lighten cognitive load and stress, while increasing confidence. Instead of fretting about the unknown sides effects of the code you are writing, you can be confident the unit tests will flag it if something goes wrong. Arguably any "hotfix" (other than 3rd party zero day security patches) required is a testing escape, and ideally those should be rare. And usually there's a subset of tests that can give you 95% to 99% confidence the hotfix will work.
- mjr00 4y ago
  > The bottom line is that developers experience higher cognitive load (and therefore more stress) when writing code in untested/undertested code bases. This is where burnout starts. Unit tests and testing suites ideally will lighten cognitive load and stress, while increasing confidence. > Instead of fretting about the quality of the code your writing, you can be confident the unit tests will flag it if something goes wrong. I agree with all this. I'm not in any way saying you should do less testing because it makes CI/CD faster. I'm saying that there's a 95%[0] chance your tests are taking hours or days because they are doing something suboptimal that can be improved. Generally, it's a combination of 1. lack of overall test ownership and structure -- for example, every team maintaining their own UI integration tests and repeating an expensive login process unnecessarily, or repeatedly writing new test data to a database rather than working with an existing dataset where possible, or (worst of all) relying on shared resources like databases and servers rather than having ephemeral environments per-branch, resulting in queueing of tests; 2. people who write tests not giving a damn, and this not-giving-a-damnness accumulating over time. If you sneak in a unit test that takes 300ms when it should have taken 10ms into the build, it's likely nobody will notice. If you have a suite of 1000 tests and everyone does that, your test suite now takes 5 minutes instead of 10 seconds; 3. management and engineering leadership also not giving a damn unless it becomes really untenable. Usually devs just grumble about a slow build and nobody does anything, especially in larger corps where there's less feeling of ownership. It's very hard to have a business justification as to why you would pay people to investigate speeding up tests. By the time the test suite is taking 4+ hours, it feels "easier" to just add processes to mitigate this, like making the build server run nightly, rather than address the fundamental issue of tests being slow. [0] There are likely specific domains where this is untrue, but rarely in the generic "CRUD software with background data processing & storage" case.
pydry 4y ago
>The author isn't saying that long-running tests aren't valuable, though. They kind of are though, since they don't acknowledge the trade offs inherent in reducing the duration of the integration tests. >In most cases I've seen, slow tests happen because nobody bothers optimizing test code IME they typically they come about because realism comes with a price tag. An actual UI is more expensive to load than a mock, an actual webserver is slower than a fake web server, a postgres database spins up slower than an SQLite, a real API call takes longer than a mocked one, etc.
- mjr00 4y ago
  > IME they typically they come about because realism comes with a price tag. An actual UI is more expensive to load than a mock, an actual webserver is slower than a fake web server, a postgres database spins up slower than an SQLite, a real API call takes longer than a mocked one, etc. Sure, but spinning up a postgres database or running a Selenium test should change your test times from being executed in seconds to minutes, not seconds to hours or days. > They kind of are though, since they don't acknowledge the trade offs inherent in reducing the duration of the integration tests. As I say in another comment in this thread, there is very, very, very rarely a legitimate reason for integration tests to need multiple hours to run. Even in the worst case, you can just parallelize the tests on different hardware; if you can't parallelize, that is a problem with your tests and not a real excuse.
  - pydry 4y ago
    I agree that parallelization is often a good way to speed up your CI pipeline, but the OP's suggestion of shifting integration tests to unit tests is frequently the worst thing you can do because it very often changes a test that tests into a test that is just a pale imitation of the code under test - a mimetic test. I worked on a big old ball of mud once where business logic was smeared across 6 or 7 different microservices. The ONLY way to gain confidence that the app did the right thing was to test it end to end which, of course, took ages, and meant hours long CI pipeline (longer if not parallelized). My coworker tried to replace some of those tests with "faster" unit tests around bits of those microservices but all they did was "lock down" the horrible architecture such that it became impossible to refactor and consolidate the business logic without breaking a bunch of unit tests.
rakoo 4y ago
How many times have we complained the state of the infrastructure is bad and we should spend some time to fix it, but business imperatives prevent us ? In your story the team was given time and resources to actually do it, at last. If it wasn't, the problem might have continued for months or years still unnoticed. It's not that hollow a victory.

by mjr00 4y ago

bb88 4y ago

The bottom line is that developers experience higher cognitive load (and therefore more stress) when writing code in untested/undertested code bases. This is where burnout starts. Unit tests and testing suites ideally will lighten cognitive load and stress, while increasing confidence. Instead of fretting about the unknown sides effects of the code you are writing, you can be confident the unit tests will flag it if something goes wrong. Arguably any "hotfix" (other than 3rd party zero day security patches) required is a testing escape, and ideally those should be rare. And usually there's a subset of tests that can give you 95% to 99% confidence the hotfix will work.

mjr00 4y ago
> The bottom line is that developers experience higher cognitive load (and therefore more stress) when writing code in untested/undertested code bases. This is where burnout starts. Unit tests and testing suites ideally will lighten cognitive load and stress, while increasing confidence. > Instead of fretting about the quality of the code your writing, you can be confident the unit tests will flag it if something goes wrong. I agree with all this. I'm not in any way saying you should do less testing because it makes CI/CD faster. I'm saying that there's a 95%[0] chance your tests are taking hours or days because they are doing something suboptimal that can be improved. Generally, it's a combination of 1. lack of overall test ownership and structure -- for example, every team maintaining their own UI integration tests and repeating an expensive login process unnecessarily, or repeatedly writing new test data to a database rather than working with an existing dataset where possible, or (worst of all) relying on shared resources like databases and servers rather than having ephemeral environments per-branch, resulting in queueing of tests; 2. people who write tests not giving a damn, and this not-giving-a-damnness accumulating over time. If you sneak in a unit test that takes 300ms when it should have taken 10ms into the build, it's likely nobody will notice. If you have a suite of 1000 tests and everyone does that, your test suite now takes 5 minutes instead of 10 seconds; 3. management and engineering leadership also not giving a damn unless it becomes really untenable. Usually devs just grumble about a slow build and nobody does anything, especially in larger corps where there's less feeling of ownership. It's very hard to have a business justification as to why you would pay people to investigate speeding up tests. By the time the test suite is taking 4+ hours, it feels "easier" to just add processes to mitigate this, like making the build server run nightly, rather than address the fundamental issue of tests being slow. [0] There are likely specific domains where this is untrue, but rarely in the generic "CRUD software with background data processing & storage" case.

pydry 4y ago

>The author isn't saying that long-running tests aren't valuable, though. They kind of are though, since they don't acknowledge the trade offs inherent in reducing the duration of the integration tests. >In most cases I've seen, slow tests happen because nobody bothers optimizing test code IME they typically they come about because realism comes with a price tag. An actual UI is more expensive to load than a mock, an actual webserver is slower than a fake web server, a postgres database spins up slower than an SQLite, a real API call takes longer than a mocked one, etc.

mjr00 4y ago
> IME they typically they come about because realism comes with a price tag. An actual UI is more expensive to load than a mock, an actual webserver is slower than a fake web server, a postgres database spins up slower than an SQLite, a real API call takes longer than a mocked one, etc. Sure, but spinning up a postgres database or running a Selenium test should change your test times from being executed in seconds to minutes, not seconds to hours or days. > They kind of are though, since they don't acknowledge the trade offs inherent in reducing the duration of the integration tests. As I say in another comment in this thread, there is very, very, very rarely a legitimate reason for integration tests to need multiple hours to run. Even in the worst case, you can just parallelize the tests on different hardware; if you can't parallelize, that is a problem with your tests and not a real excuse.
- pydry 4y ago
  I agree that parallelization is often a good way to speed up your CI pipeline, but the OP's suggestion of shifting integration tests to unit tests is frequently the worst thing you can do because it very often changes a test that tests into a test that is just a pale imitation of the code under test - a mimetic test. I worked on a big old ball of mud once where business logic was smeared across 6 or 7 different microservices. The ONLY way to gain confidence that the app did the right thing was to test it end to end which, of course, took ages, and meant hours long CI pipeline (longer if not parallelized). My coworker tried to replace some of those tests with "faster" unit tests around bits of those microservices but all they did was "lock down" the horrible architecture such that it became impossible to refactor and consolidate the business logic without breaking a bunch of unit tests.

rakoo 4y ago

How many times have we complained the state of the infrastructure is bad and we should spend some time to fix it, but business imperatives prevent us ? In your story the team was given time and resources to actually do it, at last. If it wasn't, the problem might have continued for months or years still unnoticed. It's not that hollow a victory.