4 ms·

Going to dispute this one: > The value of a CI/CD Pipeline is inversely proportional to how long the pipeline takes to run. Priority #1 is : test your product

Going to dispute this one: > The value of a CI/CD Pipeline is inversely proportional to how long the pipeline takes to run. Priority #1 is : test your product properly. This means "if it is supposed to do something, then you should have a test to check that it does that thing". Sometimes it just takes some time to put together and run such tests. In my experience some people (usually my perception is they are younger, more impatient people) who have read somewhere that tests should run really quickly, will argue that Priority #1 above doesn't matter, in the interest of their patience being assuaged. Long ago we had product builds that took hours and tests that often ran overnight, and sometimes over days. And we shipped sh*t that worked!

aliasxneo 4y ago
I suspect this is the result of an overbearing importance on cycle time. I know a colleague who is founding a company solely based on reducing CI cycle time because business owners seem to view it as a critical measurement.
mjr00 4y ago
The author isn't saying that long-running tests aren't valuable, though. They're asking the question: how much more valuable would they be if they ran in minutes instead of days? Slow CI/CD and slow releases are a big problem because they change your entire workflow. Critical bugfix that needs to be shipped? If your CI/CD takes minutes, not a problem: fix it, test it, ship it and have a 5 whys. If it takes days, well, you either have a side-channel process where all your senior engineers have to gather around and speculate about the impact of the change, then you have them sign off, and get the VP to sign off to approve the hotfix, then hope for the best. Or you just wait for days while your customers complain on Twitter. Tests should, ideally, run really quickly. Not at the expense of correctness or completeness, of course, but in most cases I've seen, slow tests happen because nobody bothers optimizing test code, not because they're inherently slow. Sometimes CI/CD is slow for even dumber reasons. At one place I was at (1000+ SV-funded hot startup in the mid-10s), tests got relatively slow enough (~40 min) that VPs got involved and a "test speed task force" with a mandate of reducing CI/CD time to less than 20 minutes. This was announced at a company all-hands with a ton of fanfare. In the first week, one of the engineers did some quick glancing at the logs and noticed that the very old build scripts were still hardcoded to deploy to two servers which no longer existed... and the DNS resolution for those hosts was taking 10 minutes to time out. Needless to say, the task force accomplished its goal, though in a bit of a hollow victory.
- bb88 4y ago
  The bottom line is that developers experience higher cognitive load (and therefore more stress) when writing code in untested/undertested code bases. This is where burnout starts. Unit tests and testing suites ideally will lighten cognitive load and stress, while increasing confidence. Instead of fretting about the unknown sides effects of the code you are writing, you can be confident the unit tests will flag it if something goes wrong. Arguably any "hotfix" (other than 3rd party zero day security patches) required is a testing escape, and ideally those should be rare. And usually there's a subset of tests that can give you 95% to 99% confidence the hotfix will work.
  - mjr00 4y ago
    > The bottom line is that developers experience higher cognitive load (and therefore more stress) when writing code in untested/undertested code bases. This is where burnout starts. Unit tests and testing suites ideally will lighten cognitive load and stress, while increasing confidence. > Instead of fretting about the quality of the code your writing, you can be confident the unit tests will flag it if something goes wrong. I agree with all this. I'm not in any way saying you should do less testing because it makes CI/CD faster. I'm saying that there's a 95%[0] chance your tests are taking hours or days because they are doing something suboptimal that can be improved. Generally, it's a combination of 1. lack of overall test ownership and structure -- for example, every team maintaining their own UI integration tests and repeating an expensive login process unnecessarily, or repeatedly writing new test data to a database rather than working with an existing dataset where possible, or (worst of all) relying on shared resources like databases and servers rather than having ephemeral environments per-branch, resulting in queueing of tests; 2. people who write tests not giving a damn, and this not-giving-a-damnness accumulating over time. If you sneak in a unit test that takes 300ms when it should have taken 10ms into the build, it's likely nobody will notice. If you have a suite of 1000 tests and everyone does that, your test suite now takes 5 minutes instead of 10 seconds; 3. management and engineering leadership also not giving a damn unless it becomes really untenable. Usually devs just grumble about a slow build and nobody does anything, especially in larger corps where there's less feeling of ownership. It's very hard to have a business justification as to why you would pay people to investigate speeding up tests. By the time the test suite is taking 4+ hours, it feels "easier" to just add processes to mitigate this, like making the build server run nightly, rather than address the fundamental issue of tests being slow. [0] There are likely specific domains where this is untrue, but rarely in the generic "CRUD software with background data processing & storage" case.
lisasays 4y ago
Long ago we had product builds that took hours and tests that often ran overnight, and sometimes over days. And I miss those days as well. I think the author's point was not a wholesale dismissal of the value of extensive testing, as such. But rather - whole point of a CI/CD pipeline is that you're supposed to get the feedback ... reasonably fast, right? That's what the "C" is supposed to stand for, pretty much (along with adjacent matters of being explicitly tied to your commit/merge process). So they aren't saying there's no value of all that testing -- but rather, if it takes hours and hours (when it was supposed to be nearly synchronous with your commit process) -- its values does attenuate. Or if you're spending resources making it faster - rather than figuring out what's wrong with your code or models in the first place - that's another tax to pay.
rickette 4y ago
100% this. The number of discussions I've had about the duration of a build/deploy pipeline... Like you said: having proper tests is prio #1, execution speed is second. Great so your build finishes in 3 minutes but you have no idea if the product works. While my build takes over 30 minutes but I fully confident the product works. Which one would you rather have.

aliasxneo 4y ago

I suspect this is the result of an overbearing importance on cycle time. I know a colleague who is founding a company solely based on reducing CI cycle time because business owners seem to view it as a critical measurement.

mjr00 4y ago

The author isn't saying that long-running tests aren't valuable, though. They're asking the question: how much more valuable would they be if they ran in minutes instead of days? Slow CI/CD and slow releases are a big problem because they change your entire workflow. Critical bugfix that needs to be shipped? If your CI/CD takes minutes, not a problem: fix it, test it, ship it and have a 5 whys. If it takes days, well, you either have a side-channel process where all your senior engineers have to gather around and speculate about the impact of the change, then you have them sign off, and get the VP to sign off to approve the hotfix, then hope for the best. Or you just wait for days while your customers complain on Twitter. Tests should, ideally, run really quickly. Not at the expense of correctness or completeness, of course, but in most cases I've seen, slow tests happen because nobody bothers optimizing test code, not because they're inherently slow. Sometimes CI/CD is slow for even dumber reasons. At one place I was at (1000+ SV-funded hot startup in the mid-10s), tests got relatively slow enough (~40 min) that VPs got involved and a "test speed task force" with a mandate of reducing CI/CD time to less than 20 minutes. This was announced at a company all-hands with a ton of fanfare. In the first week, one of the engineers did some quick glancing at the logs and noticed that the very old build scripts were still hardcoded to deploy to two servers which no longer existed... and the DNS resolution for those hosts was taking 10 minutes to time out. Needless to say, the task force accomplished its goal, though in a bit of a hollow victory.

bb88 4y ago
The bottom line is that developers experience higher cognitive load (and therefore more stress) when writing code in untested/undertested code bases. This is where burnout starts. Unit tests and testing suites ideally will lighten cognitive load and stress, while increasing confidence. Instead of fretting about the unknown sides effects of the code you are writing, you can be confident the unit tests will flag it if something goes wrong. Arguably any "hotfix" (other than 3rd party zero day security patches) required is a testing escape, and ideally those should be rare. And usually there's a subset of tests that can give you 95% to 99% confidence the hotfix will work.
- mjr00 4y ago
  > The bottom line is that developers experience higher cognitive load (and therefore more stress) when writing code in untested/undertested code bases. This is where burnout starts. Unit tests and testing suites ideally will lighten cognitive load and stress, while increasing confidence. > Instead of fretting about the quality of the code your writing, you can be confident the unit tests will flag it if something goes wrong. I agree with all this. I'm not in any way saying you should do less testing because it makes CI/CD faster. I'm saying that there's a 95%[0] chance your tests are taking hours or days because they are doing something suboptimal that can be improved. Generally, it's a combination of 1. lack of overall test ownership and structure -- for example, every team maintaining their own UI integration tests and repeating an expensive login process unnecessarily, or repeatedly writing new test data to a database rather than working with an existing dataset where possible, or (worst of all) relying on shared resources like databases and servers rather than having ephemeral environments per-branch, resulting in queueing of tests; 2. people who write tests not giving a damn, and this not-giving-a-damnness accumulating over time. If you sneak in a unit test that takes 300ms when it should have taken 10ms into the build, it's likely nobody will notice. If you have a suite of 1000 tests and everyone does that, your test suite now takes 5 minutes instead of 10 seconds; 3. management and engineering leadership also not giving a damn unless it becomes really untenable. Usually devs just grumble about a slow build and nobody does anything, especially in larger corps where there's less feeling of ownership. It's very hard to have a business justification as to why you would pay people to investigate speeding up tests. By the time the test suite is taking 4+ hours, it feels "easier" to just add processes to mitigate this, like making the build server run nightly, rather than address the fundamental issue of tests being slow. [0] There are likely specific domains where this is untrue, but rarely in the generic "CRUD software with background data processing & storage" case.

lisasays 4y ago

Long ago we had product builds that took hours and tests that often ran overnight, and sometimes over days. And I miss those days as well. I think the author's point was not a wholesale dismissal of the value of extensive testing, as such. But rather - whole point of a CI/CD pipeline is that you're supposed to get the feedback ... reasonably fast, right? That's what the "C" is supposed to stand for, pretty much (along with adjacent matters of being explicitly tied to your commit/merge process). So they aren't saying there's no value of all that testing -- but rather, if it takes hours and hours (when it was supposed to be nearly synchronous with your commit process) -- its values does attenuate. Or if you're spending resources making it faster - rather than figuring out what's wrong with your code or models in the first place - that's another tax to pay.

rickette 4y ago

100% this. The number of discussions I've had about the duration of a build/deploy pipeline... Like you said: having proper tests is prio #1, execution speed is second. Great so your build finishes in 3 minutes but you have no idea if the product works. While my build takes over 30 minutes but I fully confident the product works. Which one would you rather have.