3 ms·

The premise of this article sounds an awful lot like a solution desperately searching for a problem. The scenario used to drive this idea is a slippery slope f

The premise of this article sounds an awful lot like a solution desperately searching for a problem. The scenario used to drive this idea is a slippery slope fallacy that tests can take over an hour to run after years. That's rather dubious, but still this leaves out the fact that tests can be run in parallel. In fact, that is also a necessary condition of selective testing. So why bother with introducing with yet more complexity? To make matters worse if your project grows so large that your hypothetical tests take over an hour to run, it sounds like the project would already be broken down into modules. That, alone, already allows tests to only run if a specific part of the project run. So it's clear that test run time is not a problem that justifies throwing complexity to solve it. Excluding the test runtime argument, is there anything at all that justifies this?

mhlakhani 2y ago
In large mono-repos, like this one is presumably targeting, running all tests in the repo for a given PR would take years (maybe even decades/centuries) of compute time. You have to do some level of test selection, and there are full time engineers who just work on optimizing this. The test runtime argument is the main one IMO. (source: while I did not work on this at a prior job, I worked closely with the team that did this work).
- motorest 2y ago
  > In large mono-repos, like this one is presumably targeting, running all tests in the repo for a given PR would take years (maybe even decades/centuries) of compute time. No, not really. That's a silly example. Your assertion makes as much sense as arguing that your monorepo would take years to build because all code is tracked by the same repo. It doesn't it, does it? Why? How you store your code has nothing to do with how you build it.
  - danpalmer 2y ago
    Being a monorepo is typically about more than just how code is stored, it leads to very different practices about dependency management, building, etc. On the monorepo I work on, the fact it is a monorepo is intrinsically linked to the build, testing, and deployment processes in many ways. The ideal is that your build system by-necessity contains the data to be able to selectively test – typically the case if you're linking code in some way. You import a library, now your tests get run if that library changes. As the article suggests, this breaks down over service boundaries, but as you suggest, you still hopefully have modules you can link up. The problem is when you have hundreds of services, maintaining those dependencies manually could be hard. When you have thousands it may be nearly impossible. When you have hundreds of thousands it may be impossible. I think applying ML to that problem so that you can incrementally understand the ever changing dependencies across services. I can also assure you that however smart the build system is, there will always be spooky action at a distance between components.
    - mhlakhani 2y ago
      yeah. I am talking about repos with hundreds of thousands (probably millions) of tests. Here's just one example of the scale: https://engineering.fb.com/2018/11/21/developer-tools/predictive-test-selection/ https://engineering.fb.com/2018/11/21/developer-tools/predic... if you change a low level library that's the equivalent of the C++ standard library and you want to test the changes, you effectively have to rebuild the world. And you don't want to.
      - danpalmer 2y ago
        Exactly, or when "one" build rule connecting a service to another is actually a hundred deeply nested build rules doing a lot more work, and every one of those code paths would need to correctly convey whether a dependency is required at the test level or not.
lihaoyi 2y ago
> So it's clear that test run time is not a problem that justifies throwing complexity to solve it. There's a lot I can respond to in this post, but I think the bottom line is: if you have not experienced the problem, count your blessings. Lots of places do face these problems, with test suites that take hours or days to run if not parallelized. And while parallelization reduces latency, it does not reduce costs, and test suites taking 10, 20, or 50USD every time you update a pull request are not uncommon either If you never hit these scenarios, just know that many others are not so lucky
- motorest 2y ago
  > There's a lot I can respond to in this post, but I think the bottom line is: if you have not experienced the problem, count your blessings. You don't even specify what problem is it. Again, this is a solution searching for a problem, and one you can't even describe. > Lots of places do face these problems, with test suites that take hours or days to run if not parallelized. What do you mean "if not parallelized"? Are you running tests sequentially and then complaining about how long they take to run? I won't even touch on the red flag which is the apparent lack of modularization. > And while parallelization reduces latency, it does not reduce costs, and test suites taking 10, 20, or 50USD every time you update a pull request are not uncommon either Please explain exactly how you managed to put together a test suite that costs up to 50€ to run. I assure you the list of problems and red flags you state along the way will never even feature selective testing as a factor or as a solution.
  - mhlakhani 2y ago
    > Please explain exactly how you managed to put together a test suite that costs up to 50€ to run. I'm not OP but have worked with them: have you considered a repo that might have tents of thousands of committers over decades? It's very easy to just have an insane amount of tests.
  - TypingOutBugs 2y ago
    > What do you mean "if not parallelized"? Are you running tests sequentially and then complaining about how long they take to run? Some CI runners are single core unless you pay more, some tests are very hard to parallelise (integration, E2E), some test code bases are large and unwieldy and would require a lot of cost to improve to allow parallelisation. > Please explain exactly how you managed to put together a test suite that costs up to 50€ to run. If you need to run a hefty windows runner on GitHub it’s $0.54 per minute, so a 90 minute full suite will cost $50. You could be better with different tests running at PR/Nightly but if quality is a known issue some orgs will push to run total test coverage each PR (saw this at a large finance company).
csomar 2y ago
How does modules solve this problem? If you changed Module A, don't you need to also test every component that depends on that module to account for any possible regression? > The scenario used to drive this idea is a slippery slope fallacy that tests can take over an hour to run after years. It doesn't really take much for a project to grow to one hour for tests to run. If you have a CI/CD pipeline that executes tests on every commit and you have 30-40 commits daily, that's 30-40 hours. Still, I'd rather setup a full 2 machines to run the tests than rather have to deal with selective testing. It might make sense if you have hundreds of commits per day.
atq2119 2y ago
> To make matters worse if your project grows so large that your hypothetical tests take over an hour to run, it sounds like the project would already be broken down into modules. Story time so that you can revisit your assumptions. Imagine your product is a graphics driver. Graphics APIs have extensive test suites with millions of individual tests. Running them serially typically takes many hours, depending on the target hardware. But over the years you invariably also run across bugs exposed by real applications that the conformance suites don't catch. So, you also accumulate additional tests, some of them distilled versions of those triggers, some of them captured frames with known good "golden" output pictures. Those add further to the test runtime. Then, mostly due to performance pressures, there are many options that can affect the precise details of how the driver runs. Many of them are heuristics auto-tuned, but the conformance suite is unlikely to hit all the heuristic cases, so really you should also run all your tests with overrides for the heuristic. Now you have a combinatorial explosion that means the space of tests you really ought to run is at least in the quadrillions. It's simply infeasible to run all tests on every PR, so what tends to happen in practice is that a manually curated subset of tests is run on every commit, and then more thorough testing happens on various asynchronous schedules (e.g. on release branches, daily on the development branch). I'm not convinced that the article is the solution, but it could be part of one. New ideas in this space are certainly welcome.

mhlakhani 2y ago

In large mono-repos, like this one is presumably targeting, running all tests in the repo for a given PR would take years (maybe even decades/centuries) of compute time. You have to do some level of test selection, and there are full time engineers who just work on optimizing this. The test runtime argument is the main one IMO. (source: while I did not work on this at a prior job, I worked closely with the team that did this work).

motorest 2y ago
> In large mono-repos, like this one is presumably targeting, running all tests in the repo for a given PR would take years (maybe even decades/centuries) of compute time. No, not really. That's a silly example. Your assertion makes as much sense as arguing that your monorepo would take years to build because all code is tracked by the same repo. It doesn't it, does it? Why? How you store your code has nothing to do with how you build it.
- danpalmer 2y ago
  Being a monorepo is typically about more than just how code is stored, it leads to very different practices about dependency management, building, etc. On the monorepo I work on, the fact it is a monorepo is intrinsically linked to the build, testing, and deployment processes in many ways. The ideal is that your build system by-necessity contains the data to be able to selectively test – typically the case if you're linking code in some way. You import a library, now your tests get run if that library changes. As the article suggests, this breaks down over service boundaries, but as you suggest, you still hopefully have modules you can link up. The problem is when you have hundreds of services, maintaining those dependencies manually could be hard. When you have thousands it may be nearly impossible. When you have hundreds of thousands it may be impossible. I think applying ML to that problem so that you can incrementally understand the ever changing dependencies across services. I can also assure you that however smart the build system is, there will always be spooky action at a distance between components.
  - mhlakhani 2y ago
    yeah. I am talking about repos with hundreds of thousands (probably millions) of tests. Here's just one example of the scale: https://engineering.fb.com/2018/11/21/developer-tools/predictive-test-selection/ https://engineering.fb.com/2018/11/21/developer-tools/predic... if you change a low level library that's the equivalent of the C++ standard library and you want to test the changes, you effectively have to rebuild the world. And you don't want to.
    - danpalmer 2y ago
      Exactly, or when "one" build rule connecting a service to another is actually a hundred deeply nested build rules doing a lot more work, and every one of those code paths would need to correctly convey whether a dependency is required at the test level or not.

lihaoyi 2y ago

> So it's clear that test run time is not a problem that justifies throwing complexity to solve it. There's a lot I can respond to in this post, but I think the bottom line is: if you have not experienced the problem, count your blessings. Lots of places do face these problems, with test suites that take hours or days to run if not parallelized. And while parallelization reduces latency, it does not reduce costs, and test suites taking 10, 20, or 50USD every time you update a pull request are not uncommon either If you never hit these scenarios, just know that many others are not so lucky

motorest 2y ago
> There's a lot I can respond to in this post, but I think the bottom line is: if you have not experienced the problem, count your blessings. You don't even specify what problem is it. Again, this is a solution searching for a problem, and one you can't even describe. > Lots of places do face these problems, with test suites that take hours or days to run if not parallelized. What do you mean "if not parallelized"? Are you running tests sequentially and then complaining about how long they take to run? I won't even touch on the red flag which is the apparent lack of modularization. > And while parallelization reduces latency, it does not reduce costs, and test suites taking 10, 20, or 50USD every time you update a pull request are not uncommon either Please explain exactly how you managed to put together a test suite that costs up to 50€ to run. I assure you the list of problems and red flags you state along the way will never even feature selective testing as a factor or as a solution.
- mhlakhani 2y ago
  > Please explain exactly how you managed to put together a test suite that costs up to 50€ to run. I'm not OP but have worked with them: have you considered a repo that might have tents of thousands of committers over decades? It's very easy to just have an insane amount of tests.
- TypingOutBugs 2y ago
  > What do you mean "if not parallelized"? Are you running tests sequentially and then complaining about how long they take to run? Some CI runners are single core unless you pay more, some tests are very hard to parallelise (integration, E2E), some test code bases are large and unwieldy and would require a lot of cost to improve to allow parallelisation. > Please explain exactly how you managed to put together a test suite that costs up to 50€ to run. If you need to run a hefty windows runner on GitHub it’s $0.54 per minute, so a 90 minute full suite will cost $50. You could be better with different tests running at PR/Nightly but if quality is a known issue some orgs will push to run total test coverage each PR (saw this at a large finance company).

csomar 2y ago

How does modules solve this problem? If you changed Module A, don't you need to also test every component that depends on that module to account for any possible regression? > The scenario used to drive this idea is a slippery slope fallacy that tests can take over an hour to run after years. It doesn't really take much for a project to grow to one hour for tests to run. If you have a CI/CD pipeline that executes tests on every commit and you have 30-40 commits daily, that's 30-40 hours. Still, I'd rather setup a full 2 machines to run the tests than rather have to deal with selective testing. It might make sense if you have hundreds of commits per day.

atq2119 2y ago

> To make matters worse if your project grows so large that your hypothetical tests take over an hour to run, it sounds like the project would already be broken down into modules. Story time so that you can revisit your assumptions. Imagine your product is a graphics driver. Graphics APIs have extensive test suites with millions of individual tests. Running them serially typically takes many hours, depending on the target hardware. But over the years you invariably also run across bugs exposed by real applications that the conformance suites don't catch. So, you also accumulate additional tests, some of them distilled versions of those triggers, some of them captured frames with known good "golden" output pictures. Those add further to the test runtime. Then, mostly due to performance pressures, there are many options that can affect the precise details of how the driver runs. Many of them are heuristics auto-tuned, but the conformance suite is unlikely to hit all the heuristic cases, so really you should also run all your tests with overrides for the heuristic. Now you have a combinatorial explosion that means the space of tests you really ought to run is at least in the quadrillions. It's simply infeasible to run all tests on every PR, so what tends to happen in practice is that a manually curated subset of tests is run on every commit, and then more thorough testing happens on various asynchronous schedules (e.g. on release branches, daily on the development branch). I'm not convinced that the article is the solution, but it could be part of one. New ideas in this space are certainly welcome.