4 ms·

> So it's clear that test run time is not a problem that justifies throwing complexity to solve it. There's a lot I can respond to in this post, but I think th

> So it's clear that test run time is not a problem that justifies throwing complexity to solve it. There's a lot I can respond to in this post, but I think the bottom line is: if you have not experienced the problem, count your blessings. Lots of places do face these problems, with test suites that take hours or days to run if not parallelized. And while parallelization reduces latency, it does not reduce costs, and test suites taking 10, 20, or 50USD every time you update a pull request are not uncommon either If you never hit these scenarios, just know that many others are not so lucky

motorest 2y ago
> There's a lot I can respond to in this post, but I think the bottom line is: if you have not experienced the problem, count your blessings. You don't even specify what problem is it. Again, this is a solution searching for a problem, and one you can't even describe. > Lots of places do face these problems, with test suites that take hours or days to run if not parallelized. What do you mean "if not parallelized"? Are you running tests sequentially and then complaining about how long they take to run? I won't even touch on the red flag which is the apparent lack of modularization. > And while parallelization reduces latency, it does not reduce costs, and test suites taking 10, 20, or 50USD every time you update a pull request are not uncommon either Please explain exactly how you managed to put together a test suite that costs up to 50€ to run. I assure you the list of problems and red flags you state along the way will never even feature selective testing as a factor or as a solution.
- mhlakhani 2y ago
  > Please explain exactly how you managed to put together a test suite that costs up to 50€ to run. I'm not OP but have worked with them: have you considered a repo that might have tents of thousands of committers over decades? It's very easy to just have an insane amount of tests.
- TypingOutBugs 2y ago
  > What do you mean "if not parallelized"? Are you running tests sequentially and then complaining about how long they take to run? Some CI runners are single core unless you pay more, some tests are very hard to parallelise (integration, E2E), some test code bases are large and unwieldy and would require a lot of cost to improve to allow parallelisation. > Please explain exactly how you managed to put together a test suite that costs up to 50€ to run. If you need to run a hefty windows runner on GitHub it’s $0.54 per minute, so a 90 minute full suite will cost $50. You could be better with different tests running at PR/Nightly but if quality is a known issue some orgs will push to run total test coverage each PR (saw this at a large finance company).
  - lihaoyi 2y ago
    Notably Github Actions runners are about 4x more expensive than AWS on demand, and windows is also more expensive than Linux. 50USD gets you about 960 linux core-hours on AWS. A lot more than 90 minutes on 1 box on Windows/GHA, but not so much that someone running a lot of unit/integration/e2e tests in a large codebase won't be able to use
  - motorest 2y ago
    > Some CI runners are single core unless you pay more I don't know which hypothetical CI runner you have in mind. Back in reality, most CICD services bill you on time spent running tests, not "core". Even core count is reflected as runtime multpliers. Moreover, services like GitHub not only offer a baseline of minutes per month, and on top of that provide support for self-hosted runners that cost you nothing. > some tests are very hard to parallelise (integration, E2E) No, that's simply false and fundamentally wrong. You can run any test on any scenario independently of any other test from another scenario. You need to go way out of your way to screw up a test suite so badly you have dependencies between tests covering different sets of features. > If you need to run a hefty windows runner on GitHub it’s $0.54 per minute, so a 90 minute full suite will cost $50. This argument just proves you're desperately grasping at straws. The only Windows test runner from GitHub that costs that is their largest, most expensive runner: Windows 64-core runner. Do you need a Threadripper to use the app you're testing? Even so, that hypothetical cost is only factored in if you run out of minutes from your plan, and you certainly do not order Windows 64-core runners from your free tier plan. Even in your hypothetical cost-conscious scenario, GitHub actions do support self-hosted runners, which cost zero per minute. > You could be better with different tests running at PR/Nightly (...) None of the hypothetical scenarios you fabricated pose a concern. Even assuming you need a Windows machine with 64 processor cores to do an E2E test run of your app, the most basic intro to automated testing tutorial and test pyramids mentions how these tests can and often run on deployments to pre-prod and prod stages. This means that anyone following the most basic intro tutorial on the topic will manage to sidestep any of your hypothetical scenarios by gating auto promotions to prod.
    - TypingOutBugs 2y ago
- csomar 2y ago
  I can see tests running 50$ if you are running them on Github compute which is x10-x15 times more expensive than AWS. Still, even at 2-3$ that's still quite expensive. Maybe if you have a really large low-level build. I have a really small one and it consumes roughly 10 minutes on Github.

motorest 2y ago

> There's a lot I can respond to in this post, but I think the bottom line is: if you have not experienced the problem, count your blessings. You don't even specify what problem is it. Again, this is a solution searching for a problem, and one you can't even describe. > Lots of places do face these problems, with test suites that take hours or days to run if not parallelized. What do you mean "if not parallelized"? Are you running tests sequentially and then complaining about how long they take to run? I won't even touch on the red flag which is the apparent lack of modularization. > And while parallelization reduces latency, it does not reduce costs, and test suites taking 10, 20, or 50USD every time you update a pull request are not uncommon either Please explain exactly how you managed to put together a test suite that costs up to 50€ to run. I assure you the list of problems and red flags you state along the way will never even feature selective testing as a factor or as a solution.

mhlakhani 2y ago
> Please explain exactly how you managed to put together a test suite that costs up to 50€ to run. I'm not OP but have worked with them: have you considered a repo that might have tents of thousands of committers over decades? It's very easy to just have an insane amount of tests.
TypingOutBugs 2y ago
> What do you mean "if not parallelized"? Are you running tests sequentially and then complaining about how long they take to run? Some CI runners are single core unless you pay more, some tests are very hard to parallelise (integration, E2E), some test code bases are large and unwieldy and would require a lot of cost to improve to allow parallelisation. > Please explain exactly how you managed to put together a test suite that costs up to 50€ to run. If you need to run a hefty windows runner on GitHub it’s $0.54 per minute, so a 90 minute full suite will cost $50. You could be better with different tests running at PR/Nightly but if quality is a known issue some orgs will push to run total test coverage each PR (saw this at a large finance company).
- lihaoyi 2y ago
  Notably Github Actions runners are about 4x more expensive than AWS on demand, and windows is also more expensive than Linux. 50USD gets you about 960 linux core-hours on AWS. A lot more than 90 minutes on 1 box on Windows/GHA, but not so much that someone running a lot of unit/integration/e2e tests in a large codebase won't be able to use
- motorest 2y ago
  > Some CI runners are single core unless you pay more I don't know which hypothetical CI runner you have in mind. Back in reality, most CICD services bill you on time spent running tests, not "core". Even core count is reflected as runtime multpliers. Moreover, services like GitHub not only offer a baseline of minutes per month, and on top of that provide support for self-hosted runners that cost you nothing. > some tests are very hard to parallelise (integration, E2E) No, that's simply false and fundamentally wrong. You can run any test on any scenario independently of any other test from another scenario. You need to go way out of your way to screw up a test suite so badly you have dependencies between tests covering different sets of features. > If you need to run a hefty windows runner on GitHub it’s $0.54 per minute, so a 90 minute full suite will cost $50. This argument just proves you're desperately grasping at straws. The only Windows test runner from GitHub that costs that is their largest, most expensive runner: Windows 64-core runner. Do you need a Threadripper to use the app you're testing? Even so, that hypothetical cost is only factored in if you run out of minutes from your plan, and you certainly do not order Windows 64-core runners from your free tier plan. Even in your hypothetical cost-conscious scenario, GitHub actions do support self-hosted runners, which cost zero per minute. > You could be better with different tests running at PR/Nightly (...) None of the hypothetical scenarios you fabricated pose a concern. Even assuming you need a Windows machine with 64 processor cores to do an E2E test run of your app, the most basic intro to automated testing tutorial and test pyramids mentions how these tests can and often run on deployments to pre-prod and prod stages. This means that anyone following the most basic intro tutorial on the topic will manage to sidestep any of your hypothetical scenarios by gating auto promotions to prod.
  - TypingOutBugs 2y ago
csomar 2y ago
I can see tests running 50$ if you are running them on Github compute which is x10-x15 times more expensive than AWS. Still, even at 2-3$ that's still quite expensive. Maybe if you have a really large low-level build. I have a really small one and it consumes roughly 10 minutes on Github.