33 ms·

> What do you mean "if not parallelized"? Are you running tests sequentially and then complaining about how long they take to run? Some CI runners are single c

> What do you mean "if not parallelized"? Are you running tests sequentially and then complaining about how long they take to run? Some CI runners are single core unless you pay more, some tests are very hard to parallelise (integration, E2E), some test code bases are large and unwieldy and would require a lot of cost to improve to allow parallelisation. > Please explain exactly how you managed to put together a test suite that costs up to 50€ to run. If you need to run a hefty windows runner on GitHub it’s $0.54 per minute, so a 90 minute full suite will cost $50. You could be better with different tests running at PR/Nightly but if quality is a known issue some orgs will push to run total test coverage each PR (saw this at a large finance company).

lihaoyi 2y ago
Notably Github Actions runners are about 4x more expensive than AWS on demand, and windows is also more expensive than Linux. 50USD gets you about 960 linux core-hours on AWS. A lot more than 90 minutes on 1 box on Windows/GHA, but not so much that someone running a lot of unit/integration/e2e tests in a large codebase won't be able to use
motorest 2y ago
> Some CI runners are single core unless you pay more I don't know which hypothetical CI runner you have in mind. Back in reality, most CICD services bill you on time spent running tests, not "core". Even core count is reflected as runtime multpliers. Moreover, services like GitHub not only offer a baseline of minutes per month, and on top of that provide support for self-hosted runners that cost you nothing. > some tests are very hard to parallelise (integration, E2E) No, that's simply false and fundamentally wrong. You can run any test on any scenario independently of any other test from another scenario. You need to go way out of your way to screw up a test suite so badly you have dependencies between tests covering different sets of features. > If you need to run a hefty windows runner on GitHub it’s $0.54 per minute, so a 90 minute full suite will cost $50. This argument just proves you're desperately grasping at straws. The only Windows test runner from GitHub that costs that is their largest, most expensive runner: Windows 64-core runner. Do you need a Threadripper to use the app you're testing? Even so, that hypothetical cost is only factored in if you run out of minutes from your plan, and you certainly do not order Windows 64-core runners from your free tier plan. Even in your hypothetical cost-conscious scenario, GitHub actions do support self-hosted runners, which cost zero per minute. > You could be better with different tests running at PR/Nightly (...) None of the hypothetical scenarios you fabricated pose a concern. Even assuming you need a Windows machine with 64 processor cores to do an E2E test run of your app, the most basic intro to automated testing tutorial and test pyramids mentions how these tests can and often run on deployments to pre-prod and prod stages. This means that anyone following the most basic intro tutorial on the topic will manage to sidestep any of your hypothetical scenarios by gating auto promotions to prod.
- TypingOutBugs 2y ago
  > I don't know which hypothetical CI runner you have in mind. Back in reality, most CICD services bill you on time spent running tests, not "core". Even core count is reflected as runtime multpliers. If you’re charging a runtime multiplier per core then there’s a cost per core. Included minutes on most runners are limited to basic versions with limited cores. Try xdist pytest on a default GitHub runner and get any speed up… > Moreover, services like GitHub not only offer a baseline of minutes per month, and on top of that provide support for self-hosted runners that cost you nothing. Except the cost of the hardware, sysadmin time for setting up and supporting ephemeral runners, monitoring, etc… > No, that's simply false and fundamentally wrong. You can run any test on any scenario independently of any other test from another scenario. In end to end system tests if you have 100 hitting at the same time how do you guarantee the underlying state is the same so the tests are idempotent? Also another example, I set up testing pipelines for an OS that ran in an FPGA in a HIL CI test. I had three of these due to operating costs. How could I parallelise tests that required flashing firmware AND have the most pipelines running as possible?

lihaoyi 2y ago

Notably Github Actions runners are about 4x more expensive than AWS on demand, and windows is also more expensive than Linux. 50USD gets you about 960 linux core-hours on AWS. A lot more than 90 minutes on 1 box on Windows/GHA, but not so much that someone running a lot of unit/integration/e2e tests in a large codebase won't be able to use

motorest 2y ago

> Some CI runners are single core unless you pay more I don't know which hypothetical CI runner you have in mind. Back in reality, most CICD services bill you on time spent running tests, not "core". Even core count is reflected as runtime multpliers. Moreover, services like GitHub not only offer a baseline of minutes per month, and on top of that provide support for self-hosted runners that cost you nothing. > some tests are very hard to parallelise (integration, E2E) No, that's simply false and fundamentally wrong. You can run any test on any scenario independently of any other test from another scenario. You need to go way out of your way to screw up a test suite so badly you have dependencies between tests covering different sets of features. > If you need to run a hefty windows runner on GitHub it’s $0.54 per minute, so a 90 minute full suite will cost $50. This argument just proves you're desperately grasping at straws. The only Windows test runner from GitHub that costs that is their largest, most expensive runner: Windows 64-core runner. Do you need a Threadripper to use the app you're testing? Even so, that hypothetical cost is only factored in if you run out of minutes from your plan, and you certainly do not order Windows 64-core runners from your free tier plan. Even in your hypothetical cost-conscious scenario, GitHub actions do support self-hosted runners, which cost zero per minute. > You could be better with different tests running at PR/Nightly (...) None of the hypothetical scenarios you fabricated pose a concern. Even assuming you need a Windows machine with 64 processor cores to do an E2E test run of your app, the most basic intro to automated testing tutorial and test pyramids mentions how these tests can and often run on deployments to pre-prod and prod stages. This means that anyone following the most basic intro tutorial on the topic will manage to sidestep any of your hypothetical scenarios by gating auto promotions to prod.

TypingOutBugs 2y ago
> I don't know which hypothetical CI runner you have in mind. Back in reality, most CICD services bill you on time spent running tests, not "core". Even core count is reflected as runtime multpliers. If you’re charging a runtime multiplier per core then there’s a cost per core. Included minutes on most runners are limited to basic versions with limited cores. Try xdist pytest on a default GitHub runner and get any speed up… > Moreover, services like GitHub not only offer a baseline of minutes per month, and on top of that provide support for self-hosted runners that cost you nothing. Except the cost of the hardware, sysadmin time for setting up and supporting ephemeral runners, monitoring, etc… > No, that's simply false and fundamentally wrong. You can run any test on any scenario independently of any other test from another scenario. In end to end system tests if you have 100 hitting at the same time how do you guarantee the underlying state is the same so the tests are idempotent? Also another example, I set up testing pipelines for an OS that ran in an FPGA in a HIL CI test. I had three of these due to operating costs. How could I parallelise tests that required flashing firmware AND have the most pipelines running as possible?