3 ms·

Excepting C++ development, is slow build times really a hardware problem? CI software in general is very slow. But even building locally, there are many projec

Excepting C++ development, is slow build times really a hardware problem? CI software in general is very slow. But even building locally, there are many projects that have just ridiculous build times because it's just not a priority at all. I've seen small-medium sized Java projects with a 25 minute test suite. This has been in part due to extremely inefficient tests that keep assembling the entire system over and over each test, often with extreme redundancy in coverage, ensuring you need to keep running the entire suite whenever a test breaks. Nobody is going to run tests for half an hour unless you hold a gun to their head. So you make your change by 10 AM, usually something trivial, and then you spend the afternoon fixing the 30 tests that broke, running the suite over and over to find more broken tests to fix. It's almost comically inefficient.

Aeolun 4y ago
I mean, you spending the afternoon fixing all the tests that broke is a lot more efficient than you spending the night fixing those same issues when it gets deployed to production.
- marginalia_nu 4y ago
  Sure, but I'm not convinced this necessary or sufficient to prevent that. It's rarely obvious what these types of tests actually test. It's like... just a bit of everything for the sake of good coverage. That and the inertia from having to wait half an hour for the test results makes it very compelling to start just "fixing" the tests by changing the assertions to expect the new value. If you have a more sensible testing strategy, it's obvious what is being tested (and thus also relatively clear what broke), and the turnaround is seconds possibly low single-digit minutes, so you don't have to rebuild the mental context every time you run the tests.
  - newaccount74 4y ago
    > "fixing" the tests by changing the assertions to the new value Oh my... I spent hours trying to figure out why I broke a test, only to figure out that my change actually fixed incorrect behavior! The previous dev who worked on the code had "fixed" the test failure by changing the assertion to expect the incorrect behavior! Automatic test suites with thousands of tests sound great in theory, but in practice it's not all rosy...
  - Aeolun 4y ago
    I think I get what you are saying, but in the absence of good tests I still vastly prefer slow, inexact tests to no tests at all. Our integration test suite is absolutely horrid. Like, the thing takes literally 25 hours of real time to run, so it’s split into 50 pieces running in parallel, ultimately taking just under 30 minutes. It has like 3000 unique scenarios and pretty nearly 95% of every scenario is exactly the same. It’s a _huge_ waste. If it breaks, I pretty much only know that it’s broken (often because of some flaky behavior, or timing related thing, not an actual issue). Sometimes pinpointing the issue goes meta and is a matter of looking at which tests are broken, instead of how they fail. But when all those tests succeed, I know that my system is just as stable as it was before. All the major flows are covered, and if it runs successfully, I can be fairly certain my system works. Now imagine dealing with the codebase that leads to such a test suite without it…
    - marginalia_nu 4y ago
      This is a false dichotomy. The alternatives aren't having no tests having a half an hour test suite. As you say, 95% of every scenario is exactly the same, which I think is relatively common in this type of code base. The ideal test time is thus 90 seconds instead of 30 minutes. Maybe that unattainable, but I wager there exists a test suite that has the same coverage but with significantly less redundancy. Test engineering is actually hard. If testing is just some afterthought primarily aimed at coverage with not much consideration put toward performance, what you typically get is redundant tests with abysmal run-times.
EdwardDiego 4y ago
TBH, when you mentioned Java and slow test runs, I was expecting the word "Spring" to be involved.
- marginalia_nu 4y ago
  While Spring does not tend to have stellar test performance, some of the worst offenders I've encountered were using some homebrew framework. I think it's a mode of development more than a framework that's at fault, even though the two tend to correlate.

Aeolun 4y ago

I mean, you spending the afternoon fixing all the tests that broke is a lot more efficient than you spending the night fixing those same issues when it gets deployed to production.

marginalia_nu 4y ago
Sure, but I'm not convinced this necessary or sufficient to prevent that. It's rarely obvious what these types of tests actually test. It's like... just a bit of everything for the sake of good coverage. That and the inertia from having to wait half an hour for the test results makes it very compelling to start just "fixing" the tests by changing the assertions to expect the new value. If you have a more sensible testing strategy, it's obvious what is being tested (and thus also relatively clear what broke), and the turnaround is seconds possibly low single-digit minutes, so you don't have to rebuild the mental context every time you run the tests.
- newaccount74 4y ago
  > "fixing" the tests by changing the assertions to the new value Oh my... I spent hours trying to figure out why I broke a test, only to figure out that my change actually fixed incorrect behavior! The previous dev who worked on the code had "fixed" the test failure by changing the assertion to expect the incorrect behavior! Automatic test suites with thousands of tests sound great in theory, but in practice it's not all rosy...
- Aeolun 4y ago
  I think I get what you are saying, but in the absence of good tests I still vastly prefer slow, inexact tests to no tests at all. Our integration test suite is absolutely horrid. Like, the thing takes literally 25 hours of real time to run, so it’s split into 50 pieces running in parallel, ultimately taking just under 30 minutes. It has like 3000 unique scenarios and pretty nearly 95% of every scenario is exactly the same. It’s a _huge_ waste. If it breaks, I pretty much only know that it’s broken (often because of some flaky behavior, or timing related thing, not an actual issue). Sometimes pinpointing the issue goes meta and is a matter of looking at which tests are broken, instead of how they fail. But when all those tests succeed, I know that my system is just as stable as it was before. All the major flows are covered, and if it runs successfully, I can be fairly certain my system works. Now imagine dealing with the codebase that leads to such a test suite without it…
  - marginalia_nu 4y ago
    This is a false dichotomy. The alternatives aren't having no tests having a half an hour test suite. As you say, 95% of every scenario is exactly the same, which I think is relatively common in this type of code base. The ideal test time is thus 90 seconds instead of 30 minutes. Maybe that unattainable, but I wager there exists a test suite that has the same coverage but with significantly less redundancy. Test engineering is actually hard. If testing is just some afterthought primarily aimed at coverage with not much consideration put toward performance, what you typically get is redundant tests with abysmal run-times.

EdwardDiego 4y ago

TBH, when you mentioned Java and slow test runs, I was expecting the word "Spring" to be involved.

marginalia_nu 4y ago
While Spring does not tend to have stellar test performance, some of the worst offenders I've encountered were using some homebrew framework. I think it's a mode of development more than a framework that's at fault, even though the two tend to correlate.