3 ms·

From zero tests in 2019, to some 59k+ tests in just 4-5 years. On an established, significantly large product no less. Yeah I have zero faith in the quality and

by Jenk 3y ago

From zero tests in 2019, to some 59k+ tests in just 4-5 years. On an established, significantly large product no less. Yeah I have zero faith in the quality and efficacy of those tests.

MetaWhirledPeas 3y ago
Yeah that's quite a few tests. I just assumed the author was summing individual assertions, or unit tests, or something.
- alkonaut 3y ago
  The article is scarce on details about the size of the project and team. If the team is 100 people then having 10k tests written per year seems pretty normal.
hipadev23 3y ago
But hey they have 100% coverage and lots of green text in their CLI. Surely they're shipping!
evilduck 3y ago
In my own experience, post-facto automated testing just locks in the codebase to behave exactly as it did before the tests. The benefit is preventing the introduction of new regressions unless they performed significant refactors along the way. And 59k tests is an average of 32 new tests added every single day of the week for 5 years straight. While it's easy to hit a couple dozen a day as an IC on the first month of an effort like this, the long tail is the hard part. Adding 32 new and valuable tests in a single day in year three of this effort sounds hard. I'd be curious to hear about the manpower and logistical requirements behind this post. How many LoC was the original codebase? What's the current coverage at 59k? How many people were involved? How did they convince upper management to suddenly allocate this much opex spend for 5 years running, or what were they doing before this increase in testing expenditure?
- The_Colonel 3y ago
  > In my own experience, post-facto automated testing just locks in the codebase to behave exactly as it did before the tests That's a great starting point for refactoring or just normal feature development. You need the confidence that your changes won't break the system in unforeseen ways.
  - cybrox 3y ago
    Yes, I (as well as the commenter above, I assume) just doubt that the sheer number of tests will do this in a useful manner. If you have a giant bunch of code that has been written without testability in mind, adding tons of tests mostly just locks down the interfaces of every single segment of code, making refactoring really hard to a point where the tests are useless because they have to be rewritten as well anyways. Good test are written for good code that is written with testability in mind. In that case, only the relevant interfaces are proplery tested and not every little internal thing, which allows for proper refactoring and expansion without fiddling with the tests. This however also yields a lot fewer tests, which is why I doubt this large number of tests can be high quality and/or beneficial.
    - et1337 3y ago
      I tend to agree with you, although I'd say most of the tests are pretty "high quality" (for the most part they don't rely on global state and they run in parallel, randomized order). You pretty much perfectly described our current situation. There is a push to crystallize everything down to the last function. It's not uncommon to have one change result in hundreds of test failures.
- hinkley 3y ago
  > post-facto automated testing just locks in the codebase Some of us call those pinning tests. It's pretty on the nose for your complaints, since it encodes a sense of FOMO right into the name.
- wavemode 3y ago
  > post-facto automated testing just locks in the codebase to behave exactly as it did before the tests What, in your opinion, is the purpose of automated tests?

by Jenk 3y ago

From zero tests in 2019, to some 59k+ tests in just 4-5 years. On an established, significantly large product no less. Yeah I have zero faith in the quality and efficacy of those tests.

MetaWhirledPeas 3y ago

Yeah that's quite a few tests. I just assumed the author was summing individual assertions, or unit tests, or something.

alkonaut 3y ago
The article is scarce on details about the size of the project and team. If the team is 100 people then having 10k tests written per year seems pretty normal.

hipadev23 3y ago

But hey they have 100% coverage and lots of green text in their CLI. Surely they're shipping!

evilduck 3y ago

In my own experience, post-facto automated testing just locks in the codebase to behave exactly as it did before the tests. The benefit is preventing the introduction of new regressions unless they performed significant refactors along the way. And 59k tests is an average of 32 new tests added every single day of the week for 5 years straight. While it's easy to hit a couple dozen a day as an IC on the first month of an effort like this, the long tail is the hard part. Adding 32 new and valuable tests in a single day in year three of this effort sounds hard. I'd be curious to hear about the manpower and logistical requirements behind this post. How many LoC was the original codebase? What's the current coverage at 59k? How many people were involved? How did they convince upper management to suddenly allocate this much opex spend for 5 years running, or what were they doing before this increase in testing expenditure?

The_Colonel 3y ago
> In my own experience, post-facto automated testing just locks in the codebase to behave exactly as it did before the tests That's a great starting point for refactoring or just normal feature development. You need the confidence that your changes won't break the system in unforeseen ways.
- cybrox 3y ago
  Yes, I (as well as the commenter above, I assume) just doubt that the sheer number of tests will do this in a useful manner. If you have a giant bunch of code that has been written without testability in mind, adding tons of tests mostly just locks down the interfaces of every single segment of code, making refactoring really hard to a point where the tests are useless because they have to be rewritten as well anyways. Good test are written for good code that is written with testability in mind. In that case, only the relevant interfaces are proplery tested and not every little internal thing, which allows for proper refactoring and expansion without fiddling with the tests. This however also yields a lot fewer tests, which is why I doubt this large number of tests can be high quality and/or beneficial.
  - et1337 3y ago
    I tend to agree with you, although I'd say most of the tests are pretty "high quality" (for the most part they don't rely on global state and they run in parallel, randomized order). You pretty much perfectly described our current situation. There is a push to crystallize everything down to the last function. It's not uncommon to have one change result in hundreds of test failures.
hinkley 3y ago
> post-facto automated testing just locks in the codebase Some of us call those pinning tests. It's pretty on the nose for your complaints, since it encodes a sense of FOMO right into the name.
wavemode 3y ago
> post-facto automated testing just locks in the codebase to behave exactly as it did before the tests What, in your opinion, is the purpose of automated tests?