4 ms·

This is interesting, but I see a few issues with it: - Maintainability is difficult to quantify, and often subjective. It's also easy to fall into a trap of ov

by imiric 2y ago

This is interesting, but I see a few issues with it: - Maintainability is difficult to quantify, and often subjective. It's also easy to fall into a trap of overoptimizing or DRYing test code in the pursuit of improving maintainability, and actually end up doing the opposite. Striking a balance is important in this case, which takes many years of experience to get a feel for. - I interpret the chart to mean that unit tests have high maintainability, i.e. it's a good thing, when that is often not the case. If anything, unit tests are the most brittle and susceptible to low-level changes. This is good since they're your first safety net, but it also means that you spend a lot of time changing them. Considering you should have many unit tests, a lot of maintenance work is spent on them. I see the reverse for E2E tests as well. They're easier to maintain, since typically the high-level interfaces don't change as often, and you have fewer of them. But most importantly, I don't see how these definitions help me write better tests, or choose what to focus on. We all know that using fewer resources is better, but that will depend on what you're testing. Nobody likes flaky tests, but telling me that unit tests are more reliable than integration tests won't help me write better code. What I would like to see instead are concrete suggestions on how to improve each of these categories, regardless of the test type. For example, not relying on time or sleeping in tests is always good to minimize flakiness. Similarly for relying on system resources like the disk or network; that should be done almost exclusively by E2E and integration tests, and avoided (mocked) in unit tests. There should also be more discussion about what it takes to make code testable to begin with. TDD helps with this, but you don't need to practice it to the letter if you keep some design principles in mind while you're writing code that will make it easier to test later. I've seen many attempts at displacing the traditional test pyramid over the years, but so far it's been the most effective guiding tool in all projects I've worked on. The struggle that most projects experience with tests stems primarily from not following its basic principles.

stoperaticless 2y ago
> I don't see how these definitions help me write better tests > What I would like to see instead … If you hire me, I can address those needs.
joshuamorton 2y ago
> If anything, unit tests are the most brittle and susceptible to low-level changes. I don't find this to be the case if the unit tests are precise (which they should be). That is, if you are writing non-flaky unit tests which do all the "right" unit-testy things (using fakes/dependency injecting well and so isolating and testing only the unit under test), you should end up with a set of tests that - Fails only when you change the file/component the test relates to - Isn't flaky (can be run ~10000 times without failing) - Is quick (you can do the 10000 run loop above approximately interactively, in a few minutes, by running in parallel saturating a beefy workstation) This compares to integration/e2e tests which inherently break due to other systems and unrelated assumptions changing (sometimes legitimate, sometimes not), and can have rates of flakyness of 1-10% due to the inherent nature of "real" systems failing to start occasionally and the inherently longer test-debug cycle that makes fixing issues more painful (root causing bug that causes a test to fail 1% of the time is much easier when the test takes .3 CPU-seconds than when it takes 30 or 300 CPU-seconds). Very few tests I see are actually unit tests in the above sense, many people only write integration tests because the code under test is structured in inherently un- or difficult- to test ways.
- imiric 2y ago
  That's true. I probably used brittle in the wrong sense there. What I mean is that after any code change that isn't a strict refactoring you will inevitably have to touch one or more unit tests. If you're adding new functionality, you need to test different code paths; if you change existing functionality, you need to update the related test(s); if you're fixing a bug, you need to add a unit test that reproduces it, and so on. All this means is that unit tests take the most effort to maintain, so I'm not sure why the article claims they have "high" maintainability, or that that's a good thing. In contrast, higher-level tests usually require less maintenance, assuming they're stable and not flaky, since you're not touching them as often. > Very few tests I see are actually unit tests in the above sense, many people only write integration tests because the code under test is structured in inherently un- or difficult- to test ways. Very true. I think the recent popularization of alternative guidelines like The Testing Trophy is precisely because developers are too lazy to properly structure their code to make pure unit testing possible, and see the work of maintaining unit tests as too much of a chore, so they make up arguments that there's no value in them. This couldn't be farther from the truth, and is IMO an irresponsible way to approach software development.
js8 2y ago
> There should also be more discussion about what it takes to make code testable to begin with. IME, testable pretty much just means referentially transparent.
- imiric 2y ago
  Sure. But how do you achieve that in practice? All functions can't be pure, and at some point you need to handle messy I/O and deal with nondeterminism. How you structure your code in the best way to do that, while also ensuring you can write useful tests? None of this is trivial.
  - js8 2y ago
    After I wrote my comment, I realized I was actually somewhat wrong (I have been in this debate, on the anti-TDD side, for quite some time now, but it was kind of hard to figure out why for me, but I think I finally got it). Testability is more than just the referential transparency, RT it is a necessary but not sufficient condition for testability. > But how do you achieve that in practice? The way to achieve referential transparency is, in my mind, functional programming. Specifically, use monads to model side effects. You can model any computer system as a composition of lambda terms exchanging data (specific subset of terms) monadically, so in theory, this can be achieved. So you can imagine any program as a tree of functions, each builds a more complex function as a composition of smaller, simpler functions, until the whole program is put together in the main() function. However, I need to add two other conditions that for a program to be testable: Each function to which you decompose your program has to (2) be reasonably short (has to have limited number of compositions) and (3) has to have a clear specification, based on which it can be determined, whether the function in itself is correct. The condition (2) is strictly speaking not required, but because we are humans with limited ability to understand, we want it to help us create (3). Now I believe that the more you have RT and (3), your program is more testable. This is because that testing is pretty much just partial type-checking by sampling - you create some sample values of the type you expect, and you verify that the program produces expected values. The advantage of sampling is that you don't have to formally specify your types, so you don't need complete formal specification. The conditions RT and (3) are pretty much necessary if you want to properly type your programs (for example, we could specify every function using a type in dependent type theory). So testable (is a spectrum) really means "close to type-checkable" (which is a binary). I however need to address a misconception (which I had), the types that we assign to functions (i.e. the specification) do not come from the program itself, but rather from the domain knowledge, which we expect to impart into the program. Literally, types are (or can be) the specification. And by the way, the condition (2) determines how small are the units of the program you can be testing. Now after the above setup, let me get to the main point, which I will call testability tradeoff: The conditions (2) and (3) are mutually exclusive for some programs, i.e. there is a tradeoff between (2) making units small and (3) giving them a good (easy to interpret) specification. Let me give some extreme examples of testability tradeoffs for different programs to illustrate this concept. Library of functions has usually only little testability tradeoff, because most functions are independent of each other, so each of them (on the API level) can satisfy both (2) and (3). On the other end of spectrum you have things like a trained neural network or program that implements a tax code - even if you can decompose those programs into little pieces to satisfy condition (2), it is not possible to then assign these pieces a meaning sensible enough to construct useful tests per condition (3). Such programs are simply not understandable in detail (or better to say, we don't know how to understand them). The hidden assumption of TDD folks (and proponents of massive unit testing, in the test pyramid) is that we can always convert program as much to have (2) and (3), i.e. in their view, the testability tradeoff can be always made as low as needed. But I don't think this is true in practice, and I have given examples above - we have complex useful programs that cannot be decomposed to little pieces, where each of the little pieces can be meaningfully specified in the business domain. Such programs, I claim, cannot be effectively unit tested, and can only be e2e or integration tested. (However, they can be type-checked against the full specification.) So because (as stated above) testability of a program is pretty much your ability to meaningfully assign (expected) types and typecheck, now I think that TDD proponents, when they talk about testability, want to have as much specification as possible. Which is kind of funny, because they started as a kind of opposition to that idea. Ah well, paradoxes of life.. Anyway, I know my response is a bit messy, but hopefully I explained the main idea enough so it will make more sense on rereading.

by imiric 2y ago

stoperaticless 2y ago

> I don't see how these definitions help me write better tests > What I would like to see instead … If you hire me, I can address those needs.

joshuamorton 2y ago

> If anything, unit tests are the most brittle and susceptible to low-level changes. I don't find this to be the case if the unit tests are precise (which they should be). That is, if you are writing non-flaky unit tests which do all the "right" unit-testy things (using fakes/dependency injecting well and so isolating and testing only the unit under test), you should end up with a set of tests that - Fails only when you change the file/component the test relates to - Isn't flaky (can be run ~10000 times without failing) - Is quick (you can do the 10000 run loop above approximately interactively, in a few minutes, by running in parallel saturating a beefy workstation) This compares to integration/e2e tests which inherently break due to other systems and unrelated assumptions changing (sometimes legitimate, sometimes not), and can have rates of flakyness of 1-10% due to the inherent nature of "real" systems failing to start occasionally and the inherently longer test-debug cycle that makes fixing issues more painful (root causing bug that causes a test to fail 1% of the time is much easier when the test takes .3 CPU-seconds than when it takes 30 or 300 CPU-seconds). Very few tests I see are actually unit tests in the above sense, many people only write integration tests because the code under test is structured in inherently un- or difficult- to test ways.

imiric 2y ago
That's true. I probably used brittle in the wrong sense there. What I mean is that after any code change that isn't a strict refactoring you will inevitably have to touch one or more unit tests. If you're adding new functionality, you need to test different code paths; if you change existing functionality, you need to update the related test(s); if you're fixing a bug, you need to add a unit test that reproduces it, and so on. All this means is that unit tests take the most effort to maintain, so I'm not sure why the article claims they have "high" maintainability, or that that's a good thing. In contrast, higher-level tests usually require less maintenance, assuming they're stable and not flaky, since you're not touching them as often. > Very few tests I see are actually unit tests in the above sense, many people only write integration tests because the code under test is structured in inherently un- or difficult- to test ways. Very true. I think the recent popularization of alternative guidelines like The Testing Trophy is precisely because developers are too lazy to properly structure their code to make pure unit testing possible, and see the work of maintaining unit tests as too much of a chore, so they make up arguments that there's no value in them. This couldn't be farther from the truth, and is IMO an irresponsible way to approach software development.

js8 2y ago

> There should also be more discussion about what it takes to make code testable to begin with. IME, testable pretty much just means referentially transparent.

imiric 2y ago
Sure. But how do you achieve that in practice? All functions can't be pure, and at some point you need to handle messy I/O and deal with nondeterminism. How you structure your code in the best way to do that, while also ensuring you can write useful tests? None of this is trivial.
- js8 2y ago
  After I wrote my comment, I realized I was actually somewhat wrong (I have been in this debate, on the anti-TDD side, for quite some time now, but it was kind of hard to figure out why for me, but I think I finally got it). Testability is more than just the referential transparency, RT it is a necessary but not sufficient condition for testability. > But how do you achieve that in practice? The way to achieve referential transparency is, in my mind, functional programming. Specifically, use monads to model side effects. You can model any computer system as a composition of lambda terms exchanging data (specific subset of terms) monadically, so in theory, this can be achieved. So you can imagine any program as a tree of functions, each builds a more complex function as a composition of smaller, simpler functions, until the whole program is put together in the main() function. However, I need to add two other conditions that for a program to be testable: Each function to which you decompose your program has to (2) be reasonably short (has to have limited number of compositions) and (3) has to have a clear specification, based on which it can be determined, whether the function in itself is correct. The condition (2) is strictly speaking not required, but because we are humans with limited ability to understand, we want it to help us create (3). Now I believe that the more you have RT and (3), your program is more testable. This is because that testing is pretty much just partial type-checking by sampling - you create some sample values of the type you expect, and you verify that the program produces expected values. The advantage of sampling is that you don't have to formally specify your types, so you don't need complete formal specification. The conditions RT and (3) are pretty much necessary if you want to properly type your programs (for example, we could specify every function using a type in dependent type theory). So testable (is a spectrum) really means "close to type-checkable" (which is a binary). I however need to address a misconception (which I had), the types that we assign to functions (i.e. the specification) do not come from the program itself, but rather from the domain knowledge, which we expect to impart into the program. Literally, types are (or can be) the specification. And by the way, the condition (2) determines how small are the units of the program you can be testing. Now after the above setup, let me get to the main point, which I will call testability tradeoff: The conditions (2) and (3) are mutually exclusive for some programs, i.e. there is a tradeoff between (2) making units small and (3) giving them a good (easy to interpret) specification. Let me give some extreme examples of testability tradeoffs for different programs to illustrate this concept. Library of functions has usually only little testability tradeoff, because most functions are independent of each other, so each of them (on the API level) can satisfy both (2) and (3). On the other end of spectrum you have things like a trained neural network or program that implements a tax code - even if you can decompose those programs into little pieces to satisfy condition (2), it is not possible to then assign these pieces a meaning sensible enough to construct useful tests per condition (3). Such programs are simply not understandable in detail (or better to say, we don't know how to understand them). The hidden assumption of TDD folks (and proponents of massive unit testing, in the test pyramid) is that we can always convert program as much to have (2) and (3), i.e. in their view, the testability tradeoff can be always made as low as needed. But I don't think this is true in practice, and I have given examples above - we have complex useful programs that cannot be decomposed to little pieces, where each of the little pieces can be meaningfully specified in the business domain. Such programs, I claim, cannot be effectively unit tested, and can only be e2e or integration tested. (However, they can be type-checked against the full specification.) So because (as stated above) testability of a program is pretty much your ability to meaningfully assign (expected) types and typecheck, now I think that TDD proponents, when they talk about testability, want to have as much specification as possible. Which is kind of funny, because they started as a kind of opposition to that idea. Ah well, paradoxes of life.. Anyway, I know my response is a bit messy, but hopefully I explained the main idea enough so it will make more sense on rereading.