4 ms·

This article seems to fundamentally misunderstand why the testing pyramid is a pyramid in the first place. You shouldn't have more unit tests because they're e

This article seems to fundamentally misunderstand why the testing pyramid is a pyramid in the first place. You shouldn't have more unit tests because they're easier to write. You should have more unit tests because they're easier to maintain. By definition, they focus on a small part of the software and should not break if an unrelated part of the software changes. Of course, just because each individual part of a machine works, it doesn't mean the machine itself works. That's where integration tests come in. Turns out it's usually fairly easy to make sure all the parts work together once you've verified all of the parts individually, so you usually need fewer integration tests. Once you've verified all the parts work and that they work together correctly, now all you need to do is make sure you actually made the right thing. Enter functional tests and E2E tests. These are usually especially costly to maintain, but they're also so high-level that you usually need very few of them. Writing tests is easy, but they're only useful if you run them. It's not enough for a test to work when you write it - it has to continue working into the future. Maintenance is a recurring cost and, in my experience, is where automated testing can get really expensive. The testing pyramid helps minimize that cost. In an ideal world, you would maximize testing at every level - but in the real world that's often just too expensive.

pydry 6y ago
Unit tests are not intrinsically easier to maintain. The lower level the test the more it reflects implementation rather than specification and hence the more changes it demands when refactoring. The practical upshot of these types of tests is often that they catch very few actual bugs but nonetheless demand constant and expensive maintenance when code is changed.
- Tainnor 6y ago
  This is true if your units under tests are bad abstractions. If you don't know yet what you will build, if your requirements change all the time, good abstractions maybe rarely emerge and then maybe unit tests make less sense. However, I've worked on a number of applications where there were very well-defined units that were quite stable in terms of their abstraction, and then unit tests are a joy. "I have this tree structure and I need to transform it into another structure" and stuff like that. It's also worth keeping in mind that "unit testing" doesn't have to mean "class (or method) testing". Sometimes the class is the right level of abstraction but sometimes it can also be a small module of classes working in concert etc. I agree that tests shouldn't simply reflect implementation, but there are functional requirements that can be specified at a lower level.
  - closeparen 6y ago
    If I design tests to capture the spirit of the class / module, the line coverage is only 50-60%. In Go each method needs a lot of individual attention to exercise all the error return branches up to a 90%+ coverage standard.
  - pydry 6y ago
    I agree. Good, stable abstractions covering complicated logic are where unit tests shine. Unfortunately I find this scenario is also more the exception than the rule. Big ball of mud apps with few effective tests are the norm and apps which do very little complex logic under the hood (focused instead mainly on hooking together services) are also not at all uncommon.
- closeparen 6y ago
  I think of it as double entry bookkeeping. You have to say what you mean two different ways, and those two phrasings have to be reconcilable. This is still a useful guard against careless errors! But it is a different problem from checking whether your software actually does the right thing. The magnitude of change needed in unit tests while changing code is definitely maddening. For example: add an argument to a method. It's called only 3 places in production code, but each of those sites has 10 test cases, so there are 30 mock expectations to update now. And the IDE doesn't understand those so no automated support. On the other hand, integration and end-to-end tests will go red when (seemingly) nothing at all has changed, because of drift in tangentially related components from other codebases. Usually the fixes are simple, but it requires a willingness to drop what you're doing and investigate when they happen. Otherwise they just get ignored.
  - pydry 6y ago
    I think of it as writing everything twice to appease the unit testing gods. If done indiscriminately it really is more of a ritual to ward of the bug gremlins than an effective development practice. Integration tests do suffer from a whole set of problems that unit tests do not but with one fundamental difference: they're usually problems that are soluble with better engineering techniques. Integration tests that go red when something other than the code changes often provide useful feedback, so I wouldn't necessarily rule them out.
Tainnor 6y ago
It's not just that the article misunderstands that unit tests are easier to maintain (if well written at least), but they also don't seem to discuss the problem of combinatorial explosion that plagues end-to-end tests. There are only two ways of avoiding that problem: ignoring it and simply give up on comprehensive testing, or realise that comprehensive testing can only be done by writing mostly unit tests.
btilly 6y ago
I find that unit tests are much less useful than integration tests. When we write unit tests, we ask the person to write the code to spend a lot of work documenting how they expect the system that they are integrating with to work. What happens when that system behaves differently? Or changes in the future? And the way that mocks are set up are very sensitive to implementation details, which is a maintenance burden when you have to change that. As a simple example, I had a function that was loading data into a database with a series of insert statements. I sped it up by several orders of magnitude by using a COPY to send CSV data. Guess how well that worked with the mock database interface in the unit tests? (Long story short, it didn't. I replaced that unit test with an integration test.) And yes, I've heard all of the lectures from test first folks. But somehow I've never had the experience of things working that way with my projects. (Possibly because most of what I do is close to the software/database interface.)
- mpweiher 6y ago
  If it's talking to the database, it's not a unit test. https://blog.metaobject.com/2014/05/why-i-don-mock.html https://blog.metaobject.com/2014/05/why-i-don-mock.html
  - btilly 6y ago
    I'm aware of this. If it is talking to a database, it's not a unit test. If you've mocked out the database, it is a unit test. My point is that the unit test version is a lot of work to create and tends to be useless. (Which is similar to the point that your blog link made.)
    - mpweiher 6y ago
      No. Stubbing out the database does not turn a test into a unit test , it turns it into a stubbed-out integration test. (Mocking is something different) And since the point of an integration test is to test the integration, having a stubbed out version of that is fairly pointless, as you have discovered. If you create actual unit tests, and allow those to drive your design, you will find much more joy.

pydry 6y ago

Unit tests are not intrinsically easier to maintain. The lower level the test the more it reflects implementation rather than specification and hence the more changes it demands when refactoring. The practical upshot of these types of tests is often that they catch very few actual bugs but nonetheless demand constant and expensive maintenance when code is changed.

Tainnor 6y ago
This is true if your units under tests are bad abstractions. If you don't know yet what you will build, if your requirements change all the time, good abstractions maybe rarely emerge and then maybe unit tests make less sense. However, I've worked on a number of applications where there were very well-defined units that were quite stable in terms of their abstraction, and then unit tests are a joy. "I have this tree structure and I need to transform it into another structure" and stuff like that. It's also worth keeping in mind that "unit testing" doesn't have to mean "class (or method) testing". Sometimes the class is the right level of abstraction but sometimes it can also be a small module of classes working in concert etc. I agree that tests shouldn't simply reflect implementation, but there are functional requirements that can be specified at a lower level.
- closeparen 6y ago
  If I design tests to capture the spirit of the class / module, the line coverage is only 50-60%. In Go each method needs a lot of individual attention to exercise all the error return branches up to a 90%+ coverage standard.
- pydry 6y ago
  I agree. Good, stable abstractions covering complicated logic are where unit tests shine. Unfortunately I find this scenario is also more the exception than the rule. Big ball of mud apps with few effective tests are the norm and apps which do very little complex logic under the hood (focused instead mainly on hooking together services) are also not at all uncommon.
closeparen 6y ago
I think of it as double entry bookkeeping. You have to say what you mean two different ways, and those two phrasings have to be reconcilable. This is still a useful guard against careless errors! But it is a different problem from checking whether your software actually does the right thing. The magnitude of change needed in unit tests while changing code is definitely maddening. For example: add an argument to a method. It's called only 3 places in production code, but each of those sites has 10 test cases, so there are 30 mock expectations to update now. And the IDE doesn't understand those so no automated support. On the other hand, integration and end-to-end tests will go red when (seemingly) nothing at all has changed, because of drift in tangentially related components from other codebases. Usually the fixes are simple, but it requires a willingness to drop what you're doing and investigate when they happen. Otherwise they just get ignored.
- pydry 6y ago
  I think of it as writing everything twice to appease the unit testing gods. If done indiscriminately it really is more of a ritual to ward of the bug gremlins than an effective development practice. Integration tests do suffer from a whole set of problems that unit tests do not but with one fundamental difference: they're usually problems that are soluble with better engineering techniques. Integration tests that go red when something other than the code changes often provide useful feedback, so I wouldn't necessarily rule them out.

Tainnor 6y ago

It's not just that the article misunderstands that unit tests are easier to maintain (if well written at least), but they also don't seem to discuss the problem of combinatorial explosion that plagues end-to-end tests. There are only two ways of avoiding that problem: ignoring it and simply give up on comprehensive testing, or realise that comprehensive testing can only be done by writing mostly unit tests.

btilly 6y ago

I find that unit tests are much less useful than integration tests. When we write unit tests, we ask the person to write the code to spend a lot of work documenting how they expect the system that they are integrating with to work. What happens when that system behaves differently? Or changes in the future? And the way that mocks are set up are very sensitive to implementation details, which is a maintenance burden when you have to change that. As a simple example, I had a function that was loading data into a database with a series of insert statements. I sped it up by several orders of magnitude by using a COPY to send CSV data. Guess how well that worked with the mock database interface in the unit tests? (Long story short, it didn't. I replaced that unit test with an integration test.) And yes, I've heard all of the lectures from test first folks. But somehow I've never had the experience of things working that way with my projects. (Possibly because most of what I do is close to the software/database interface.)

mpweiher 6y ago
If it's talking to the database, it's not a unit test. https://blog.metaobject.com/2014/05/why-i-don-mock.html https://blog.metaobject.com/2014/05/why-i-don-mock.html
- btilly 6y ago
  I'm aware of this. If it is talking to a database, it's not a unit test. If you've mocked out the database, it is a unit test. My point is that the unit test version is a lot of work to create and tends to be useless. (Which is similar to the point that your blog link made.)
  - mpweiher 6y ago
    No. Stubbing out the database does not turn a test into a unit test , it turns it into a stubbed-out integration test. (Mocking is something different) And since the point of an integration test is to test the integration, having a stubbed out version of that is fairly pointless, as you have discovered. If you create actual unit tests, and allow those to drive your design, you will find much more joy.