5 ms·

In small companies where there is no time to "waste" on tests, my view is that 80% of the problems can be caught with 20% of the work by writing integration tes

In small companies where there is no time to "waste" on tests, my view is that 80% of the problems can be caught with 20% of the work by writing integration tests that cover large areas of the application. Writing unit tests would be ideal, but time-consuming. For a web project, that would involve testing all pages for HTTP 200 (< 1 hour bash script that will catch most major bugs), automatically testing most interfaces to see if filling data and clicking "save" works. Of course, for very important/dangerous/complex algorithms in the code, unit tests are useful, but generally, that represents a very low fraction of a web application's code.

rdsubhas 9y ago
Yep. There is a kind of goldilocks zone: - Company starts out, lots of change and RFCs, testing at a shallow depth (high-level tests wherever possible) helps a ton - [Goldilocks zone] Company and team settles down, predictable workflow, "not changing the world" anymore, unit tests help - Project goes into legacy, needs to be split/ merged/ refactored/ rearchitected/, again unit tests don't help. Back to shallow tests and stranglers Only very small part of programming today is doing hardcode datascience and algorithms. Rest are all just plumbing from A to B to C. I always argued in teams to focus on higher level tests, but then, its easy to get shouted down because of "TDD". Sadly, very few people seem to focus on objective rather than process.
- murukesh_s 9y ago
  Yea exactly, it's often painful to see when all you need is an API level test which will anyway cover all the use cases instead of the low level unit tests written for the sake of TDD.. leaner the code more maintainable it is
- humanrebar 9y ago
  > I always argued in teams to focus on higher level tests... In my experience, your inclinations are true, but it actually takes more discipline and buy in from leadership for it to work. If broken tests are lower priority than feature work, your test suite ends up rotting, getting ignored, then abandoned. At least with unit tests, individual contributors can maintain pockets and layers of verified-working code. I'm not arguing against any kind of test. But you do need the people handing out raises, planning sprints, scheduling deadlines, etc. to be on board for end-to-end testing to work. It also helps a lot to introduce those early in the system's life, since the team culture develops with the tests already being there.
- rimliu 9y ago
  I am confused, how unit tests do not help refactoring? Is not this a most often cited benefit of unit tests?
- spronkey 9y ago
  I think the definition of a 'unit' here is possibly causing some strife. If your product consists of somewhat reusable libraries that you compose together, and extend when needed, having strong unit tests for the low level functionality helps immensely when it comes time to refactor. You know, stuff like "when given int(2) and int(3) as arguments, add(int, int) returns int(5)". Likewise those high level API tests that ensure the system-as-a-whole is still doing approximately what it's supposed to, regardless of what mess of wires and small aliens is inside the black box are super useful for moving forward as both proof of functionality and guard against regression. It's the stuff in the middle that's icky IMO. Integration tests that are testing your glue that, especially these days with clouds, containers, microservices etc, can be quite far removed from any codebases you might maintain. I don't have a good solution for it, but I'm searchin'...
GordonS 9y ago
I'm reminded of this article: https://blog.kentcdodds.com/write-tests-not-too-many-mostly-integration-5e8c7fff591c https://blog.kentcdodds.com/write-tests-not-too-many-mostly-... This is advice I generally follow - most of the time, I find I get the most value from integration tests.
ajeet_dhaliwal 9y ago
Agreed and not just small companies - large companies don't have unlimited resources either and they work on a lot of different projects in parallel which can sometimes make the scope of work no less daunting. This is a great essay but I'd say it's the ideal and I would challenge Anti-Pattern 2 in many cases. The time, money and lack of resource crunch is always present. In the video game industry where I've been responsible for automation infrastructure around testing I would say there's the added difficulty where projects can have a short lifespan or be cancelled entirely quite quickly. Integration tests are highly valuable for being a catch-all for issues. Having only integration tests without unit tests is dramatically less work to both setup and maintain. They are most 'bang for buck' because they can catch so many issues. The points mentioned, that they are complex, slow, and harder to debug than unit tests are all true. However there are some ways to counter these problems. In the section marked 'integration test results' he writes that it's hard to know where to start debugging and that a unit test would have isolated the issue to a smaller scope. While that's true, I don't believe that's a good reason to write unit tests because this issue can be resolved by having better logging and reporting. The integration tests shouldn't just report a fail and nothing else, report the specific reason for failure or at least save off the log so that this information is easily extractable. Also useful is to save screen captures, if there's a GUI, and call stack and crash dump data, if it's a crash. With integration tests and either few or no unit tests and great reporting you can get enormous value from automated testing. Most real world environments will demand this sort of high ROI for less unless it's projected to be a very long running project or has massive funding, like a government program or grant. Automated testing saves companies so much cost and helps deliver higher quality software faster, especially with larger number of devs checking in constantly with continuous integration. You're not doing it wrong if you break some of the rules in this essay, you may be required to be more programmatic. There are some great points in this though.
collyw 9y ago
While I agree, integration tests can be pretty costly in term of time to set up. In the small places I have worked, it been acceptable to have a few bugs, and the applications were completely or partially used by internal users, so maybe 40 -50 people who can put up with a bug here or there and report issues personally.
- crdoconnor 9y ago
  I find that integration tests tend to have a much higher set up cost (especially if you are serious about test isolation), but that cost is amortized over the lifetime of the project because the code is much more reusable. Unit tests are less expensive to set up on a per test basis but the code is less reusable and the ongoing maintenance tends to be higher.
  - collyw 9y ago
    Probably, provided that your system is evolving too quickly - that can often be a problem.
Cofike 9y ago
My last company made us stop writing integration tests because "they don't actually find any issues". We then switched to blackbox testing which seemed to basically only be integration tests anyways (at least within a single component/module) so that's, good?
oselhn 9y ago
I am C++ programmer so maybe it is different but from my experience projects with only integration tests are nightmare. 1. Integration tests let you know that something is wrong but they are mostly useless for locating bug in larger code base. They do not help you much when you write code. 2. Unit tests force you to think about code structure. Code not covered by unit tests is usually much worse in terms of readability and maintainability. (It is hard to write unit tests for huge functions or for spaghetti code but integration tests are not affected by this.)
- spdionis 9y ago
  > 2. Unit tests force you to think about code structure. Code not covered by unit tests is usually much worse in terms of readability and maintainability. (It is hard to write unit tests for huge functions or for spaghetti code but integration tests are not affected by this.) This is my biggest issue with TDD proponents. Like... you don't need TDD to learn to code properly!
regularfry 9y ago
I disagree. Typing isn't the bottleneck, so the amount of additional time taken is small.

rdsubhas 9y ago

Yep. There is a kind of goldilocks zone: - Company starts out, lots of change and RFCs, testing at a shallow depth (high-level tests wherever possible) helps a ton - [Goldilocks zone] Company and team settles down, predictable workflow, "not changing the world" anymore, unit tests help - Project goes into legacy, needs to be split/ merged/ refactored/ rearchitected/, again unit tests don't help. Back to shallow tests and stranglers Only very small part of programming today is doing hardcode datascience and algorithms. Rest are all just plumbing from A to B to C. I always argued in teams to focus on higher level tests, but then, its easy to get shouted down because of "TDD". Sadly, very few people seem to focus on objective rather than process.

murukesh_s 9y ago
Yea exactly, it's often painful to see when all you need is an API level test which will anyway cover all the use cases instead of the low level unit tests written for the sake of TDD.. leaner the code more maintainable it is
humanrebar 9y ago
> I always argued in teams to focus on higher level tests... In my experience, your inclinations are true, but it actually takes more discipline and buy in from leadership for it to work. If broken tests are lower priority than feature work, your test suite ends up rotting, getting ignored, then abandoned. At least with unit tests, individual contributors can maintain pockets and layers of verified-working code. I'm not arguing against any kind of test. But you do need the people handing out raises, planning sprints, scheduling deadlines, etc. to be on board for end-to-end testing to work. It also helps a lot to introduce those early in the system's life, since the team culture develops with the tests already being there.
rimliu 9y ago
I am confused, how unit tests do not help refactoring? Is not this a most often cited benefit of unit tests?
spronkey 9y ago
I think the definition of a 'unit' here is possibly causing some strife. If your product consists of somewhat reusable libraries that you compose together, and extend when needed, having strong unit tests for the low level functionality helps immensely when it comes time to refactor. You know, stuff like "when given int(2) and int(3) as arguments, add(int, int) returns int(5)". Likewise those high level API tests that ensure the system-as-a-whole is still doing approximately what it's supposed to, regardless of what mess of wires and small aliens is inside the black box are super useful for moving forward as both proof of functionality and guard against regression. It's the stuff in the middle that's icky IMO. Integration tests that are testing your glue that, especially these days with clouds, containers, microservices etc, can be quite far removed from any codebases you might maintain. I don't have a good solution for it, but I'm searchin'...

GordonS 9y ago

I'm reminded of this article: https://blog.kentcdodds.com/write-tests-not-too-many-mostly-integration-5e8c7fff591c https://blog.kentcdodds.com/write-tests-not-too-many-mostly-... This is advice I generally follow - most of the time, I find I get the most value from integration tests.

ajeet_dhaliwal 9y ago

Agreed and not just small companies - large companies don't have unlimited resources either and they work on a lot of different projects in parallel which can sometimes make the scope of work no less daunting. This is a great essay but I'd say it's the ideal and I would challenge Anti-Pattern 2 in many cases. The time, money and lack of resource crunch is always present. In the video game industry where I've been responsible for automation infrastructure around testing I would say there's the added difficulty where projects can have a short lifespan or be cancelled entirely quite quickly. Integration tests are highly valuable for being a catch-all for issues. Having only integration tests without unit tests is dramatically less work to both setup and maintain. They are most 'bang for buck' because they can catch so many issues. The points mentioned, that they are complex, slow, and harder to debug than unit tests are all true. However there are some ways to counter these problems. In the section marked 'integration test results' he writes that it's hard to know where to start debugging and that a unit test would have isolated the issue to a smaller scope. While that's true, I don't believe that's a good reason to write unit tests because this issue can be resolved by having better logging and reporting. The integration tests shouldn't just report a fail and nothing else, report the specific reason for failure or at least save off the log so that this information is easily extractable. Also useful is to save screen captures, if there's a GUI, and call stack and crash dump data, if it's a crash. With integration tests and either few or no unit tests and great reporting you can get enormous value from automated testing. Most real world environments will demand this sort of high ROI for less unless it's projected to be a very long running project or has massive funding, like a government program or grant. Automated testing saves companies so much cost and helps deliver higher quality software faster, especially with larger number of devs checking in constantly with continuous integration. You're not doing it wrong if you break some of the rules in this essay, you may be required to be more programmatic. There are some great points in this though.

collyw 9y ago

While I agree, integration tests can be pretty costly in term of time to set up. In the small places I have worked, it been acceptable to have a few bugs, and the applications were completely or partially used by internal users, so maybe 40 -50 people who can put up with a bug here or there and report issues personally.

crdoconnor 9y ago
I find that integration tests tend to have a much higher set up cost (especially if you are serious about test isolation), but that cost is amortized over the lifetime of the project because the code is much more reusable. Unit tests are less expensive to set up on a per test basis but the code is less reusable and the ongoing maintenance tends to be higher.
- collyw 9y ago
  Probably, provided that your system is evolving too quickly - that can often be a problem.

Cofike 9y ago

My last company made us stop writing integration tests because "they don't actually find any issues". We then switched to blackbox testing which seemed to basically only be integration tests anyways (at least within a single component/module) so that's, good?

oselhn 9y ago

I am C++ programmer so maybe it is different but from my experience projects with only integration tests are nightmare. 1. Integration tests let you know that something is wrong but they are mostly useless for locating bug in larger code base. They do not help you much when you write code. 2. Unit tests force you to think about code structure. Code not covered by unit tests is usually much worse in terms of readability and maintainability. (It is hard to write unit tests for huge functions or for spaghetti code but integration tests are not affected by this.)

spdionis 9y ago
> 2. Unit tests force you to think about code structure. Code not covered by unit tests is usually much worse in terms of readability and maintainability. (It is hard to write unit tests for huge functions or for spaghetti code but integration tests are not affected by this.) This is my biggest issue with TDD proponents. Like... you don't need TDD to learn to code properly!

regularfry 9y ago

I disagree. Typing isn't the bottleneck, so the amount of additional time taken is small.