5 ms·

> Even with good design, w/o tests you will fear change and so the code will rot. With good tests, there’s no fear, so you’ll clean the code Has anyone ever ac

> Even with good design, w/o tests you will fear change and so the code will rot. With good tests, there’s no fear, so you’ll clean the code Has anyone ever actually found this to be true? I work in a place with 99% test coverage requirements and it's honestly still a super brittle system that everyone is afraid to make big changes to In my experience, automated tests don't promote quality engineering, they just calcify whatever quality exists And before you get into it, TDD is not a panacea for this problem

nickjj 3y ago
> Has anyone ever actually found this to be true? It depends. I've written a number of apps where I'm fearless with making changes because I trust the tests I wrote. One of the apps has been running for 7 years and it's had a lot of big updates, refactors, etc.. I'm breaking every rule there is on jinxing things but there hasn't been a single bug introduced due to those updates and there's ~85% coverage. It's only a ~3k line Flask app though, but it does get deployed straight to production with no staging environment and gets hundreds of requests per day. It writes to a DB, Redis, interacts with multiple 3rd party APIs and sends out webhooks so it does quite a few "external" things. I've never been a fan of TDD and personally I think tests that really hit your DB, Redis, etc. help a lot more than mocked out unit tests or a billion unit tests and nothing else. I tend to write more tests that really test things together. Not full blown Selenium style tests, but I do really write to the DB and other data stores in tests and often use a framework's built-in test client for making HTTP calls. Everything can still be really fast too (~100 tests in 2 seconds is my usual rough guide for having an assortment of "real" tests with Flask).
- solraph 3y ago
  > I've never been a fan of TDD and personally I think tests that really hit your DB, Redis, etc. help a lot more than mocked out unit tests or a billion unit tests and nothing else... ...but I do really write to the DB and other data stores in tests I've come to the conclusion that the idea that "unit tests" should test functions/objects in isolation with completely mocked dependencies is based more on the slow speed of those dependencies in the past than what actually makes for good tests. Now that we have faster computers and storage devices, and easy/fast store creation, we should move past this. Obviously, this is very dependant (no pun intended) on the dependency in question, but as a minimum, anything with SQL should have a test that hits a real SQL DB (PG in docker for example) at some point.
  - arp242 3y ago
    It's not even about faster computers and storage, but just about better test techniques, and better written tests. You don't actually need very fast hardware for fast tests; you just need to spend some time on it. And roughly know what you're doing. External dependencies (such as PostgreSQL, Redis, what-have-you) are a pain though. I feel pretty strongly that just a single command ("go test", "cargo test", "npm test", etc.) should run all the tests on all platforms, reliably, with a minimal of fuss and messing about. Things like docker-compose or whatnot quality as "a fuss and messing about".
    - solraph 3y ago
      They can be a bit of a pain, but if you have a lot of logic tied up in your external dependencies (SQL usually bring main offender here) it's not possible to properly test without using said dependency. That said, in the current job, we have a dedicated Postgres container that comes up with with a just file command. The setup, test, and teardown of schema all happen within the standard test system of the tested platform (pytest or go test specifically).
vidugavia 3y ago
99% coverage does not mean you have good tests. Yes, I have experienced that good tests, combined with a good understanding which parts are critical and which are not, create a great comfort about introducing changes, whether it's Friday afternoon or Monday morning.
- deleted 3y ago
  [deleted]
lukev 3y ago
This is why I'm neutral on the value of unit tests. Sometimes they are helpful, other times less so. I am always in favor of tests around the boundaries between system components (regardless of what you call them; integration tests, API tests, etc.) The more robust the better. If I'm working on a system with robust (ideally generative) integration tests covering the interface of a component, I feel I have near-complete freedom to rewrite any aspect of the component I want with high confidence.
- adonovan 3y ago
  Unit tests are good when there are clear and durable units, but what many people seem to mean by a unit test is one that exercises the internal functions of a package, or the API of an internal package that evolves as fast as your product. Stable interfaces that the user cares about (e.g. CLI, public API, RPCs) make for the highest-value tests.
- nomel 3y ago
  > ideally generative How have you seen this done well? In my experience, this usually ends up being some hello-world type simplicity that doesn't really represent the real world use corner cases.
  - lukev 3y ago
    I have, on multiple occasions. Unfortunately mostly on contracting gigs so I can't cite specifics but I've worked on a major ecommerce platform's api, and two separate databases using this approach. For public examples, I'd point you to (e.g.) Jepsen. I'm not going to deny it requires a pretty high level of time/money to implement. But done well it's super powerful.
- randomdata 3y ago
  > I am always in favor of tests around the boundaries between system components (regardless of what you call them; integration tests, API tests, etc.) Funnily enough, "unit tests" is what this was called originally. The boundary is what "unit" referred to. In my experience, these days most people just call them "tests". That they happen at the boundary is implied as by this point it is generally agreed upon that testing anything else is a waste of time at best and sometimes even detrimental. Which is why nobody in fake internet argument land can settle on what "unit", "integration", etc. mean in the modern age. There is nothing related to testing in need of additional qualification to communicate to other developers what isn't already communicated with "tests" alone.
jasonpeacock 3y ago
I have, and it was amazing. You could quickly make the changes you want not worry about what else breaks, then run the tests. The tests that you expect to fail would fail, and you fix them. Then you find other tests that you didn't expect also failed; you'd review the impact of your changes on those parts of the system and make appropriate changes until the tests were happy. Because the testing was so thorough (and quality), you had high confidence that nothing unexpected was broken. It was "move fast & break [tests], then fix them and deploy" :)
sealeck 3y ago
I think if you evolve your testing system to feed random numbers into your program and ensure that various correctness properties hold for all these random inputs you are likely to get a pretty robust program. You can make this slightly efficient by using a fuzzer (essentially uses feedback from running previous inputs to see how running new inputs works). See for example https://propertesting.com/ https://propertesting.com/ or https://en.wikipedia.org/wiki/American_Fuzzy_Lop_(software) https://en.wikipedia.org/wiki/American_Fuzzy_Lop_(software) Of course if you need a higher standard of guarantee you can try model checking or other formal verification techniques.
Izkata 3y ago
Test-a-single-function style, what most people default to when they hear "unit test", is probably the thing that most leads into that situation. I've been occasionally pushing coworkers to restructure code and tests into a higher level style, defining an API boundary and calling it from the rest of the code (as if you're writing a library except it's inside your codebase instead of installed), then writing tests to that API. That does make for some easily refactorable code where the tests don't need to change at all. That pseudo-library style is kind of what "unit test" originally meant: a business unit, not a code unit. Examples simplified it too much and people copied the style instead of the substance, and the original meaning was lost. For another example, some of our recent projects have been on updating daemons, those have also been really good candidates for this since there are clear entry and exit points.
- maw 3y ago
  > That pseudo-library style is kind of what "unit test" originally meant: a business unit, not a code unit. Examples simplified it too much and people copied the style instead of the substance, and the original meaning was lost. Very interesting. It sounds a bit like what happened with the term "Hungarian notation".
- GuB-42 3y ago
  But you probably won't get 99% coverage doing that. Some conditions are just too difficult to get from a higher level, and you still need to take them in consideration: hardware-related problems, race conditions, etc... I don't consider it a problem, but it is to those who want to see big numbers.
  - yellowapple 3y ago
    I feel like if you're unable to get 99% coverage with that then there's probably some dead code that needs pruned. As for hardware-related problems and race conditions, testing at a higher level of abstraction seems like it'd help more than it'd hurt - in the former case ensuring graceful handling as part of the tests, and in the latter case making the race conditions more likely to hit (and hopefully fix).
solraph 3y ago
> > Even with good design, w/o tests you will fear change and so the code will rot. With good tests, there’s no fear, so you’ll clean the code > Has anyone ever actually found this to be true? Yes, but you have to have the right kind of test coverage, and that's the tricky part. Yesterday, I refactored a bunch of functions that changed a bunch of unit tests. However, since we also have integration/system tests on that code, I'm confident that I haven't broken the code as a whole. Without those system tests, I would not have confidence that the change would be successful, and probably would not have refactored. In another codebase that hadn't been touched for a year, as part of a feature change I refactored an SQL statement to what I thought was a more optimal design and immediately broke a bunch of tests. Based on that, I was able to understand the original intent of the SQL, and updated it in line with the feature change. I added test scenarios for the new feature, but left the existing scenarios as is. Without those tests, would have broken the system in a subtle way.
furstenheim 3y ago
Take a look at hexagon pattern from Spotify. Once you start testing the user contract of your services against real databases (testcontainners is a good option), then you can change all the internals and be sure that the externals will work. https://engineering.atspotify.com/2018/01/testing-of-microservices/ https://engineering.atspotify.com/2018/01/testing-of-microse...
barfard 3y ago
could still get 99% test coverage with useless tests
hinkley 3y ago
I’ve experienced it. But only at companies that don’t give a shit about test coverage %. The percent kills the purpose of the tests, which is certainty.
vineyardmike 3y ago
Just to re-iterate another commenter, I had a positive experience before. When I was at BigCompanyA, we had 95% coverage requirements as a management fiat. The company generally was very management-decree-driven. People would unit test individual methods and helper functions and each “unit” of code. If you wanted to change an API, you had to dig through a sea of broken tests because each little bit of code was individually tested. We literally had unit tests that validated one line methods for string concatenation of a prefix (in Java). Management said add tests with each commit so everyone added tests without thinking about what is valuable. At BigCompanyB, our testing was engineering driven, not some management metric being tracked. The goal was to test “public interfaces” and ensure that these tests captured all the helper methods along the way. This helped catch dead or extraneous code if you couldn’t write a test to exercise a particular path. Changing an API didn’t require changing a bunch of silly tests. We still had equal >95% coverage, and it was very useful. Basically you need to actually think about what makes sense to test and write logical tests.
agumonkey 3y ago
You're both right. TDD is no panacea, and having no test is being blind. There's a black art into estimating what angles to test and how deep to avoid calcification.
- randomdata 3y ago
  > There's a black art into estimating what angles to test and how deep to avoid calcification. There may be a black art to recognizing what you overlooked, but otherwise it is pretty straightforward. Tests are your documentation. The angles needed to be covered are exactly what the user needs to know to use your software – how to use it and, when using it, what happens both in the expected case and when failure occurs (especially what happens when failure occurs!). Calcification is of little concern as the inverse is breaking changes, and the user does not want to deal with your breaking changes. You can put your mind at ease knowing that once you commit to a documented feature, it should remain there until the end of time.
  - bluefirebrand 3y ago
    > Calcification is of little concern as the inverse is breaking changes, and the user does not want to deal with your breaking changes Please explain this to Product Managers. :)
zelphirkalt 3y ago
At some point you start to fear changing too much, because of all the tests you will have to rewrite, which can be tedious, especially, when the code mutates things all over the place and you have to mock 5 things per unit test.
- randomdata 3y ago
  > because of all the tests you will have to rewrite If you have to rewrite tests that means you've changed the user experience in ways that are not backwards compatible. Which is sometimes valid, but not exactly what is being talked about here. The discussion here is more about changing the code in ways that makes the code better, but still delivers the same user experience – possibly with new features added, but not where anything is taken away.
  - Jensson 3y ago
    > If you have to rewrite tests that means you've changed the user experience in ways that are not backwards compatible. That is only true for integration tests. You can rewrite a set of local functions without changing user behavior, and then you need to change tests, such refactors becomes a pain when you have too many unit tests but are really easy when you have many integration tests.
    - randomdata 3y ago
      I don't get it. Beck was quite explicit when coining the term "unit tests" that the unit refers to the set of functionality found at the integration point – which seems to be what you refer to as "integration tests", and what everyone else these days call "tests". It's all the same. If changes to "local functions" calls for tests to be rewritten, that means you've exposed "local functionality", even if by accident, to the outside user. Which means it is not actually local functionality, but something you have exported and are committed to maintaining. Rewriting the tests is not the correct course of action. You need to fix the code that you just broke as the functional contract was violated. With any luck that hard lesson will teach you to be more careful next time.
elliottkember 3y ago
I think you missed a word. "Good" tests is a crucial part of the statement. It's totally possible to write bad tests, the same way it's possible to write bad code.
dimal 3y ago
Only for limited areas of a server codebase on a small team who understood what tests were meant to accomplish and wrote excellent tests. Basically, about a year and a half out of a twenty five year career. I’ve never seen them useful on front end code except in extremely limited circumstances. In most other cases, we had tests that only covered the happy path and did little more than slow down builds and make refactoring more difficult. In other words, they made things worse. E2E tests, in particular, are the Afghanistan of web development. I’ve seen more time wasted getting useless tests green than I’ve seen wasted on any other programming exercise.
Aeolun 3y ago
> Has anyone ever actually found this to be true? We have a 300kloc monster, and I find that going up from 0 to 60% test coverage has given me appreciably more confidence that the system still works after any change. Sure, the test code is almost twice that size, and breaks almost as often as the code, but my confidence in the system itself is definitely higher. To prevent people from writing tests that need too many mocks we now have explicit dependency injection. So much easier to reason about stuff, and prevent’s people from not testing the important bits.
bazoom42 3y ago
It is easy to create bad tests. I have seen an over-reliance on mocking giving high code coverage but still quite useless tests.
onion2k 3y ago
I work in a place with 99% test coverage requirements and it's honestly still a super brittle system that everyone is afraid to make big changes to. Obviously tests are going to break when you change the code's APIs and functionality. That's expected, and does nothing to boost your confidence in the part of the code you're working in. The point of tests is to improve your confidence that you're not doing things that have an unexpected impact somewhere else (hence the Bob Martin quote in the article about tests being useful even if you have good design). Tests aren't an alternative to thinking. You still have to consider what changes you're making, and what tests should break as a result. That's just part of your code change. When tests that shouldn't have broken start failing that's when they show their value.
PaulStatezny 3y ago
You need mostly integration tests, not unit tests. Unit tests which dependency-inject mocks of other parts of your codebase are 99% worthless. Source: Spent many years writing the latter and they caught close to 0 bugs. Moved to Elixir ecosystem where integration tests are the norm, and they catch bugs on a regular basis.
- rramadass 3y ago
  Bingo! I never understood the fascination with Unit Tests. For Testing to be useful the code being tested should have a certain degree of Complexity (algorithmic/behavioural/state-transition/etc.). But what i see from most unit tests is mere "busy work" as if mock-testing a trivial class/api/etc. would somehow make your code better. A similar criticism is also applicable to TDD based programming.
  - hitchstory 3y ago
    The "busy work" of low level unit tests is perfectly valid, but TDD still works extremely well with high level integration tests so I don't think a similar criticism necessarily applies.
  - randomdata 3y ago
    > I never understood the fascination with Unit Tests. Me neither. The legacy forms of testing that "unit" was trying to differentiate from have died out. All tests are unit tests today, which is why most developers just say "tests". "Unit" is already implied and adds no additional information not already understood by "test" alone. Same goes for integration tests. Integration tests, as it was historically defined, died out. It seems anyone still trying to use the term today is using it to also just mean "testing", or what was historically known as unit testing. Like unit, a pointless qualification.
- pjerem 3y ago
  In my experience, Unit tests are interesting when they are inlined with the function they test, i.e. in the same file (like python doctests). Of course it’s harder to set the testing environment but that may in fact be encouraging you to write stateless code so there is that.
lijok 3y ago
> With good tests > I work in a place with 99% test coverage requirements Sounds like your problem is bad tests. In fact, I guarantee they’re bad tests because that’s exactly what coverage requirements lead to
okeuro49 3y ago
> Has anyone ever actually found this to be true? Yes, always.

nickjj 3y ago

> Has anyone ever actually found this to be true? It depends. I've written a number of apps where I'm fearless with making changes because I trust the tests I wrote. One of the apps has been running for 7 years and it's had a lot of big updates, refactors, etc.. I'm breaking every rule there is on jinxing things but there hasn't been a single bug introduced due to those updates and there's ~85% coverage. It's only a ~3k line Flask app though, but it does get deployed straight to production with no staging environment and gets hundreds of requests per day. It writes to a DB, Redis, interacts with multiple 3rd party APIs and sends out webhooks so it does quite a few "external" things. I've never been a fan of TDD and personally I think tests that really hit your DB, Redis, etc. help a lot more than mocked out unit tests or a billion unit tests and nothing else. I tend to write more tests that really test things together. Not full blown Selenium style tests, but I do really write to the DB and other data stores in tests and often use a framework's built-in test client for making HTTP calls. Everything can still be really fast too (~100 tests in 2 seconds is my usual rough guide for having an assortment of "real" tests with Flask).

solraph 3y ago
> I've never been a fan of TDD and personally I think tests that really hit your DB, Redis, etc. help a lot more than mocked out unit tests or a billion unit tests and nothing else... ...but I do really write to the DB and other data stores in tests I've come to the conclusion that the idea that "unit tests" should test functions/objects in isolation with completely mocked dependencies is based more on the slow speed of those dependencies in the past than what actually makes for good tests. Now that we have faster computers and storage devices, and easy/fast store creation, we should move past this. Obviously, this is very dependant (no pun intended) on the dependency in question, but as a minimum, anything with SQL should have a test that hits a real SQL DB (PG in docker for example) at some point.
- arp242 3y ago
  It's not even about faster computers and storage, but just about better test techniques, and better written tests. You don't actually need very fast hardware for fast tests; you just need to spend some time on it. And roughly know what you're doing. External dependencies (such as PostgreSQL, Redis, what-have-you) are a pain though. I feel pretty strongly that just a single command ("go test", "cargo test", "npm test", etc.) should run all the tests on all platforms, reliably, with a minimal of fuss and messing about. Things like docker-compose or whatnot quality as "a fuss and messing about".
  - solraph 3y ago
    They can be a bit of a pain, but if you have a lot of logic tied up in your external dependencies (SQL usually bring main offender here) it's not possible to properly test without using said dependency. That said, in the current job, we have a dedicated Postgres container that comes up with with a just file command. The setup, test, and teardown of schema all happen within the standard test system of the tested platform (pytest or go test specifically).

vidugavia 3y ago

99% coverage does not mean you have good tests. Yes, I have experienced that good tests, combined with a good understanding which parts are critical and which are not, create a great comfort about introducing changes, whether it's Friday afternoon or Monday morning.

deleted 3y ago
[deleted]

lukev 3y ago

This is why I'm neutral on the value of unit tests. Sometimes they are helpful, other times less so. I am always in favor of tests around the boundaries between system components (regardless of what you call them; integration tests, API tests, etc.) The more robust the better. If I'm working on a system with robust (ideally generative) integration tests covering the interface of a component, I feel I have near-complete freedom to rewrite any aspect of the component I want with high confidence.

adonovan 3y ago
Unit tests are good when there are clear and durable units, but what many people seem to mean by a unit test is one that exercises the internal functions of a package, or the API of an internal package that evolves as fast as your product. Stable interfaces that the user cares about (e.g. CLI, public API, RPCs) make for the highest-value tests.
nomel 3y ago
> ideally generative How have you seen this done well? In my experience, this usually ends up being some hello-world type simplicity that doesn't really represent the real world use corner cases.
- lukev 3y ago
  I have, on multiple occasions. Unfortunately mostly on contracting gigs so I can't cite specifics but I've worked on a major ecommerce platform's api, and two separate databases using this approach. For public examples, I'd point you to (e.g.) Jepsen. I'm not going to deny it requires a pretty high level of time/money to implement. But done well it's super powerful.
randomdata 3y ago
> I am always in favor of tests around the boundaries between system components (regardless of what you call them; integration tests, API tests, etc.) Funnily enough, "unit tests" is what this was called originally. The boundary is what "unit" referred to. In my experience, these days most people just call them "tests". That they happen at the boundary is implied as by this point it is generally agreed upon that testing anything else is a waste of time at best and sometimes even detrimental. Which is why nobody in fake internet argument land can settle on what "unit", "integration", etc. mean in the modern age. There is nothing related to testing in need of additional qualification to communicate to other developers what isn't already communicated with "tests" alone.

jasonpeacock 3y ago

I have, and it was amazing. You could quickly make the changes you want not worry about what else breaks, then run the tests. The tests that you expect to fail would fail, and you fix them. Then you find other tests that you didn't expect also failed; you'd review the impact of your changes on those parts of the system and make appropriate changes until the tests were happy. Because the testing was so thorough (and quality), you had high confidence that nothing unexpected was broken. It was "move fast & break [tests], then fix them and deploy" :)

sealeck 3y ago

I think if you evolve your testing system to feed random numbers into your program and ensure that various correctness properties hold for all these random inputs you are likely to get a pretty robust program. You can make this slightly efficient by using a fuzzer (essentially uses feedback from running previous inputs to see how running new inputs works). See for example https://propertesting.com/ https://propertesting.com/ or https://en.wikipedia.org/wiki/American_Fuzzy_Lop_(software) https://en.wikipedia.org/wiki/American_Fuzzy_Lop_(software) Of course if you need a higher standard of guarantee you can try model checking or other formal verification techniques.

Izkata 3y ago

Test-a-single-function style, what most people default to when they hear "unit test", is probably the thing that most leads into that situation. I've been occasionally pushing coworkers to restructure code and tests into a higher level style, defining an API boundary and calling it from the rest of the code (as if you're writing a library except it's inside your codebase instead of installed), then writing tests to that API. That does make for some easily refactorable code where the tests don't need to change at all. That pseudo-library style is kind of what "unit test" originally meant: a business unit, not a code unit. Examples simplified it too much and people copied the style instead of the substance, and the original meaning was lost. For another example, some of our recent projects have been on updating daemons, those have also been really good candidates for this since there are clear entry and exit points.

maw 3y ago
> That pseudo-library style is kind of what "unit test" originally meant: a business unit, not a code unit. Examples simplified it too much and people copied the style instead of the substance, and the original meaning was lost. Very interesting. It sounds a bit like what happened with the term "Hungarian notation".
GuB-42 3y ago
But you probably won't get 99% coverage doing that. Some conditions are just too difficult to get from a higher level, and you still need to take them in consideration: hardware-related problems, race conditions, etc... I don't consider it a problem, but it is to those who want to see big numbers.
- yellowapple 3y ago
  I feel like if you're unable to get 99% coverage with that then there's probably some dead code that needs pruned. As for hardware-related problems and race conditions, testing at a higher level of abstraction seems like it'd help more than it'd hurt - in the former case ensuring graceful handling as part of the tests, and in the latter case making the race conditions more likely to hit (and hopefully fix).

solraph 3y ago

> > Even with good design, w/o tests you will fear change and so the code will rot. With good tests, there’s no fear, so you’ll clean the code > Has anyone ever actually found this to be true? Yes, but you have to have the right kind of test coverage, and that's the tricky part. Yesterday, I refactored a bunch of functions that changed a bunch of unit tests. However, since we also have integration/system tests on that code, I'm confident that I haven't broken the code as a whole. Without those system tests, I would not have confidence that the change would be successful, and probably would not have refactored. In another codebase that hadn't been touched for a year, as part of a feature change I refactored an SQL statement to what I thought was a more optimal design and immediately broke a bunch of tests. Based on that, I was able to understand the original intent of the SQL, and updated it in line with the feature change. I added test scenarios for the new feature, but left the existing scenarios as is. Without those tests, would have broken the system in a subtle way.

furstenheim 3y ago

Take a look at hexagon pattern from Spotify. Once you start testing the user contract of your services against real databases (testcontainners is a good option), then you can change all the internals and be sure that the externals will work. https://engineering.atspotify.com/2018/01/testing-of-microservices/ https://engineering.atspotify.com/2018/01/testing-of-microse...

barfard 3y ago

could still get 99% test coverage with useless tests

hinkley 3y ago

I’ve experienced it. But only at companies that don’t give a shit about test coverage %. The percent kills the purpose of the tests, which is certainty.

vineyardmike 3y ago

Just to re-iterate another commenter, I had a positive experience before. When I was at BigCompanyA, we had 95% coverage requirements as a management fiat. The company generally was very management-decree-driven. People would unit test individual methods and helper functions and each “unit” of code. If you wanted to change an API, you had to dig through a sea of broken tests because each little bit of code was individually tested. We literally had unit tests that validated one line methods for string concatenation of a prefix (in Java). Management said add tests with each commit so everyone added tests without thinking about what is valuable. At BigCompanyB, our testing was engineering driven, not some management metric being tracked. The goal was to test “public interfaces” and ensure that these tests captured all the helper methods along the way. This helped catch dead or extraneous code if you couldn’t write a test to exercise a particular path. Changing an API didn’t require changing a bunch of silly tests. We still had equal >95% coverage, and it was very useful. Basically you need to actually think about what makes sense to test and write logical tests.

agumonkey 3y ago

You're both right. TDD is no panacea, and having no test is being blind. There's a black art into estimating what angles to test and how deep to avoid calcification.

randomdata 3y ago
> There's a black art into estimating what angles to test and how deep to avoid calcification. There may be a black art to recognizing what you overlooked, but otherwise it is pretty straightforward. Tests are your documentation. The angles needed to be covered are exactly what the user needs to know to use your software – how to use it and, when using it, what happens both in the expected case and when failure occurs (especially what happens when failure occurs!). Calcification is of little concern as the inverse is breaking changes, and the user does not want to deal with your breaking changes. You can put your mind at ease knowing that once you commit to a documented feature, it should remain there until the end of time.
- bluefirebrand 3y ago
  > Calcification is of little concern as the inverse is breaking changes, and the user does not want to deal with your breaking changes Please explain this to Product Managers. :)

zelphirkalt 3y ago

At some point you start to fear changing too much, because of all the tests you will have to rewrite, which can be tedious, especially, when the code mutates things all over the place and you have to mock 5 things per unit test.

randomdata 3y ago
> because of all the tests you will have to rewrite If you have to rewrite tests that means you've changed the user experience in ways that are not backwards compatible. Which is sometimes valid, but not exactly what is being talked about here. The discussion here is more about changing the code in ways that makes the code better, but still delivers the same user experience – possibly with new features added, but not where anything is taken away.
- Jensson 3y ago
  > If you have to rewrite tests that means you've changed the user experience in ways that are not backwards compatible. That is only true for integration tests. You can rewrite a set of local functions without changing user behavior, and then you need to change tests, such refactors becomes a pain when you have too many unit tests but are really easy when you have many integration tests.
  - randomdata 3y ago
    I don't get it. Beck was quite explicit when coining the term "unit tests" that the unit refers to the set of functionality found at the integration point – which seems to be what you refer to as "integration tests", and what everyone else these days call "tests". It's all the same. If changes to "local functions" calls for tests to be rewritten, that means you've exposed "local functionality", even if by accident, to the outside user. Which means it is not actually local functionality, but something you have exported and are committed to maintaining. Rewriting the tests is not the correct course of action. You need to fix the code that you just broke as the functional contract was violated. With any luck that hard lesson will teach you to be more careful next time.

elliottkember 3y ago

I think you missed a word. "Good" tests is a crucial part of the statement. It's totally possible to write bad tests, the same way it's possible to write bad code.

dimal 3y ago

Only for limited areas of a server codebase on a small team who understood what tests were meant to accomplish and wrote excellent tests. Basically, about a year and a half out of a twenty five year career. I’ve never seen them useful on front end code except in extremely limited circumstances. In most other cases, we had tests that only covered the happy path and did little more than slow down builds and make refactoring more difficult. In other words, they made things worse. E2E tests, in particular, are the Afghanistan of web development. I’ve seen more time wasted getting useless tests green than I’ve seen wasted on any other programming exercise.

Aeolun 3y ago

> Has anyone ever actually found this to be true? We have a 300kloc monster, and I find that going up from 0 to 60% test coverage has given me appreciably more confidence that the system still works after any change. Sure, the test code is almost twice that size, and breaks almost as often as the code, but my confidence in the system itself is definitely higher. To prevent people from writing tests that need too many mocks we now have explicit dependency injection. So much easier to reason about stuff, and prevent’s people from not testing the important bits.

bazoom42 3y ago

It is easy to create bad tests. I have seen an over-reliance on mocking giving high code coverage but still quite useless tests.

onion2k 3y ago

I work in a place with 99% test coverage requirements and it's honestly still a super brittle system that everyone is afraid to make big changes to. Obviously tests are going to break when you change the code's APIs and functionality. That's expected, and does nothing to boost your confidence in the part of the code you're working in. The point of tests is to improve your confidence that you're not doing things that have an unexpected impact somewhere else (hence the Bob Martin quote in the article about tests being useful even if you have good design). Tests aren't an alternative to thinking. You still have to consider what changes you're making, and what tests should break as a result. That's just part of your code change. When tests that shouldn't have broken start failing that's when they show their value.

PaulStatezny 3y ago

You need mostly integration tests, not unit tests. Unit tests which dependency-inject mocks of other parts of your codebase are 99% worthless. Source: Spent many years writing the latter and they caught close to 0 bugs. Moved to Elixir ecosystem where integration tests are the norm, and they catch bugs on a regular basis.

rramadass 3y ago
Bingo! I never understood the fascination with Unit Tests. For Testing to be useful the code being tested should have a certain degree of Complexity (algorithmic/behavioural/state-transition/etc.). But what i see from most unit tests is mere "busy work" as if mock-testing a trivial class/api/etc. would somehow make your code better. A similar criticism is also applicable to TDD based programming.
- hitchstory 3y ago
  The "busy work" of low level unit tests is perfectly valid, but TDD still works extremely well with high level integration tests so I don't think a similar criticism necessarily applies.
- randomdata 3y ago
  > I never understood the fascination with Unit Tests. Me neither. The legacy forms of testing that "unit" was trying to differentiate from have died out. All tests are unit tests today, which is why most developers just say "tests". "Unit" is already implied and adds no additional information not already understood by "test" alone. Same goes for integration tests. Integration tests, as it was historically defined, died out. It seems anyone still trying to use the term today is using it to also just mean "testing", or what was historically known as unit testing. Like unit, a pointless qualification.
pjerem 3y ago
In my experience, Unit tests are interesting when they are inlined with the function they test, i.e. in the same file (like python doctests). Of course it’s harder to set the testing environment but that may in fact be encouraging you to write stateless code so there is that.

lijok 3y ago

> With good tests > I work in a place with 99% test coverage requirements Sounds like your problem is bad tests. In fact, I guarantee they’re bad tests because that’s exactly what coverage requirements lead to

okeuro49 3y ago

> Has anyone ever actually found this to be true? Yes, always.