4 ms·

>Do not isolate code when you test it. If you do so, the tests become fragile and do not help you in case you refactor the software. I don't agree with this bi

>Do not isolate code when you test it. If you do so, the tests become fragile and do not help you in case you refactor the software. I don't agree with this bit. This is how you make a fragile test suites that are difficult to refactor. Why do we go out of our way to hide information in our code? Answer because code that spaghettis in an out of a lot of other code means small changes cause unexpected regressions, and the changes to fix that regression causes other regressions. Suites that don't isolate code have exactly this problem. They are like a scavenger hunt of assumptions and inevitably lead to people fixing the tests instead of the code. In addition because you have introduced a network effect a small change will likely invalidate a huge number of tests. Meaning you have to update them all. Integration suites have value, but I use them with discretion because I know that they are a challenge to maintain. Tight unit tests that test small pieces of logic thoroughly and make ample use of mocks to ensure deterministic outcomes are a must. These tests are highly durable. The code can change all around it but that function will work as advertised until someone changes it. Then they can update its test suite and you're back in the golden zone. In my experience if you build a complex structure out of a lot of quality components that complex structure will tend to be high quality itself. My pattern usually pushes the complexity into the units and I leave "orchestration" units that tie everything together untested but intentionally simple. I know the code coverage folks will chime for sure, but I don't aim for 100% code coverage. I aim for a balanced test suite that ensures the best possible outcome for the minimum investment with the least amount of hassle when introducing new changes. Trying to test everything never seems to get me there.

Diggsey 5y ago
> Answer because code that spaghettis in an out of a lot of other code means small changes cause unexpected regressions That's not a fragile test suite, that's a test suite working well: you want tests to start failing if you introduced a regression to the overall program behaviour. > the changes to fix that regression causes other regressions It sounds like what you're really getting at is something different from test fragility: specifically, if a test fails, how easy is it to identify where in the code a fix needs to be applied. That is also an important consideration when writing tests but it's completely separate from test fragility. > Integration suites have value, but I use them with discretion because I know that they are a challenge to maintain. This is the opposite of how the test pyramid is supposed to work. Integration tests are supposed to be less fragile because they don't rely on specific implementation details or mocks: they're mimicking a real user or a consumer of your API. If you are getting false positives from your integration tests a lot, then either the tests are written poorly, or you're making too many breaking changes to your library/product/whatever. > Tight unit tests that test small pieces of logic thoroughly and make ample use of mocks to ensure deterministic outcomes are a must. The code can change all around it but that function will work as advertised until someone changes it. Then they can update its test suite and you're back in the golden zone. That's completely useless as a test then. Of course if you change code around it the test won't break. The whoe point of a test is that it doesn't break even for (some) changes to the code being tested, and if you always have to update the unit test when you change the code then it is providing no vlaue and giving you a false sense of security that all your tests are green. Unit tests are great when you have a particular irreducable complexity in a piece of code, or more generally they are good when there is some constraint that should be locally upheld that is likely to be more permanent than the specific implementation.
- SassyGrapefruit 5y ago
  > Integration tests are supposed to be less fragile because they don't rely on specific implementation details or mocks I think you misunderstand what I mean by fragility. By fragility I mean a few things. First, it's sensitive to conditions other than the logic under test. Imagine I have an integration test that round trips to a database. What if there is a momentary TCP/IP burp and the connection drops? The test will fail. Nothing with the wrong the test; nothing wrong with the logic, yet it still fails. Then under different circumstances same test/same logic passes. What do now? This is a fragile test. Second, fragility can happen when the outcome of a test is not deterministic generally because the function under test is not referentially transparent. Good functions are referentially transparent by definition. Mocks can turn non-RF functions into RF functions because you can exercise complete control over the mock's return value. Integration suites struggle to embody this property. If you are careful with setup data you can get around this but its tedious. If you are querying external systems then good luck you are at the mercy of whatever data happens to be there. Same deal as number 1. Sometimes the same test will pass or fail without changes to the logic. Fragile. Third, Fragility can mean that small changes have cascading impact. If I write an integration suite that tests a RESTful endpoint and I refactor that endpoint's return type I run the risk of invalidating huge swathes of tests. Here I would be judicious. If the endpoint was critical and didn't churn much I'd probably invest in an integration suite. I cannot fathom a situation where an integration test that requires setup data, database connectivity, possibly network connectivity, specific sequencing, etc. Is going to be less fragile than a test that does nothing but test the specific logic conditions within a function and has no other dependencies. Your style is not invalid I just wouldn't adopt it myself. >This is the opposite of how the test pyramid is supposed to work. Where I'm from we'd call this having a case of the "sposedas". Nothing is "supposed" to work one way or another. There is no universal truth here despite what conference speakers would have you believe. There is only the trade-offs implied between a specific course of action vs. another.
winddude 5y ago
In reality it depends if you should isolate the code or not, are you writing a unit test or an integration test? If you wrote an algorithmic function crucial to your app, isolate that MF and write a unit test. If you're testing a transformer or API response, you don't need to isolate each layer and run/write your integration tests at this level. If you can't isolate your code for unittest and you're writing OOP your architecture probably needs to be rethought. That said my goal when writing something new is to have all the critical/core functionality covered on the backend, with a test case for edge cases. The biggest goal is I don't know who will be working on it next, and if they miss-understand what a function is doing because they are just trying to fix a "bug", the test case should catch it. That's why it's import in TTD to have one person write the tests and someone else write the code. More so if your team , is a clusterfuck particularly some of the senior devs are hacks.

Diggsey 5y ago

> Answer because code that spaghettis in an out of a lot of other code means small changes cause unexpected regressions That's not a fragile test suite, that's a test suite working well: you want tests to start failing if you introduced a regression to the overall program behaviour. > the changes to fix that regression causes other regressions It sounds like what you're really getting at is something different from test fragility: specifically, if a test fails, how easy is it to identify where in the code a fix needs to be applied. That is also an important consideration when writing tests but it's completely separate from test fragility. > Integration suites have value, but I use them with discretion because I know that they are a challenge to maintain. This is the opposite of how the test pyramid is supposed to work. Integration tests are supposed to be less fragile because they don't rely on specific implementation details or mocks: they're mimicking a real user or a consumer of your API. If you are getting false positives from your integration tests a lot, then either the tests are written poorly, or you're making too many breaking changes to your library/product/whatever. > Tight unit tests that test small pieces of logic thoroughly and make ample use of mocks to ensure deterministic outcomes are a must. The code can change all around it but that function will work as advertised until someone changes it. Then they can update its test suite and you're back in the golden zone. That's completely useless as a test then. Of course if you change code around it the test won't break. The whoe point of a test is that it doesn't break even for (some) changes to the code being tested, and if you always have to update the unit test when you change the code then it is providing no vlaue and giving you a false sense of security that all your tests are green. Unit tests are great when you have a particular irreducable complexity in a piece of code, or more generally they are good when there is some constraint that should be locally upheld that is likely to be more permanent than the specific implementation.

SassyGrapefruit 5y ago
> Integration tests are supposed to be less fragile because they don't rely on specific implementation details or mocks I think you misunderstand what I mean by fragility. By fragility I mean a few things. First, it's sensitive to conditions other than the logic under test. Imagine I have an integration test that round trips to a database. What if there is a momentary TCP/IP burp and the connection drops? The test will fail. Nothing with the wrong the test; nothing wrong with the logic, yet it still fails. Then under different circumstances same test/same logic passes. What do now? This is a fragile test. Second, fragility can happen when the outcome of a test is not deterministic generally because the function under test is not referentially transparent. Good functions are referentially transparent by definition. Mocks can turn non-RF functions into RF functions because you can exercise complete control over the mock's return value. Integration suites struggle to embody this property. If you are careful with setup data you can get around this but its tedious. If you are querying external systems then good luck you are at the mercy of whatever data happens to be there. Same deal as number 1. Sometimes the same test will pass or fail without changes to the logic. Fragile. Third, Fragility can mean that small changes have cascading impact. If I write an integration suite that tests a RESTful endpoint and I refactor that endpoint's return type I run the risk of invalidating huge swathes of tests. Here I would be judicious. If the endpoint was critical and didn't churn much I'd probably invest in an integration suite. I cannot fathom a situation where an integration test that requires setup data, database connectivity, possibly network connectivity, specific sequencing, etc. Is going to be less fragile than a test that does nothing but test the specific logic conditions within a function and has no other dependencies. Your style is not invalid I just wouldn't adopt it myself. >This is the opposite of how the test pyramid is supposed to work. Where I'm from we'd call this having a case of the "sposedas". Nothing is "supposed" to work one way or another. There is no universal truth here despite what conference speakers would have you believe. There is only the trade-offs implied between a specific course of action vs. another.

winddude 5y ago

In reality it depends if you should isolate the code or not, are you writing a unit test or an integration test? If you wrote an algorithmic function crucial to your app, isolate that MF and write a unit test. If you're testing a transformer or API response, you don't need to isolate each layer and run/write your integration tests at this level. If you can't isolate your code for unittest and you're writing OOP your architecture probably needs to be rethought. That said my goal when writing something new is to have all the critical/core functionality covered on the backend, with a test case for edge cases. The biggest goal is I don't know who will be working on it next, and if they miss-understand what a function is doing because they are just trying to fix a "bug", the test case should catch it. That's why it's import in TTD to have one person write the tests and someone else write the code. More so if your team , is a clusterfuck particularly some of the senior devs are hacks.