3 ms·

> Answer because code that spaghettis in an out of a lot of other code means small changes cause unexpected regressions That's not a fragile test suite, that's

> Answer because code that spaghettis in an out of a lot of other code means small changes cause unexpected regressions That's not a fragile test suite, that's a test suite working well: you want tests to start failing if you introduced a regression to the overall program behaviour. > the changes to fix that regression causes other regressions It sounds like what you're really getting at is something different from test fragility: specifically, if a test fails, how easy is it to identify where in the code a fix needs to be applied. That is also an important consideration when writing tests but it's completely separate from test fragility. > Integration suites have value, but I use them with discretion because I know that they are a challenge to maintain. This is the opposite of how the test pyramid is supposed to work. Integration tests are supposed to be less fragile because they don't rely on specific implementation details or mocks: they're mimicking a real user or a consumer of your API. If you are getting false positives from your integration tests a lot, then either the tests are written poorly, or you're making too many breaking changes to your library/product/whatever. > Tight unit tests that test small pieces of logic thoroughly and make ample use of mocks to ensure deterministic outcomes are a must. The code can change all around it but that function will work as advertised until someone changes it. Then they can update its test suite and you're back in the golden zone. That's completely useless as a test then. Of course if you change code around it the test won't break. The whoe point of a test is that it doesn't break even for (some) changes to the code being tested, and if you always have to update the unit test when you change the code then it is providing no vlaue and giving you a false sense of security that all your tests are green. Unit tests are great when you have a particular irreducable complexity in a piece of code, or more generally they are good when there is some constraint that should be locally upheld that is likely to be more permanent than the specific implementation.

SassyGrapefruit 5y ago
> Integration tests are supposed to be less fragile because they don't rely on specific implementation details or mocks I think you misunderstand what I mean by fragility. By fragility I mean a few things. First, it's sensitive to conditions other than the logic under test. Imagine I have an integration test that round trips to a database. What if there is a momentary TCP/IP burp and the connection drops? The test will fail. Nothing with the wrong the test; nothing wrong with the logic, yet it still fails. Then under different circumstances same test/same logic passes. What do now? This is a fragile test. Second, fragility can happen when the outcome of a test is not deterministic generally because the function under test is not referentially transparent. Good functions are referentially transparent by definition. Mocks can turn non-RF functions into RF functions because you can exercise complete control over the mock's return value. Integration suites struggle to embody this property. If you are careful with setup data you can get around this but its tedious. If you are querying external systems then good luck you are at the mercy of whatever data happens to be there. Same deal as number 1. Sometimes the same test will pass or fail without changes to the logic. Fragile. Third, Fragility can mean that small changes have cascading impact. If I write an integration suite that tests a RESTful endpoint and I refactor that endpoint's return type I run the risk of invalidating huge swathes of tests. Here I would be judicious. If the endpoint was critical and didn't churn much I'd probably invest in an integration suite. I cannot fathom a situation where an integration test that requires setup data, database connectivity, possibly network connectivity, specific sequencing, etc. Is going to be less fragile than a test that does nothing but test the specific logic conditions within a function and has no other dependencies. Your style is not invalid I just wouldn't adopt it myself. >This is the opposite of how the test pyramid is supposed to work. Where I'm from we'd call this having a case of the "sposedas". Nothing is "supposed" to work one way or another. There is no universal truth here despite what conference speakers would have you believe. There is only the trade-offs implied between a specific course of action vs. another.

SassyGrapefruit 5y ago

> Integration tests are supposed to be less fragile because they don't rely on specific implementation details or mocks I think you misunderstand what I mean by fragility. By fragility I mean a few things. First, it's sensitive to conditions other than the logic under test. Imagine I have an integration test that round trips to a database. What if there is a momentary TCP/IP burp and the connection drops? The test will fail. Nothing with the wrong the test; nothing wrong with the logic, yet it still fails. Then under different circumstances same test/same logic passes. What do now? This is a fragile test. Second, fragility can happen when the outcome of a test is not deterministic generally because the function under test is not referentially transparent. Good functions are referentially transparent by definition. Mocks can turn non-RF functions into RF functions because you can exercise complete control over the mock's return value. Integration suites struggle to embody this property. If you are careful with setup data you can get around this but its tedious. If you are querying external systems then good luck you are at the mercy of whatever data happens to be there. Same deal as number 1. Sometimes the same test will pass or fail without changes to the logic. Fragile. Third, Fragility can mean that small changes have cascading impact. If I write an integration suite that tests a RESTful endpoint and I refactor that endpoint's return type I run the risk of invalidating huge swathes of tests. Here I would be judicious. If the endpoint was critical and didn't churn much I'd probably invest in an integration suite. I cannot fathom a situation where an integration test that requires setup data, database connectivity, possibly network connectivity, specific sequencing, etc. Is going to be less fragile than a test that does nothing but test the specific logic conditions within a function and has no other dependencies. Your style is not invalid I just wouldn't adopt it myself. >This is the opposite of how the test pyramid is supposed to work. Where I'm from we'd call this having a case of the "sposedas". Nothing is "supposed" to work one way or another. There is no universal truth here despite what conference speakers would have you believe. There is only the trade-offs implied between a specific course of action vs. another.