7 ms·

> There are functional tests that exercise these classes working together. Fixing a bug in one class breaks many functional tests .. Huh? If fixing a bug brea

> There are functional tests that exercise these classes working together. Fixing a bug in one class breaks many functional tests .. Huh? If fixing a bug breaks a functional test, then either the test was correct and you changed your expected system behavior and the test saved your bacon, or the test was a change detector test not a functional test. Saying "remove functional years" is just blinding yourself. A unit test will never tell you if the class you are changing has any relationship to the actual bug reported in production. It seems like OP has never worked on a large system, or doesn't know about fakes (not mentioned in the article) to run end-to-end functional tests quickly in memory.

spawarotti 6y ago
Came here to say exactly that. The author is advocating for mockist style of unit testing, which is quite terrible for many, many reasons. What you are describing is classic unit testing, and this is the way to go. Good article and book about it: - https://enterprisecraftsmanship.com/posts/unit-testing-dependencies https://enterprisecraftsmanship.com/posts/unit-testing-depen... - Unit Testing Principles, Practices, and Patterns
- pydry 6y ago
  Main reason it it tests unrealistically, I think. The other issues (doesnt catch bugs, hard to maintain) tend to flow from that.
- gwbas1c 6y ago
  > The author is advocating for mockist style of unit testing IMO: "mockist" style of unit tests are best for corner cases that can only be tested in isolation. For example, error conditions that are important to verify, but so difficult or impossible to test that the only way to test them is to test a single class in isolation.
asdfasgasdgasdg 6y ago
Yep, I've been a professional programmer for about twelve years now working in all kinds of large scale systems at a certain internet company. I can say definitively that I've been saved by functional and end to end tests far more often than I have been saved by unit tests. Not that you shouldn't do both, but err on the side of writing tests that will find bugs in the entire system.
- Sodman 6y ago
  > err on the side of writing tests that will find bugs in the entire system While I agree these tests are going to be closer to your real production system, they also usually take longer to write, and take way longer to run. I can run thousands of unit tests and still have them be an order of magnitude quicker than a full e2e test which requires spinning up a k8s cluster, for example. I agree that both have their place, but I'd err on the side of writing tests that will run quicker where possible so that my build pipeline doesn't take forever (and eventually cost a fortune in compute power).
  - exdsq 6y ago
    Just run longer e2e tests nightly and have unit/integration tests in the normal CI checks.
    - Sodman 6y ago
      My problem with this is that now every single production release is either pushed back by at least a day (multiple days if issues are found), OR there's code running in production that hasn't run through e2e tests yet. Plus it adds a layer of complexity to your version control. I'm assuming you're running these e2e tests against a branch that's up to date with the latest master. What happens if I have 4 PRs I need to merge - they all might pass tests when run against the current master but as soon as I merge one, the other 3 haven't technically been tested with master anymore, so I'll have to re-run e2e, which pushes everything back by another day (per PR)?
      - asdfasgasdgasdg 6y ago
        > is either pushed back by at least a day (multiple days if issues are found), Which is better than issues making it to production -- presumably what would have happened if you didn't have the tests. > OR there's code running in production that hasn't run through e2e tests yet. This situation should never occur except if there's a production emergency causing immediate loss of revenue. Even then, the proper answer is usually to roll back to a known-good version. > I'm assuming you're running these e2e tests against a branch that's up to date with the latest master. What happens if I have 4 PRs I need to merge - they all might pass tests when run against the current master but as soon as I merge one, the other 3 haven't technically been tested with master anymore, so I'll have to re-run e2e, which pushes everything back by another day (per PR)? The way releases should work is that periodically you cut a release in a new release branch. After that, further changes to master don't affect the release. All the e2e tests need to run against the release branch before it can be deployed. It should be deployed immediately to your test realm and given time to soak. If bugs are found, minimal fixes should be made against master and then cherry-picked into the release branch, followed by a rerun of tests and redeployment to the test realm. You should aim for a zero-cherry-pick situation. Do this by enabling features and changes via configuration rather than via binary changes. Once the release branch reaches the appropriate level of stability, it should be gradually rolled out, zone by zone, to production. Zones should have some soak time before further zones are touched. The response to production anomalies should be a roll back to the previous version. This applies to big, complex services that are mission-critical. For small, less critical services, a daily/push-on-green type of approach might be fine. But for such services testing should be much easier and it shouldn't be a huge imposition to run all tests before each release.
danShumway 6y ago
I think you're arguing about definitions. If you define a functional test as perfect, that it will never break unless the actual functionality changes, then sure, functional tests are purely good. But that's not what most people end up writing when they try to write functional tests. They don't get them right, they write code-change tests. I've seen some large systems crippled by code-change tests disguised as other types -- functional/unit tests that rely on implementation details that aren't related to the actual behavior being tested, and that make it harder to refactor code, not easier. My criticism with the author isn't that I think they're wrong about the dangers of functional tests when poorly written, it's that the author seems to be over-eager to recommend unit tests as the solution. Unit tests can be just as problematic if they're poorly written; in fact they can be even more problematic if they're relying on a lot of fragile mocks. One of the biggest issues I see in large codebases maintained by large teams is people building single-purpose mocks for each test that rely on implementation details. This is especially common when team members are mocking components that they haven't built or that don't have clearly defined public interfaces. The end result is that people end up building tests that are too 'specific' -- they rely on behaviors that are subject to change that we never intended to test in the first place (like translation strings, styling, results from 3rd-party data sources, etc). This slows down development, but more importantly, it actually masks bugs because after a while people start assuming that all code changes will cause tests to fail. You can say that those aren't real functional/unit tests if you want, you can say that they're poorly written so they don't count. It doesn't matter -- that they're poorly written is the point. People write bad tests when they aren't trained how to write good tests, and an organizational policy that only focuses on increasing test coverage without focusing on the quality of tests being written is bound to fail in the long term. We spend a lot of energy in some orgs training people to write tests without training them on how to write tests, or conceptually talking about what a unit/functional test is supposed to accomplish. We don't have good education in this area, especially on platforms like the web. In the worst case, I've even seen managers argue that testing static constant files and translation strings was good policy because, "forcing devs to update a constant in two or three files instead of one will mean that they think harder about the change." Needless to say, that policy did not end up reducing bugs.
- pydry 6y ago
nullspace 6y ago
My initial reaction was to downvote you for your last sentence - I think OP has obviously worked on medium-large systems, and is just strongly opinionated. However, I agree with your first sentence. > If fixing a bug breaks a functional test, then either the test was correct and you changed your expected system behavior and the test saved your bacon, or the test was a change detector test not a functional test. And I think this is a _hugely_ valuable for medium-to-large systems, which typically have many, many layers of abstractions. I used to work on one such system that had a lot of, what this author would refer to as, functional tests. Sometimes, changing a couple of lines in one of the core modules would cause 50 tests to fail. While this is just a "change detector", it's still really valuable, say about 20/50 of these changes in behavior are unexpected but acceptable, 5/50 are actually intended, but 25/50 of these are actually regressions in ideal behavior. This means the layer at which you made the change is just wrong, or you didn't put sufficient nuance in your change. It's a big time-sink for sure and made many new developers angry, but honestly the gain was that the overall product had fewer regressions. I'm somewhat unsure what I would replace these "functional tests" with.
gwbas1c 6y ago
>> There are functional tests that exercise these classes working together. Fixing a bug in one class breaks many functional tests .. > Huh? If fixing a bug breaks a functional test, then either the test was correct and you changed your expected system behavior and the test saved your bacon, or the test was a change detector test not a functional test. I think it's more a symptom of fragile mocking, or just making a unit too small. IMO, when I was in that situation it was because the mocking framework was too fragile. In hindsight, it may have been useful to just have a library of hand-written mock classes instead of relying on a framework to do it for us.
outworlder 6y ago
> or the test was a change detector test not a functional test. Or the test was flat out broken and was testing implementation detail minutiae that don't really matter. As in, in a strange limbo between unit and functional tests. I've seen this many times. > It seems like OP has never worked on a large system That's an extreme logic leap + ad hominem in one. You got that from just one article?

spawarotti 6y ago

Came here to say exactly that. The author is advocating for mockist style of unit testing, which is quite terrible for many, many reasons. What you are describing is classic unit testing, and this is the way to go. Good article and book about it: - https://enterprisecraftsmanship.com/posts/unit-testing-dependencies https://enterprisecraftsmanship.com/posts/unit-testing-depen... - Unit Testing Principles, Practices, and Patterns

pydry 6y ago
Main reason it it tests unrealistically, I think. The other issues (doesnt catch bugs, hard to maintain) tend to flow from that.
gwbas1c 6y ago
> The author is advocating for mockist style of unit testing IMO: "mockist" style of unit tests are best for corner cases that can only be tested in isolation. For example, error conditions that are important to verify, but so difficult or impossible to test that the only way to test them is to test a single class in isolation.

asdfasgasdgasdg 6y ago

Yep, I've been a professional programmer for about twelve years now working in all kinds of large scale systems at a certain internet company. I can say definitively that I've been saved by functional and end to end tests far more often than I have been saved by unit tests. Not that you shouldn't do both, but err on the side of writing tests that will find bugs in the entire system.

Sodman 6y ago
> err on the side of writing tests that will find bugs in the entire system While I agree these tests are going to be closer to your real production system, they also usually take longer to write, and take way longer to run. I can run thousands of unit tests and still have them be an order of magnitude quicker than a full e2e test which requires spinning up a k8s cluster, for example. I agree that both have their place, but I'd err on the side of writing tests that will run quicker where possible so that my build pipeline doesn't take forever (and eventually cost a fortune in compute power).
- exdsq 6y ago
  Just run longer e2e tests nightly and have unit/integration tests in the normal CI checks.
  - Sodman 6y ago
    My problem with this is that now every single production release is either pushed back by at least a day (multiple days if issues are found), OR there's code running in production that hasn't run through e2e tests yet. Plus it adds a layer of complexity to your version control. I'm assuming you're running these e2e tests against a branch that's up to date with the latest master. What happens if I have 4 PRs I need to merge - they all might pass tests when run against the current master but as soon as I merge one, the other 3 haven't technically been tested with master anymore, so I'll have to re-run e2e, which pushes everything back by another day (per PR)?
    - asdfasgasdgasdg 6y ago
      > is either pushed back by at least a day (multiple days if issues are found), Which is better than issues making it to production -- presumably what would have happened if you didn't have the tests. > OR there's code running in production that hasn't run through e2e tests yet. This situation should never occur except if there's a production emergency causing immediate loss of revenue. Even then, the proper answer is usually to roll back to a known-good version. > I'm assuming you're running these e2e tests against a branch that's up to date with the latest master. What happens if I have 4 PRs I need to merge - they all might pass tests when run against the current master but as soon as I merge one, the other 3 haven't technically been tested with master anymore, so I'll have to re-run e2e, which pushes everything back by another day (per PR)? The way releases should work is that periodically you cut a release in a new release branch. After that, further changes to master don't affect the release. All the e2e tests need to run against the release branch before it can be deployed. It should be deployed immediately to your test realm and given time to soak. If bugs are found, minimal fixes should be made against master and then cherry-picked into the release branch, followed by a rerun of tests and redeployment to the test realm. You should aim for a zero-cherry-pick situation. Do this by enabling features and changes via configuration rather than via binary changes. Once the release branch reaches the appropriate level of stability, it should be gradually rolled out, zone by zone, to production. Zones should have some soak time before further zones are touched. The response to production anomalies should be a roll back to the previous version. This applies to big, complex services that are mission-critical. For small, less critical services, a daily/push-on-green type of approach might be fine. But for such services testing should be much easier and it shouldn't be a huge imposition to run all tests before each release.

danShumway 6y ago

I think you're arguing about definitions. If you define a functional test as perfect, that it will never break unless the actual functionality changes, then sure, functional tests are purely good. But that's not what most people end up writing when they try to write functional tests. They don't get them right, they write code-change tests. I've seen some large systems crippled by code-change tests disguised as other types -- functional/unit tests that rely on implementation details that aren't related to the actual behavior being tested, and that make it harder to refactor code, not easier. My criticism with the author isn't that I think they're wrong about the dangers of functional tests when poorly written, it's that the author seems to be over-eager to recommend unit tests as the solution. Unit tests can be just as problematic if they're poorly written; in fact they can be even more problematic if they're relying on a lot of fragile mocks. One of the biggest issues I see in large codebases maintained by large teams is people building single-purpose mocks for each test that rely on implementation details. This is especially common when team members are mocking components that they haven't built or that don't have clearly defined public interfaces. The end result is that people end up building tests that are too 'specific' -- they rely on behaviors that are subject to change that we never intended to test in the first place (like translation strings, styling, results from 3rd-party data sources, etc). This slows down development, but more importantly, it actually masks bugs because after a while people start assuming that all code changes will cause tests to fail. You can say that those aren't real functional/unit tests if you want, you can say that they're poorly written so they don't count. It doesn't matter -- that they're poorly written is the point. People write bad tests when they aren't trained how to write good tests, and an organizational policy that only focuses on increasing test coverage without focusing on the quality of tests being written is bound to fail in the long term. We spend a lot of energy in some orgs training people to write tests without training them on how to write tests, or conceptually talking about what a unit/functional test is supposed to accomplish. We don't have good education in this area, especially on platforms like the web. In the worst case, I've even seen managers argue that testing static constant files and translation strings was good policy because, "forcing devs to update a constant in two or three files instead of one will mean that they think harder about the change." Needless to say, that policy did not end up reducing bugs.

pydry 6y ago

nullspace 6y ago

My initial reaction was to downvote you for your last sentence - I think OP has obviously worked on medium-large systems, and is just strongly opinionated. However, I agree with your first sentence. > If fixing a bug breaks a functional test, then either the test was correct and you changed your expected system behavior and the test saved your bacon, or the test was a change detector test not a functional test. And I think this is a _hugely_ valuable for medium-to-large systems, which typically have many, many layers of abstractions. I used to work on one such system that had a lot of, what this author would refer to as, functional tests. Sometimes, changing a couple of lines in one of the core modules would cause 50 tests to fail. While this is just a "change detector", it's still really valuable, say about 20/50 of these changes in behavior are unexpected but acceptable, 5/50 are actually intended, but 25/50 of these are actually regressions in ideal behavior. This means the layer at which you made the change is just wrong, or you didn't put sufficient nuance in your change. It's a big time-sink for sure and made many new developers angry, but honestly the gain was that the overall product had fewer regressions. I'm somewhat unsure what I would replace these "functional tests" with.

gwbas1c 6y ago

>> There are functional tests that exercise these classes working together. Fixing a bug in one class breaks many functional tests .. > Huh? If fixing a bug breaks a functional test, then either the test was correct and you changed your expected system behavior and the test saved your bacon, or the test was a change detector test not a functional test. I think it's more a symptom of fragile mocking, or just making a unit too small. IMO, when I was in that situation it was because the mocking framework was too fragile. In hindsight, it may have been useful to just have a library of hand-written mock classes instead of relying on a framework to do it for us.

outworlder 6y ago

> or the test was a change detector test not a functional test. Or the test was flat out broken and was testing implementation detail minutiae that don't really matter. As in, in a strange limbo between unit and functional tests. I've seen this many times. > It seems like OP has never worked on a large system That's an extreme logic leap + ad hominem in one. You got that from just one article?