3 ms·

No-one ever said that integration tests and unit tests are mutually exclusive. Integration test the happy paths, small integration suite that proves everything

No-one ever said that integration tests and unit tests are mutually exclusive. Integration test the happy paths, small integration suite that proves everything works together. Unit test the individual units of functionality or behavior, whichever you're more comfortable with. Unit tests aren't supposed to tell you that the whole system works. Unit tests exist for two main reasons: Confidence that you can refactor your code without breaking functionality, and proving that your components are doing what they're supposed to do. Yes you technically can accomplish this with just an integration test suite but you will find refactoring bugs much harder to track down because the test failures aren't telling you exactly what went wrong. You can never have full confidence in a system with just one suite of tests. You really need multiple suites. And when you TDD it all the way down (Integration test first, then work through the stack TDD-ing each part, when done Integration test and all unit tests should pass, and feature is done!) it's not a chore but actually quite enjoyable.

brazzy 12y ago
The problem is that refactoring definitely can also cause components to not do what they're supposed to do outside the happy path.
- jameskilton 12y ago
  Then you're not refactoring. Refactoring is defined as changing how the code is implemented without affecting the functionality / behavior of said code. Also, Unit tests are where you ensure the non-happy-paths are functional (error handling, input robustness, etc).
  - Retric 12y ago
    That's a ridiculous definition, that's like suggesting coding is turning requirements into bug free computer code. Refracting is attempting to replace working code with code better suited to your long term goals.
    - jameskilton 12y ago
      I don't see how your definition of refactoring and mine differ.
      - Retric 12y ago
        In your definition if you introduce or remove a bug your not refactoring. More basically for things like simulations or graphics you may trade accuracy for speed so your code may behave slightly differently, but be much faster. The important bit with refactoring is the goal is not fixing a specific bug, however realistically for any sufficiently large change bugs can and will be both added and removed. Edit: "Typically, refactoring applies a series of standardised basic micro-refactorings, each of which is (usually) a tiny change in a computer program's source code that either preserves the behaviour of the software, or at least does not modify its conformance to functional requirements." http://en.wikipedia.org/wiki/Code_refactoring http://en.wikipedia.org/wiki/Code_refactoring
        jameskilton 12y ago
        I feel we may be getting into bike-shedding territory here. When I'm refactoring, as part of the TDD Red/Green/Refactor cycle, I'm changing how code is implemented, usually to improve the design and readability, such that the functionality stays the same e.g. all of my tests pass without touching them. Anything I gain from the refactoring is bonus on top. Refactoring without introducing bugs is the #1 reason people do TDD vs Test After. But if you don't trust the test suite to catch bugs introduced when refactoring then I'd argue the test suite itself probably isn't that useful and needs work. On a side note I believe the term "refactoring" has been way overused these days. You can change code without it being "refactoring". When I've seen people use the term "refactoring" (and I've caught myself multiple times on this) they mean rewriting, for in most cases this rewriting means changing the code and changing the tests. If you have to change the tests because of changes to the code, that's not refactoring, that's just changing code.
    - twic 12y ago
      The standard definition of refactoring is that it doesn't change behaviour. As Martin Fowler put it: Refactoring is the process of changing a software system in such a way that it does not alter the external behavior of the code yet improves its internal structure. You correctly identified a consequence of this in your later comment: > In your definition if you introduce or remove a bug your not refactoring. That is indeed the case. And also correctly observed that: > realistically for any sufficiently large change bugs can and will be both added and removed. If a change adds or removes a bug, it was not a pure refactoring. It may have been an attempt at a pure refactoring, and it may have been otherwise successful, but the introduction of a change in behaviour means that it was not purely a refactoring. This isn't necessarily a bad thing - i'd rather bugs be removed than not! - but it's possible to distinguish the refactoring and bug-fixing aspects of the change. Ideally, i would like to see those aspects formally separated, for example into separate commits in source control. But even if not, we can at least use the terminology correctly and precisely.
  - jlu 12y ago
    @jameskilton you seem to be very insightful in regard of testing, mind if I ask a couple of questions in the future privately? Thanks.
    - jameskilton 12y ago
      I'm always happy to discuss testing! My personal email is in my profile.
      - jlu 12y ago
        Thanks, much appreciated.
  - brazzy 12y ago
    Well, then by that definition you are not refactoring either, and in fact nobody is refactoring. Because any change can unintentionally change the behavior. That's what we call a bug, and them bugs don't care about your definition. And Unit tests alone cannot ensure that the non-happy-paths are functional.
dreamfactory2 12y ago
> TDD it all the way down (Integration test first, then work through the stack TDD-ing each part, when done Integration test and all unit tests should pass, and feature is done!) Interesting. I think this is probably right and also the reverse of what most people try to do (they start with units and work up)
- ryanjshaw 12y ago
  I find this approach has a number of advantages: - it forces you to flesh out your conceptual/paper design in a deliverables-oriented way and makes it obvious where the gaps are, which gives you a good idea of how ready you are to wrap up design and start development, and in turn can be used to create your work breakdown structure and task prioritisation; sometimes it can result in identifying simplifications upfront (saving you time down the line) - when you have a good idea of the scope and complexity of a solution like this, you're less likely to waste time on the less important stuff - if you're the tech lead on a team, giving your developers the high-level components and tests means they're more likely to get things right (or at least limit the damage they do when they get things wrong) Without component-level unit tests or design, you can easily dive into a solution without thinking it through and really regret it.
falcolas 12y ago
In my experience, if your integration testing is only happy path testing, you're doing integration testing wrong. Integration testing should, at a minimum, include: * Combinatorial testing * i.e. testing all possible input combinations for every pair of two inputs * Fuzz testing * Limit testing * Success path testing * Failure case testing Integration testing will discover more component interaction bugs than unit testing alone, and doing only happy path testing is doing it wrong. On the other hand, I wholly believe that testing done by the developer alone will never catch as many bugs as testing done by both the developer and a QA engineer to the mix will. They think differently than we do, and as such are adept at finding our blind spots.
- jameskilton 12y ago
  We may have a wording definition issue here. I've heard multiple definitions of "integration tests". I usually use them where some people use "acceptance tests", aka, top-down be-a-user type testing (selenium in Firefox, for example). Sorry if there was any confusion there. Agreed having a QA team also go through the app is great for finding things that tests can't catch, like usability errors, or crazy edge cases you didn't think about, etc. I have gotten to work with a good QA team and it's amazing the things they find.

brazzy 12y ago

The problem is that refactoring definitely can also cause components to not do what they're supposed to do outside the happy path.

jameskilton 12y ago
Then you're not refactoring. Refactoring is defined as changing how the code is implemented without affecting the functionality / behavior of said code. Also, Unit tests are where you ensure the non-happy-paths are functional (error handling, input robustness, etc).
- Retric 12y ago
  That's a ridiculous definition, that's like suggesting coding is turning requirements into bug free computer code. Refracting is attempting to replace working code with code better suited to your long term goals.
  - jameskilton 12y ago
    I don't see how your definition of refactoring and mine differ.
    - Retric 12y ago
      In your definition if you introduce or remove a bug your not refactoring. More basically for things like simulations or graphics you may trade accuracy for speed so your code may behave slightly differently, but be much faster. The important bit with refactoring is the goal is not fixing a specific bug, however realistically for any sufficiently large change bugs can and will be both added and removed. Edit: "Typically, refactoring applies a series of standardised basic micro-refactorings, each of which is (usually) a tiny change in a computer program's source code that either preserves the behaviour of the software, or at least does not modify its conformance to functional requirements." http://en.wikipedia.org/wiki/Code_refactoring http://en.wikipedia.org/wiki/Code_refactoring
      - jameskilton 12y ago
        I feel we may be getting into bike-shedding territory here. When I'm refactoring, as part of the TDD Red/Green/Refactor cycle, I'm changing how code is implemented, usually to improve the design and readability, such that the functionality stays the same e.g. all of my tests pass without touching them. Anything I gain from the refactoring is bonus on top. Refactoring without introducing bugs is the #1 reason people do TDD vs Test After. But if you don't trust the test suite to catch bugs introduced when refactoring then I'd argue the test suite itself probably isn't that useful and needs work. On a side note I believe the term "refactoring" has been way overused these days. You can change code without it being "refactoring". When I've seen people use the term "refactoring" (and I've caught myself multiple times on this) they mean rewriting, for in most cases this rewriting means changing the code and changing the tests. If you have to change the tests because of changes to the code, that's not refactoring, that's just changing code.
  - twic 12y ago
    The standard definition of refactoring is that it doesn't change behaviour. As Martin Fowler put it: Refactoring is the process of changing a software system in such a way that it does not alter the external behavior of the code yet improves its internal structure. You correctly identified a consequence of this in your later comment: > In your definition if you introduce or remove a bug your not refactoring. That is indeed the case. And also correctly observed that: > realistically for any sufficiently large change bugs can and will be both added and removed. If a change adds or removes a bug, it was not a pure refactoring. It may have been an attempt at a pure refactoring, and it may have been otherwise successful, but the introduction of a change in behaviour means that it was not purely a refactoring. This isn't necessarily a bad thing - i'd rather bugs be removed than not! - but it's possible to distinguish the refactoring and bug-fixing aspects of the change. Ideally, i would like to see those aspects formally separated, for example into separate commits in source control. But even if not, we can at least use the terminology correctly and precisely.
- jlu 12y ago
  @jameskilton you seem to be very insightful in regard of testing, mind if I ask a couple of questions in the future privately? Thanks.
  - jameskilton 12y ago
    I'm always happy to discuss testing! My personal email is in my profile.
    - jlu 12y ago
      Thanks, much appreciated.
- brazzy 12y ago
  Well, then by that definition you are not refactoring either, and in fact nobody is refactoring. Because any change can unintentionally change the behavior. That's what we call a bug, and them bugs don't care about your definition. And Unit tests alone cannot ensure that the non-happy-paths are functional.

dreamfactory2 12y ago

> TDD it all the way down (Integration test first, then work through the stack TDD-ing each part, when done Integration test and all unit tests should pass, and feature is done!) Interesting. I think this is probably right and also the reverse of what most people try to do (they start with units and work up)

ryanjshaw 12y ago
I find this approach has a number of advantages: - it forces you to flesh out your conceptual/paper design in a deliverables-oriented way and makes it obvious where the gaps are, which gives you a good idea of how ready you are to wrap up design and start development, and in turn can be used to create your work breakdown structure and task prioritisation; sometimes it can result in identifying simplifications upfront (saving you time down the line) - when you have a good idea of the scope and complexity of a solution like this, you're less likely to waste time on the less important stuff - if you're the tech lead on a team, giving your developers the high-level components and tests means they're more likely to get things right (or at least limit the damage they do when they get things wrong) Without component-level unit tests or design, you can easily dive into a solution without thinking it through and really regret it.

falcolas 12y ago

In my experience, if your integration testing is only happy path testing, you're doing integration testing wrong. Integration testing should, at a minimum, include: * Combinatorial testing * i.e. testing all possible input combinations for every pair of two inputs * Fuzz testing * Limit testing * Success path testing * Failure case testing Integration testing will discover more component interaction bugs than unit testing alone, and doing only happy path testing is doing it wrong. On the other hand, I wholly believe that testing done by the developer alone will never catch as many bugs as testing done by both the developer and a QA engineer to the mix will. They think differently than we do, and as such are adept at finding our blind spots.

jameskilton 12y ago
We may have a wording definition issue here. I've heard multiple definitions of "integration tests". I usually use them where some people use "acceptance tests", aka, top-down be-a-user type testing (selenium in Firefox, for example). Sorry if there was any confusion there. Agreed having a QA team also go through the app is great for finding things that tests can't catch, like usability errors, or crazy edge cases you didn't think about, etc. I have gotten to work with a good QA team and it's amazing the things they find.