4 ms·

Focusing on integration tests: oh, the test fails, something is wrong somewhere... I will debug all layers for the next week, as we simply don't have too many t

Focusing on integration tests: oh, the test fails, something is wrong somewhere... I will debug all layers for the next week, as we simply don't have too many tests covering the internal code. I like to have both.

jondubois 7y ago
Why next week? Why not use the integration tests as part of your debug cycle now? The kinds of integration tests that I write only take a few milliseconds to run so if they fail I can just make a code change (or add a new breakpoint or log), then run again, then repeat. The flow should not be any different from doing TDD with unit tests. Note that integration tests don't necessarily have to be end-to-end; the point is more that each test case should cover a specific behavior.
- adwn 7y ago
  > Why next week? I think pleasecalllater's point is that without unit tests, you'll spend a lot of time just pinpointing the bug, since it could be anywhere in your module (which unit tests would cover), not just at the interface to other modules (which integration tests cover).
  - jondubois 7y ago
    That's not my experience. Usually it only takes me a few minutes to fix a bug once it has been caught by an integration test.
    - pleasecalllater 7y ago
      Depends on your system. Mine usually are huge. The integration test showing "this json reply is wrong" doesn't tell me anything else than I have to dig through all the layers.
      - jondubois 7y ago
        Maybe you should break up that large system into more manageable chunks (e.g. microservices, engines, ...) and integration test those individually.
  - Ace17 7y ago
    If your tests: - run fast (<10s) - run on your local machine - run on your local code (including uncomitted diffs) ... whether they're called "integration" or "unit" tests doesn't matter, as they will detect regressions at the earliest possible time. Unit tests might guarantee smaller code search area (because maybe the failing test only executed 1% of the code), but if running "git stash" makes all the tests pass again, I have a pretty good confidence about where the error lies.
- coldtea 7y ago
  >The kinds of integration tests that I write only take a few milliseconds to run So your codebase probably doesn't solve the problems and use the stacks most of us solve/use.
  - mpweiher 7y ago
    > your codebase doesn't solve the problems most of us solve Why not? > and use the stacks most of us use Probably. Or at least not use them the same way. I am consistently amazed by the time even unit tests take when I join projects. If they have them. And those unit tests are usually also at least part broken and very incomplete. I would argue those phenomena are closely related. The unit tests for one of my projects typically run in the ~1 second range, meaning that I can, and do, run them as part of the build process. A build is not complete until the unit tests pass. This includes Postscript/PDF interpreters, so it's not just testing completely trivial stuff. (Just checked: currently 3.9s real for 1498 tests across 9 projects, so a bit on the slow side overall)
    - golergka 7y ago
      How many external build systems and proprietary closed source tech do you have to yous to build your project and run integration tests on it? In my case, full production build that produces the same library takes about 30 minutes using asset cache: re-building everything completely from scratch would take about two hours.
- pleasecalllater 7y ago
  Not "the next week" but "for the next week". My experience shows that debugging pure integration test systems is really time consuming.
jancsika 7y ago
I don't know about integration testing, but it seems like that problem necessarily means important lower level tests are missing. Something like having a bunch of nested switch statements where some cases were left unhandled. In that context, are integration tests checked automatically the same way, say, Rust would error out on unhandled cases? Or is this left up to developer (who I assume is always writing this stuff at the last possible moment with the least possible focus)?

jondubois 7y ago

Why next week? Why not use the integration tests as part of your debug cycle now? The kinds of integration tests that I write only take a few milliseconds to run so if they fail I can just make a code change (or add a new breakpoint or log), then run again, then repeat. The flow should not be any different from doing TDD with unit tests. Note that integration tests don't necessarily have to be end-to-end; the point is more that each test case should cover a specific behavior.

adwn 7y ago
> Why next week? I think pleasecalllater's point is that without unit tests, you'll spend a lot of time just pinpointing the bug, since it could be anywhere in your module (which unit tests would cover), not just at the interface to other modules (which integration tests cover).
- jondubois 7y ago
  That's not my experience. Usually it only takes me a few minutes to fix a bug once it has been caught by an integration test.
  - pleasecalllater 7y ago
    Depends on your system. Mine usually are huge. The integration test showing "this json reply is wrong" doesn't tell me anything else than I have to dig through all the layers.
    - jondubois 7y ago
      Maybe you should break up that large system into more manageable chunks (e.g. microservices, engines, ...) and integration test those individually.
- Ace17 7y ago
  If your tests: - run fast (<10s) - run on your local machine - run on your local code (including uncomitted diffs) ... whether they're called "integration" or "unit" tests doesn't matter, as they will detect regressions at the earliest possible time. Unit tests might guarantee smaller code search area (because maybe the failing test only executed 1% of the code), but if running "git stash" makes all the tests pass again, I have a pretty good confidence about where the error lies.
coldtea 7y ago
>The kinds of integration tests that I write only take a few milliseconds to run So your codebase probably doesn't solve the problems and use the stacks most of us solve/use.
- mpweiher 7y ago
  > your codebase doesn't solve the problems most of us solve Why not? > and use the stacks most of us use Probably. Or at least not use them the same way. I am consistently amazed by the time even unit tests take when I join projects. If they have them. And those unit tests are usually also at least part broken and very incomplete. I would argue those phenomena are closely related. The unit tests for one of my projects typically run in the ~1 second range, meaning that I can, and do, run them as part of the build process. A build is not complete until the unit tests pass. This includes Postscript/PDF interpreters, so it's not just testing completely trivial stuff. (Just checked: currently 3.9s real for 1498 tests across 9 projects, so a bit on the slow side overall)
  - golergka 7y ago
    How many external build systems and proprietary closed source tech do you have to yous to build your project and run integration tests on it? In my case, full production build that produces the same library takes about 30 minutes using asset cache: re-building everything completely from scratch would take about two hours.
pleasecalllater 7y ago
Not "the next week" but "for the next week". My experience shows that debugging pure integration test systems is really time consuming.

jancsika 7y ago

I don't know about integration testing, but it seems like that problem necessarily means important lower level tests are missing. Something like having a bunch of nested switch statements where some cases were left unhandled. In that context, are integration tests checked automatically the same way, say, Rust would error out on unhandled cases? Or is this left up to developer (who I assume is always writing this stuff at the last possible moment with the least possible focus)?