5 ms·

Which by definition means your tests are legacy code. It's turtles all the way down. For what it's worth I do think Unit Tests provide value.

by zaphar 9y ago

chewbacha 9y ago
The code is the “test” for your tests ;)
- michaelmrose 9y ago
  The code and the test are probably to some degree based on the same logical construct. If that logical construct is malformed your tests may pass and your code might not work. Regardless if your code works your tests could still fail, likewise your code could not work and your tests pass so since the tests and code may vary independently if either is buggy I don't see how this can possibly be true.
  - chewbacha 9y ago
    If the logical construct is flawed, no amount of tests will ever catch it, only user testing would. The point I’m trying to make is that the verification of correctness is bidirectional. If I’m writing tests around existing code, I rely on the code to test my expectations. If I’m writing new code, I write tests to assert my new assumptions. All automated tests do are assert that I’ve written the same logic twice. Writing it a third time will also decrease the likelihood of transcription error again, but at diminished returns.
Ajedi32 9y ago
Nah, the code tests the tests just as the tests test the code. If you expect your tests to pass and they don't, something's wrong. (Either with the code, or with the tests.) Same goes for when you expect the tests to fail and they don't.
- the_af 9y ago
  The code is definitely NOT the test of the tests. It may seem that way because (sometimes) when the code breaks (e.g. fails to run at all), the tests also break. But consider this: what if your tests are poorly written and fail to detect bugs? What if they fail because the test code is buggy? Failing to detect bugs is a bug (undesirable behavior/output) of test code. There are some techniques to address this, for example mutation testing ( https://en.wikipedia.org/wiki/Mutation_testing https://en.wikipedia.org/wiki/Mutation_testing ), which effectively become some part of "the tests of the tests". The take away: no, the code is not enough. You need to test your tests.
  - Vinnl 9y ago
    Note that mutation testing is mostly just an advanced (and more brittle) form of code coverage. The flip side is that code coverage reports can also be considered tests of your tests. Not exhaustive, but I'd argue that unit tests hardly ever are as well - and that's OK.
    - the_af 9y ago
      I disagree. Coverage tests how much of your code is exercised by your tests. Mutation testing is (one way) of testing how effective your tests are -- i.e. how sensitive they are to bugs in your code. It follows that good tests must have both good coverage and good effectiveness, but the two are not the same.
      - Vinnl 9y ago
        The former seems to me like a (less comprehensive but also less brittle) way of doing the latter. Higher coverage usually means that your tests are more sensitive to bugs in your code - i.e. if a bug occurs in uncovered code, your tests won't catch it. Of course they're not literally the same, but both are ways of measuring how good your tests are at catching bugs in your code. It might sound pedantic, but this way of thinking shows you that it's not turtles all the way down: if they'd really be "tests of the tests", then you might also want "tests of the tests of the tests". And since we obviously don't want to keep doing that, you might conclude that we don't need "tests of our tests" as well. But since they're simply measurements of test quality, we can see that both test coverage and mutation testing can be useful and not a never-ending story.

Which by definition means your tests are legacy code. It's turtles all the way down. For what it's worth I do think Unit Tests provide value.

by zaphar 9y ago

Which by definition means your tests are legacy code. It's turtles all the way down. For what it's worth I do think Unit Tests provide value.

chewbacha 9y ago

The code is the “test” for your tests ;)

michaelmrose 9y ago
The code and the test are probably to some degree based on the same logical construct. If that logical construct is malformed your tests may pass and your code might not work. Regardless if your code works your tests could still fail, likewise your code could not work and your tests pass so since the tests and code may vary independently if either is buggy I don't see how this can possibly be true.
- chewbacha 9y ago
  If the logical construct is flawed, no amount of tests will ever catch it, only user testing would. The point I’m trying to make is that the verification of correctness is bidirectional. If I’m writing tests around existing code, I rely on the code to test my expectations. If I’m writing new code, I write tests to assert my new assumptions. All automated tests do are assert that I’ve written the same logic twice. Writing it a third time will also decrease the likelihood of transcription error again, but at diminished returns.

Ajedi32 9y ago

Nah, the code tests the tests just as the tests test the code. If you expect your tests to pass and they don't, something's wrong. (Either with the code, or with the tests.) Same goes for when you expect the tests to fail and they don't.

the_af 9y ago
The code is definitely NOT the test of the tests. It may seem that way because (sometimes) when the code breaks (e.g. fails to run at all), the tests also break. But consider this: what if your tests are poorly written and fail to detect bugs? What if they fail because the test code is buggy? Failing to detect bugs is a bug (undesirable behavior/output) of test code. There are some techniques to address this, for example mutation testing ( https://en.wikipedia.org/wiki/Mutation_testing https://en.wikipedia.org/wiki/Mutation_testing ), which effectively become some part of "the tests of the tests". The take away: no, the code is not enough. You need to test your tests.
- Vinnl 9y ago
  Note that mutation testing is mostly just an advanced (and more brittle) form of code coverage. The flip side is that code coverage reports can also be considered tests of your tests. Not exhaustive, but I'd argue that unit tests hardly ever are as well - and that's OK.
  - the_af 9y ago
    I disagree. Coverage tests how much of your code is exercised by your tests. Mutation testing is (one way) of testing how effective your tests are -- i.e. how sensitive they are to bugs in your code. It follows that good tests must have both good coverage and good effectiveness, but the two are not the same.
    - Vinnl 9y ago
      The former seems to me like a (less comprehensive but also less brittle) way of doing the latter. Higher coverage usually means that your tests are more sensitive to bugs in your code - i.e. if a bug occurs in uncovered code, your tests won't catch it. Of course they're not literally the same, but both are ways of measuring how good your tests are at catching bugs in your code. It might sound pedantic, but this way of thinking shows you that it's not turtles all the way down: if they'd really be "tests of the tests", then you might also want "tests of the tests of the tests". And since we obviously don't want to keep doing that, you might conclude that we don't need "tests of our tests" as well. But since they're simply measurements of test quality, we can see that both test coverage and mutation testing can be useful and not a never-ending story.