3 ms·

I'll take umbrage with your last point: "Look for invariants, and test those. Things that should always be true. Often times, there are multiple acceptable res

I'll take umbrage with your last point: "Look for invariants, and test those. Things that should always be true. Often times, there are multiple acceptable results - avoid exact-match tests when that happens (e.g. if you make a speech-to-text system, don't test that audio clip X produces exact output Y; often times, a genuine improvement in the algorithm might break some of your tests)." No, you should test for exactly the output you have coded to generate. Otherwise, you do not know when you have behavior regression. You would expect to have to update the text-to-speech tests when you modify the text-to-speech algorithm. But if you're modifying another algorithm, and you start seeing tests break in the text-to-speech algorithm, you're probably introducing a bug! A failed test means nothing other than the fact that you have changed behaviour -- and should therefore trigger on any behavioural change. It's your opportunity to vet the expectations of your changes against the actual behaviour of the changed system.

virgilp 8y ago
I respectfully disagree. A test should fail, ideally, only when behavior change is undesirable (i.e. contracts are broken). Optimizations, new features etc. should not break existing tests, unless old functionality was affected. And then there's the whole thing about separating functional from performance concerns - even degraded performance shouldn't fail the functional tests. In fact, the example I gave was real-life - a friend from Google changed their speech recognition tests to avoid exact matches and it was a significant improvement in the life & productivity of the development team. [edit] There's also another damaging aspect of exact-match tests: they often test much more than what's intended. Take for instance a file conversion software (say from PDF to HTML). You add a feature to support SVG, and test it with various SVG shapes - it's easy & tempting to just run the software on an input PDF, check that the output HTML looks right (especially in the relevant SVG parts), and then add an exact-match test. Job done, yay! Except that, you do this a lot, and it will slow you down like hell. Because when it fails in the future, it's very hard to tell why (was the SVG conversion broken? or is it some unrelated thing, like a different but valid way to produce the output HTML?). Do this a lot and you won't be able to trust your tests anymore - any change and 400 of them fail, ain't nobody got time to check in depth what happened, "it's probably just a harmless change, let me take a cursory look and then I'll just update the reference to be the new output".
- jdmichal 8y ago
  You're building a bit of a straw man. If you have 400 tests that fail with a single behavioural change, why are you testing the same thing 400 times? And you don't need an in depth investigation unless you didn't expect the test to break. And if you did expect the test to break, then you ensure that the test broke in the correct place. If a cursory glance is all you need in order to confirm that, then that's all you need. Tests are there to tell you exactly what actually changed in behaviour. The only time this should be a surprise is if you don't have a functional mental model of your code, in which case it's doubly important that you be made aware of what your changes are actually doing. In your Google example, would their tests fail if their algorithm regressed in behaviour? If it doesn't fail on minor improvements, I don't see how they would fail on minor regressions either.
  - virgilp 8y ago
    400 is an arbitrary number, but it's what sometimes (often?) happens with exact-match tests; take the second example with the PDF-to-HTML converter, an exact match tests would test too much, and thus your SVG tests will fail when nothing SVG-specific changed (maybe the way you rendered the HTML header changed). Or maybe you changed the order your HTML renderer uses to render child nodes, and it's still a valid order in 99% of your cases, but it breaks 100% of your tests. How do you identify the 1% that are broken? It's very hard if your tests just do exact textual comparison, instead of verifying isolated, relevant properties of interest. In my Google example, the problem is that functional tests were testing something that should've been a performance aspect. The way you identify minor regressions is by having a suite of performance/ accuracy tests, where you track that accuracy is trending upwards across various classes of input. Those are not functional tests - any individual sample may fail and it's not a big deal if it does. Sometimes a minor regression is actually acceptable (e.g. if the runtime performance/ resource consumption improved a lot).
    - jdmichal 8y ago
      > It's very hard if your tests just do exact textual comparison, instead of verifying isolated, relevant properties of interest. I think you have this assumption that you never actually specified that exact-match testing means testing for an exact match on the entire payload. That's a strawman, and yes you will have issues exactly like you describe. If your test is only meant to cover the SVG translation, then you should be isolating the SVG-specific portion of the payload. But then execute an exact match on that isolated translation. Now that test only breaks in two ways: It fails to isolate the SVG, or the SVG translation behaviour changes. > In my Google example, the problem is that functional tests were testing something that should've been a performance aspect. The way you identify minor regressions is by having a suite of performance/ accuracy tests, where you track that accuracy is trending upwards across various classes of input. Those are not functional tests - any individual sample may fail and it's not a big deal if it does. Sometimes a minor regression is actually acceptable (e.g. if the runtime performance/ resource consumption improved a lot). ... "Accuracy", aka the output of your functionality is a non-functional test? What? And I never said regressions aren't acceptable. I said that you should know via your test suite that the regression happened! You are phrasing it as a trade-off, but also apparently advocating an approach where you don't even know about the regression! It's not a trade-off if you are just straight-up unaware that there's downsides.

virgilp 8y ago

I respectfully disagree. A test should fail, ideally, only when behavior change is undesirable (i.e. contracts are broken). Optimizations, new features etc. should not break existing tests, unless old functionality was affected. And then there's the whole thing about separating functional from performance concerns - even degraded performance shouldn't fail the functional tests. In fact, the example I gave was real-life - a friend from Google changed their speech recognition tests to avoid exact matches and it was a significant improvement in the life & productivity of the development team. [edit] There's also another damaging aspect of exact-match tests: they often test much more than what's intended. Take for instance a file conversion software (say from PDF to HTML). You add a feature to support SVG, and test it with various SVG shapes - it's easy & tempting to just run the software on an input PDF, check that the output HTML looks right (especially in the relevant SVG parts), and then add an exact-match test. Job done, yay! Except that, you do this a lot, and it will slow you down like hell. Because when it fails in the future, it's very hard to tell why (was the SVG conversion broken? or is it some unrelated thing, like a different but valid way to produce the output HTML?). Do this a lot and you won't be able to trust your tests anymore - any change and 400 of them fail, ain't nobody got time to check in depth what happened, "it's probably just a harmless change, let me take a cursory look and then I'll just update the reference to be the new output".

jdmichal 8y ago
You're building a bit of a straw man. If you have 400 tests that fail with a single behavioural change, why are you testing the same thing 400 times? And you don't need an in depth investigation unless you didn't expect the test to break. And if you did expect the test to break, then you ensure that the test broke in the correct place. If a cursory glance is all you need in order to confirm that, then that's all you need. Tests are there to tell you exactly what actually changed in behaviour. The only time this should be a surprise is if you don't have a functional mental model of your code, in which case it's doubly important that you be made aware of what your changes are actually doing. In your Google example, would their tests fail if their algorithm regressed in behaviour? If it doesn't fail on minor improvements, I don't see how they would fail on minor regressions either.
- virgilp 8y ago
  400 is an arbitrary number, but it's what sometimes (often?) happens with exact-match tests; take the second example with the PDF-to-HTML converter, an exact match tests would test too much, and thus your SVG tests will fail when nothing SVG-specific changed (maybe the way you rendered the HTML header changed). Or maybe you changed the order your HTML renderer uses to render child nodes, and it's still a valid order in 99% of your cases, but it breaks 100% of your tests. How do you identify the 1% that are broken? It's very hard if your tests just do exact textual comparison, instead of verifying isolated, relevant properties of interest. In my Google example, the problem is that functional tests were testing something that should've been a performance aspect. The way you identify minor regressions is by having a suite of performance/ accuracy tests, where you track that accuracy is trending upwards across various classes of input. Those are not functional tests - any individual sample may fail and it's not a big deal if it does. Sometimes a minor regression is actually acceptable (e.g. if the runtime performance/ resource consumption improved a lot).
  - jdmichal 8y ago
    > It's very hard if your tests just do exact textual comparison, instead of verifying isolated, relevant properties of interest. I think you have this assumption that you never actually specified that exact-match testing means testing for an exact match on the entire payload. That's a strawman, and yes you will have issues exactly like you describe. If your test is only meant to cover the SVG translation, then you should be isolating the SVG-specific portion of the payload. But then execute an exact match on that isolated translation. Now that test only breaks in two ways: It fails to isolate the SVG, or the SVG translation behaviour changes. > In my Google example, the problem is that functional tests were testing something that should've been a performance aspect. The way you identify minor regressions is by having a suite of performance/ accuracy tests, where you track that accuracy is trending upwards across various classes of input. Those are not functional tests - any individual sample may fail and it's not a big deal if it does. Sometimes a minor regression is actually acceptable (e.g. if the runtime performance/ resource consumption improved a lot). ... "Accuracy", aka the output of your functionality is a non-functional test? What? And I never said regressions aren't acceptable. I said that you should know via your test suite that the regression happened! You are phrasing it as a trade-off, but also apparently advocating an approach where you don't even know about the regression! It's not a trade-off if you are just straight-up unaware that there's downsides.