4 ms·

Unit tests can only test certain kinds of behavior of the function under test. To give other kinds of examples, the current implementation might have used a cle

by mcherm 9y ago

Unit tests can only test certain kinds of behavior of the function under test. To give other kinds of examples, the current implementation might have used a clever trick to achieve faster execution (this cannot be evaluated with unit tests). Alternately, it might have been implemented so as to share common functionality with a different piece of code by calling a common subroutine. That sharing of functionality is probably not easy to evaluate with a unit test (nor is it wise), but is still an important property to preserve.

0xcde4c3db 9y ago
The most "fun" subcategory of "how do I write a test for this?" that I've seen is when something in your software triggers a bug in somebody else's product, and "somebody else" is both too big and important to return your calls and too relied-upon for the bug to go unaddressed (e.g. the Oracles and Ciscos of the world). So you have little recourse beyond screwing around with your code until the user stops seeing the bug.
- cptskippy 9y ago
  That happens all the time. I generally encapsulate the logic compensating for the bug and decorate it with copious comments explaining the rational then write a unit test explicitly testing for the buggy behavior. If the bug gets fixed then the test fails and the product doesn't ship until it's resolved. Yes I write unit tests for 3rd party APIs and libraries I consume if they show themselves to be inconsistent or buggy. We actually use a 3rd party API that crashes regularly and we have a suite of unit tests surrounding it so when reports come in we can isolate, reproduce, and verify the problem before bringing fury down upon the vendor. It is also handy during pre and post deployment validation to ensure that the environment your update is working in is 100% functional. I've rolled back releases before because an external resource had an unreported outage that caused our deployment validation to fail. With pre/post test you can test the environment before you even start.
  - ramses0 9y ago
    s/unit tests for 3rd party APIs/ => /acceptance tests for 3rd party APIs/
  - 0xcde4c3db 9y ago
    Those are all fine things to do if you can afford them. In my case, I'm talking about bugs that come sturdily packaged in a $5000+ chassis sitting on a private VLAN in somebody else's building. So it goes.
cptskippy 9y ago
> the current implementation might have used a clever trick to achieve faster execution (this cannot be evaluated with unit tests). Sure it can. I worked on a project with a pubic API with many time sensitive routines and we most definitely had unit tests measuring performance. The test failed if the execution time exceeded the acceptable threshold. > Alternately, it might have been implemented so as to share common functionality with a different piece of code by calling a common subroutine. You can certainly write coherency tests to ensure that two methods are doing what's expected. They might not share the same subroutine any longer but you can throw an exception when their respective results no longer align and have comments on the test explaining the rationale. Honestly if you have this sort of subtle dependency and you don't have a unit test for it then you're just asking for trouble.
- mpweiher 9y ago
  > The test failed if the execution time exceeded the acceptable threshold. In order to not be beholden to specific hardware/execution environment and absolute times, you can also compare an unoptimized version against an optimized version, so the test can be: optimized version must be 2x faster than unoptimized. Of course if you actually have real-time requirements, it is good to test against those.
btown 9y ago
Alternately, if it uses a regular expression or other form of pattern matching, it's very likely that the original coder made unit tests for the most common behaviors, but built in assumptions for edge cases. And I'm unaware of any code coverage tools that help someone test every single branch in a regex. So you're reliant on comments, documentation, etc. (thank goodness for Python's verbose regex mode, for example) to make those assumptions explicit. It's not an easy problem to solve.
- Someone 9y ago
  A gray-box fuzzer such as american fuzzy lop (https://en.wikipedia.org/wiki/American_fuzzy_lop_(fuzzer) https://en.wikipedia.org/wiki/American_fuzzy_lop_(fuzzer) ) can generate quite good test cases. If your regex engine can compile to native code, it can generate tests that test every single branch of the regex.

by mcherm 9y ago

0xcde4c3db 9y ago

The most "fun" subcategory of "how do I write a test for this?" that I've seen is when something in your software triggers a bug in somebody else's product, and "somebody else" is both too big and important to return your calls and too relied-upon for the bug to go unaddressed (e.g. the Oracles and Ciscos of the world). So you have little recourse beyond screwing around with your code until the user stops seeing the bug.

cptskippy 9y ago
That happens all the time. I generally encapsulate the logic compensating for the bug and decorate it with copious comments explaining the rational then write a unit test explicitly testing for the buggy behavior. If the bug gets fixed then the test fails and the product doesn't ship until it's resolved. Yes I write unit tests for 3rd party APIs and libraries I consume if they show themselves to be inconsistent or buggy. We actually use a 3rd party API that crashes regularly and we have a suite of unit tests surrounding it so when reports come in we can isolate, reproduce, and verify the problem before bringing fury down upon the vendor. It is also handy during pre and post deployment validation to ensure that the environment your update is working in is 100% functional. I've rolled back releases before because an external resource had an unreported outage that caused our deployment validation to fail. With pre/post test you can test the environment before you even start.
- ramses0 9y ago
  s/unit tests for 3rd party APIs/ => /acceptance tests for 3rd party APIs/
- 0xcde4c3db 9y ago
  Those are all fine things to do if you can afford them. In my case, I'm talking about bugs that come sturdily packaged in a $5000+ chassis sitting on a private VLAN in somebody else's building. So it goes.

cptskippy 9y ago

> the current implementation might have used a clever trick to achieve faster execution (this cannot be evaluated with unit tests). Sure it can. I worked on a project with a pubic API with many time sensitive routines and we most definitely had unit tests measuring performance. The test failed if the execution time exceeded the acceptable threshold. > Alternately, it might have been implemented so as to share common functionality with a different piece of code by calling a common subroutine. You can certainly write coherency tests to ensure that two methods are doing what's expected. They might not share the same subroutine any longer but you can throw an exception when their respective results no longer align and have comments on the test explaining the rationale. Honestly if you have this sort of subtle dependency and you don't have a unit test for it then you're just asking for trouble.

mpweiher 9y ago
> The test failed if the execution time exceeded the acceptable threshold. In order to not be beholden to specific hardware/execution environment and absolute times, you can also compare an unoptimized version against an optimized version, so the test can be: optimized version must be 2x faster than unoptimized. Of course if you actually have real-time requirements, it is good to test against those.

btown 9y ago

Alternately, if it uses a regular expression or other form of pattern matching, it's very likely that the original coder made unit tests for the most common behaviors, but built in assumptions for edge cases. And I'm unaware of any code coverage tools that help someone test every single branch in a regex. So you're reliant on comments, documentation, etc. (thank goodness for Python's verbose regex mode, for example) to make those assumptions explicit. It's not an easy problem to solve.

Someone 9y ago
A gray-box fuzzer such as american fuzzy lop (https://en.wikipedia.org/wiki/American_fuzzy_lop_(fuzzer) https://en.wikipedia.org/wiki/American_fuzzy_lop_(fuzzer) ) can generate quite good test cases. If your regex engine can compile to native code, it can generate tests that test every single branch of the regex.