5 ms·

Tests are not meant for ensuring it works with all inputs. You cant simply just throw values at it hoping its all okay. To prove it works with all possible inp

Tests are not meant for ensuring it works with all inputs. You cant simply just throw values at it hoping its all okay. To prove it works with all possible inputs, there are other tools at your disposal.

UweSchmidt 9y ago
What tools would that be? I'd like to hear about real world test scenarios where all possible inputs are tested.
- AstralStorm 9y ago
  Many embedded safety and medical applications prove correctness for all inputs and their combinations. Somme also verify error behaviour. (Out of range.) Granted, this is a relatively small input space, typically a few sensors.
- Kubuxu 9y ago
  One of them would be a fuzz testing. You start with a predefined 'corpus' of inputs which are then modified by the fuzzer to reach yet not covered code paths. In the process it discovers many bugs and crashes. The input fuzzing process is rarely purely random. There are advanced techniques that allow the fuzzer to link input data to conditions of not covered branches. It is quite useful mechanism for checking inputs, formats, behaviour patterns (if you have two solutions but one model, one simple that works 100% but is slowish and one more complex but very fast). See: https://github.com/dvyukov/go-fuzz#trophies https://github.com/dvyukov/go-fuzz#trophies and http://lcamtuf.coredump.cx/afl/ http://lcamtuf.coredump.cx/afl/
  - kasparsklavins 9y ago
    Fuzzing is essentially the same as just throwing values at it hoping its all okay. What I meant was what @AstralStorm said about mathematical and logic proofs that it works for all defined values.
    - jacques_chester 9y ago
      Fuzz testing is useful at testing whole systems in vivo. Proofs are powerful, but you can still make errors in the proof or the transcription into code. And of course you don't know what other emergent behaviours will surprise you when the proved code interacts with unproved code. Depending on your means and needs, you want to try both.
  - nuclx 9y ago
    Parent was talking about full coverage. While fuzzers are great tools, able to generate test inputs for easily coverable parts of the code, formally correct software has to be designed for provability.
- dahauns 9y ago
  In practice, testing _all_ possible inputs is usually not a feasible goal, and often downright impossible (since it would mean solving the halting problem). While there are some languages and problem domains where formal verification methods are a possibility, for most of software development the best you can do is reducing solution space. A sane type system would be a start for example. And it's no coincidence that functional programming and practices with emphasis on immutability are on the rise; Rusts ownership system is a direct consequence as well. TDD _is_ important, if simply for enabling well-factored code and somewhat guarding against regression bugs. But - decades after Dijkstras statement (which someone has already posted in this thread) - the code coverage honeymoon finally seems to be over.
- _pmf_ 9y ago
  Polyspace [0] allows full state space checking. Usually, this is feasible because the kind of modules you test are heavily constrained 12 bit (or fewer) fixed point values (sensor output/ actor input) and contain no algebraic loops. With floats and/or large integers, this quickly becomes unwieldy (verification then takes weeks). [0] https://de.mathworks.com/products/polyspace.html https://de.mathworks.com/products/polyspace.html
Gibbon1 9y ago
I'm reminded of a guy tasked with testing a 32 bit floating point library. After a number of false starts he realized the most effective way possible. Brute force. Oh other story. An OG (original geek) I know once proved the floating point unit on a mainframe was broken via brute force. Bad part meant the decimal floats used by and only by the accounting department sometimes produced erroneous results. Me I have a pitiful amount of unit tests for the main codebase I work on. One module which does have unit tests, also had a bug that resulted in us having to replace a couple of thousand units in the field. Otherwise most of the bugs that chap my keister aren't about procedural correctness, they're about state.
- StavrosK 9y ago
  How do you test it by brute force? How is the algorithm supposed to know what the right thing to return is, unless you rewrite the whole thing, correctly this time? And, if you've done that, just replace the actual function with the test and be done with it.
  - praptak 9y ago
    If a function takes a single 32 bit argument you can run it on every possible value.
    - detaro 9y ago
      … this reply does nothing to answer the parent's question. It's obvious that you can run it on all possible values, but that doesn't answer the question of the return values. (unless you only want to verify "doesn't crash" or something similarly basic)
      - praptak 9y ago
        I believe the question was edited after I wrote my answer. This, or I just misread it - unfortunately HN doesn't let us know. Anyway, I'm providing another answer to the question as it is now.
  - 9y ago

UweSchmidt 9y ago

What tools would that be? I'd like to hear about real world test scenarios where all possible inputs are tested.

AstralStorm 9y ago
Many embedded safety and medical applications prove correctness for all inputs and their combinations. Somme also verify error behaviour. (Out of range.) Granted, this is a relatively small input space, typically a few sensors.
Kubuxu 9y ago
One of them would be a fuzz testing. You start with a predefined 'corpus' of inputs which are then modified by the fuzzer to reach yet not covered code paths. In the process it discovers many bugs and crashes. The input fuzzing process is rarely purely random. There are advanced techniques that allow the fuzzer to link input data to conditions of not covered branches. It is quite useful mechanism for checking inputs, formats, behaviour patterns (if you have two solutions but one model, one simple that works 100% but is slowish and one more complex but very fast). See: https://github.com/dvyukov/go-fuzz#trophies https://github.com/dvyukov/go-fuzz#trophies and http://lcamtuf.coredump.cx/afl/ http://lcamtuf.coredump.cx/afl/
- kasparsklavins 9y ago
  Fuzzing is essentially the same as just throwing values at it hoping its all okay. What I meant was what @AstralStorm said about mathematical and logic proofs that it works for all defined values.
  - jacques_chester 9y ago
    Fuzz testing is useful at testing whole systems in vivo. Proofs are powerful, but you can still make errors in the proof or the transcription into code. And of course you don't know what other emergent behaviours will surprise you when the proved code interacts with unproved code. Depending on your means and needs, you want to try both.
- nuclx 9y ago
  Parent was talking about full coverage. While fuzzers are great tools, able to generate test inputs for easily coverable parts of the code, formally correct software has to be designed for provability.
dahauns 9y ago
In practice, testing _all_ possible inputs is usually not a feasible goal, and often downright impossible (since it would mean solving the halting problem). While there are some languages and problem domains where formal verification methods are a possibility, for most of software development the best you can do is reducing solution space. A sane type system would be a start for example. And it's no coincidence that functional programming and practices with emphasis on immutability are on the rise; Rusts ownership system is a direct consequence as well. TDD _is_ important, if simply for enabling well-factored code and somewhat guarding against regression bugs. But - decades after Dijkstras statement (which someone has already posted in this thread) - the code coverage honeymoon finally seems to be over.
_pmf_ 9y ago
Polyspace [0] allows full state space checking. Usually, this is feasible because the kind of modules you test are heavily constrained 12 bit (or fewer) fixed point values (sensor output/ actor input) and contain no algebraic loops. With floats and/or large integers, this quickly becomes unwieldy (verification then takes weeks). [0] https://de.mathworks.com/products/polyspace.html https://de.mathworks.com/products/polyspace.html

Gibbon1 9y ago

I'm reminded of a guy tasked with testing a 32 bit floating point library. After a number of false starts he realized the most effective way possible. Brute force. Oh other story. An OG (original geek) I know once proved the floating point unit on a mainframe was broken via brute force. Bad part meant the decimal floats used by and only by the accounting department sometimes produced erroneous results. Me I have a pitiful amount of unit tests for the main codebase I work on. One module which does have unit tests, also had a bug that resulted in us having to replace a couple of thousand units in the field. Otherwise most of the bugs that chap my keister aren't about procedural correctness, they're about state.

StavrosK 9y ago
How do you test it by brute force? How is the algorithm supposed to know what the right thing to return is, unless you rewrite the whole thing, correctly this time? And, if you've done that, just replace the actual function with the test and be done with it.
- praptak 9y ago
  If a function takes a single 32 bit argument you can run it on every possible value.
  - detaro 9y ago
    … this reply does nothing to answer the parent's question. It's obvious that you can run it on all possible values, but that doesn't answer the question of the return values. (unless you only want to verify "doesn't crash" or something similarly basic)
    - praptak 9y ago
      I believe the question was edited after I wrote my answer. This, or I just misread it - unfortunately HN doesn't let us know. Anyway, I'm providing another answer to the question as it is now.
- 9y ago