4 ms·

What about logical errors? math.pow(2, 4) vs math.pow(4, 2)

by StreamBright 7y ago

ISL 7y ago
A test can test for those: "does the code give mathematically correct results?"
- StreamBright 7y ago
  I meant pow as an example to point out that there are logical errors in code that you cannot catch with unit tests.
  - weaksauce 7y ago
    those kinds of errors should be caught a little higher in the hierarchy... some kind of feature or integration test.
- Scarblac 7y ago
  Even the best tests only catch like 50% of the bugs though.
  - WrtCdEvrydy 7y ago
    That's not really correct. Properly input space partitioned tests have something like 90 to 95 accuracy if properly written. The issues always come from people not wanting to add sufficient tests.
    - cortesoft 7y ago
      Does that mean that 1 in 20 deploys breaks, then?
      - WrtCdEvrydy 7y ago
        1 in 20 deploys has a false negative (IE, something went out without working). Continous Integration is mostly about regression, so the new tests have less value than running old tests. We tend to fix issues in production (bugs) by mandating a unit test with the failing issue to be written as part of the fix.
  - aasasd 7y ago
    Bahahaha, your tests really aren't ‘best’. Properly, if your code has an ‘if’, you need two tests, for the two branches. Same with every place the outcome may diverge. With this approach, it's basically impossible to botch the code unless something slips your mind while writing both the code and the tests. Otherwise, it's pretty much ‘deploy and go home.’
    - not_kurt_godel 7y ago
      This is good in theory, and often in practice, but not always. If this practice is followed blindly, unit tests can become too tightly bound to the code to the point where every code change is assumed to require corresponding 1:1 unit test changes and it becomes difficult to distinguish between a broken test that signals broken functionality versus one that simply hasn't been 'synced' yet.
      - deleted 7y ago
        [deleted]
    - MauranKilom 7y ago
      > Properly, if your code has an ‘if’, you need two tests, for the two branches. Same with every place the outcome may diverge. With this approach, it's basically impossible to botch the code if (x % 3 == 0) println "Fizz" if (x % 5 == 0) println "Buzz" There, solved it! And it works fine in all cases you told me to test (e.g. 3, 4, and 5), which means it's impossible I botched anything! Surely I nailed this interview? Seriously though, the criterion you mentioned is only one of many of increasing strictness (see e.g. https://en.wikipedia.org/wiki/Code_coverage#Basic_coverage_criteria https://en.wikipedia.org/wiki/Code_coverage#Basic_coverage_c...), namely branch coverage. Having branch coverage still says very little - the interaction between different branches can be trivially wrong. And desiring full path coverage immediately leads to combinatorial explosion (and the halting problem, once loops are involved). > unless something slips your mind while writing both the code and the tests. That is true for any choice of coverage metric and target.
      - 7y ago
ouaiueoaieaeoiu 7y ago
Semantic errors? I don't think a unit test will be able to distinguish 16 from 16.
- ricardobeat 7y ago
  Imagine this: the spec is Math.pow(x,y). It was implemented as Math.pow(y, x). The test was written with x=2, y=4. It is accidentally correct. This is why any unit tests dealing with absolute or enum values needs to run multiple iterations.

What about logical errors? math.pow(2, 4) vs math.pow(4, 2)

ISL 7y ago

A test can test for those: "does the code give mathematically correct results?"

StreamBright 7y ago
I meant pow as an example to point out that there are logical errors in code that you cannot catch with unit tests.
- weaksauce 7y ago
  those kinds of errors should be caught a little higher in the hierarchy... some kind of feature or integration test.
Scarblac 7y ago
Even the best tests only catch like 50% of the bugs though.
- WrtCdEvrydy 7y ago
  That's not really correct. Properly input space partitioned tests have something like 90 to 95 accuracy if properly written. The issues always come from people not wanting to add sufficient tests.
  - cortesoft 7y ago
    Does that mean that 1 in 20 deploys breaks, then?
    - WrtCdEvrydy 7y ago
      1 in 20 deploys has a false negative (IE, something went out without working). Continous Integration is mostly about regression, so the new tests have less value than running old tests. We tend to fix issues in production (bugs) by mandating a unit test with the failing issue to be written as part of the fix.
- aasasd 7y ago
  Bahahaha, your tests really aren't ‘best’. Properly, if your code has an ‘if’, you need two tests, for the two branches. Same with every place the outcome may diverge. With this approach, it's basically impossible to botch the code unless something slips your mind while writing both the code and the tests. Otherwise, it's pretty much ‘deploy and go home.’
  - not_kurt_godel 7y ago
    This is good in theory, and often in practice, but not always. If this practice is followed blindly, unit tests can become too tightly bound to the code to the point where every code change is assumed to require corresponding 1:1 unit test changes and it becomes difficult to distinguish between a broken test that signals broken functionality versus one that simply hasn't been 'synced' yet.
    - deleted 7y ago
      [deleted]
  - MauranKilom 7y ago
    > Properly, if your code has an ‘if’, you need two tests, for the two branches. Same with every place the outcome may diverge. With this approach, it's basically impossible to botch the code if (x % 3 == 0) println "Fizz" if (x % 5 == 0) println "Buzz" There, solved it! And it works fine in all cases you told me to test (e.g. 3, 4, and 5), which means it's impossible I botched anything! Surely I nailed this interview? Seriously though, the criterion you mentioned is only one of many of increasing strictness (see e.g. https://en.wikipedia.org/wiki/Code_coverage#Basic_coverage_criteria https://en.wikipedia.org/wiki/Code_coverage#Basic_coverage_c...), namely branch coverage. Having branch coverage still says very little - the interaction between different branches can be trivially wrong. And desiring full path coverage immediately leads to combinatorial explosion (and the halting problem, once loops are involved). > unless something slips your mind while writing both the code and the tests. That is true for any choice of coverage metric and target.
    - 7y ago

ouaiueoaieaeoiu 7y ago

Semantic errors? I don't think a unit test will be able to distinguish 16 from 16.

ricardobeat 7y ago
Imagine this: the spec is Math.pow(x,y). It was implemented as Math.pow(y, x). The test was written with x=2, y=4. It is accidentally correct. This is why any unit tests dealing with absolute or enum values needs to run multiple iterations.