4 ms·

Indeed. Tests do not tell you code is correct. They only tell you a list of ways it is not incorrect.

by RHSeeger 4y ago

aa-jv 4y ago
.. and combined with coverage they also tell you what you are yet to test properly.
- fulafel 4y ago
  s/properly//
- kevincox 4y ago
  Actually: It tells you some things that are not tested properly. Just because something has coverage doesn't mean that it is tested properly as evidenced by this bug even though sqlite has 100% branch coverage.
  - Macha 4y ago
    Yeah, coverage (or rather 100% - coverage) is basically a lower bound on what's not tested.
    - masklinn 4y ago
      Even "coverage" is not enough, what 100% coverage? Lines? Branches? Paths?
      - Macha 4y ago
        That's my point, coverage (whatever your chosen metric) can only prove the absence of tests, not their presence e.g. if your coverage is 80%, then you know (100-80)% = 20% is definitely not tested since it never even ran in your tests. It doesn't tell you much about the 80%
PathOfEclipse 4y ago
I think this thinking is short-sighted. Tests increase your confidence that code is correct, and they are much cheaper to do that formal verification, which itself still requires a leap of faith that there is no bug in the formal verification system, its implementation in the compiler toolchain, or the assumptions made about the underlying hardware. You can cast doubt on anything. Will you be alive 30 seconds from now? You don't know for sure! But you can be pretty confident. Similarly, testing can increase confidence in code correctness. SQLite has an excellent security history overall, and it's testing strategy is part of the reason for that success.
- pclmulqdq 4y ago
  Formal verification also becomes practically impossible at a certain level of complexity. SQLite is better tested than many silicon chips, so it's hard to fault their testing strategy. Perhaps a few static analyzers should be run on the codebase to find possible bugs like this.
  - hulitu 4y ago
    > Formal verification also becomes practically impossible at a certain level of complexity. Then reduce complexity.
    - nottorp 4y ago
      Well, int main (void) { return 1; } is most likely bug free. But it's not useful (except as /bin/true).
      - jwilk 4y ago
        That would be a very buggy /bin/true implementation.
        nottorp 4y ago
        Right, but a correct /bin/false :)

Indeed. Tests do not tell you code is correct. They only tell you a list of ways it is not incorrect.

aa-jv 4y ago

.. and combined with coverage they also tell you what you are yet to test properly.

fulafel 4y ago
s/properly//
kevincox 4y ago
Actually: It tells you some things that are not tested properly. Just because something has coverage doesn't mean that it is tested properly as evidenced by this bug even though sqlite has 100% branch coverage.
- Macha 4y ago
  Yeah, coverage (or rather 100% - coverage) is basically a lower bound on what's not tested.
  - masklinn 4y ago
    Even "coverage" is not enough, what 100% coverage? Lines? Branches? Paths?
    - Macha 4y ago
      That's my point, coverage (whatever your chosen metric) can only prove the absence of tests, not their presence e.g. if your coverage is 80%, then you know (100-80)% = 20% is definitely not tested since it never even ran in your tests. It doesn't tell you much about the 80%

PathOfEclipse 4y ago

I think this thinking is short-sighted. Tests increase your confidence that code is correct, and they are much cheaper to do that formal verification, which itself still requires a leap of faith that there is no bug in the formal verification system, its implementation in the compiler toolchain, or the assumptions made about the underlying hardware. You can cast doubt on anything. Will you be alive 30 seconds from now? You don't know for sure! But you can be pretty confident. Similarly, testing can increase confidence in code correctness. SQLite has an excellent security history overall, and it's testing strategy is part of the reason for that success.

pclmulqdq 4y ago
Formal verification also becomes practically impossible at a certain level of complexity. SQLite is better tested than many silicon chips, so it's hard to fault their testing strategy. Perhaps a few static analyzers should be run on the codebase to find possible bugs like this.
- hulitu 4y ago
  > Formal verification also becomes practically impossible at a certain level of complexity. Then reduce complexity.
  - nottorp 4y ago
    Well, int main (void) { return 1; } is most likely bug free. But it's not useful (except as /bin/true).
    - jwilk 4y ago
      That would be a very buggy /bin/true implementation.
      - nottorp 4y ago
        Right, but a correct /bin/false :)