3 ms·

> It is absolutely not easy to achieve the kind of coverage that sqlite has. It is though. Give me one project and I'll send you a test suite that "tests" 100%

> It is absolutely not easy to achieve the kind of coverage that sqlite has. It is though. Give me one project and I'll send you a test suite that "tests" 100% of executable lines. It won't have any actual assertions, but that's not what code coverage is about anyways, so hopefully you won't mind. Number of assertions per line of code is about as good metric for how well tested something is as code coverage, which means just about nil.

Someone 4y ago
> Give me one project and I'll send you a test suite that "tests" 100% of executable lines. I don’t think you can do that with every project. For example, a function f may do: if (p == null) { abort(); } If all callers also do that test, there’s no way to hit that abort() line in the program. Your test suite may not be able to hit it either. Maybe f is a private function, or that test occurs on one of its intermediate values, or f accidentally has two identical tests: if (p == null) { abort(“p is null”); } if (p == null) { // copy-paste bug abort(“q is null”); } if (r == null) { abort(“r is null”); } Even if it is possible to hit that line, finding a path through the code that passes in a null value can be extremely hard. As a simple contrived counterexample, try getting 100% coverage on a function that does if(secure_hash(foo) == 0) bar();
- phoe-krk 4y ago
  > Your test suite may not be able to hit it either. Minor nitpick, process terminations are unit-testable in general; see e.g. https://github.com/google/googletest/blob/main/docs/advanced.md#death-tests https://github.com/google/googletest/blob/main/docs/advanced...
- garaetjjte 4y ago
  They don't count assertions in testing coverage: https://www.sqlite.org/testing.html#coverage_testing_of_defensive_code https://www.sqlite.org/testing.html#coverage_testing_of_defe...
- jonhohle 4y ago
  What is fair in testing and what is being tested? If I can stub `secure_hash` and `bar`, that’s a pretty easy test to write. Assuming the symbols are visible at test time, stubbing those is relatively straight forward in C, Java, Bash, and Ruby, JavaScript, etc. Even in the above examples, a linter should point out that the second `p == null` is inaccessible. I’m very pro test coverage, but static analysis shouldn’t be thrown out just because unit tests are available.
adrianN 4y ago
How much do you charge? Because adding assertions to tests that give MC/DC coverage is a lot cheaper than writing them. I know a couple of big companies who might be interested.
chasil 4y ago
Compliance with the DO-178B standard was not easy for the SQLite maintainers. It required several years to achieve. By adhering to the DO-178B standard, the testing is confirmed extensive and satisfies requirements for use in avionics. "Rockwell Collins... introduced me to this concept of DO-178B. It’s a quality standard for safety-critical aviation products, and unlike so many quality standards, this one’s actually readable... one of the key things that they push is, they want 100% MCDC test coverage. "Your tests have to cause each branch operation in the resulting binary code to be taken and to fall through at least once. "At the machine code level... actually, MCDC’s a little stricter than that. Let’s not dwell on the details, but I had this idea, I’m going to write tests to bring SQLite up to the quality of 100% MCDC, and that took a year of 60 hour weeks. That was hard, hard work. I was putting in 12 hour days every single day. I was just getting so tired of this because with this sort of thing, it’s the old joke of, you get 95% of the functionality with the first 95% of your budget, and the last 5% on the second 95% of your budget. It’s kind of the same thing. It’s pretty easy to get up to 90 or 95% test coverage. Getting that last 5% is really, really hard and it took about a year for me to get there, but once we got to that point, we stopped getting bug reports from Android." https://corecursive.com/066-sqlite-with-richard-hipp/ https://corecursive.com/066-sqlite-with-richard-hipp/ Indeed, SQLite is used for flight systems on the Airbus A350. No other major database vendor is certified to do so. https://www.sqlite.org/famous.html https://www.sqlite.org/famous.html
deleted 4y ago
[deleted]

Someone 4y ago

> Give me one project and I'll send you a test suite that "tests" 100% of executable lines. I don’t think you can do that with every project. For example, a function f may do: if (p == null) { abort(); } If all callers also do that test, there’s no way to hit that abort() line in the program. Your test suite may not be able to hit it either. Maybe f is a private function, or that test occurs on one of its intermediate values, or f accidentally has two identical tests: if (p == null) { abort(“p is null”); } if (p == null) { // copy-paste bug abort(“q is null”); } if (r == null) { abort(“r is null”); } Even if it is possible to hit that line, finding a path through the code that passes in a null value can be extremely hard. As a simple contrived counterexample, try getting 100% coverage on a function that does if(secure_hash(foo) == 0) bar();

phoe-krk 4y ago
> Your test suite may not be able to hit it either. Minor nitpick, process terminations are unit-testable in general; see e.g. https://github.com/google/googletest/blob/main/docs/advanced.md#death-tests https://github.com/google/googletest/blob/main/docs/advanced...
garaetjjte 4y ago
They don't count assertions in testing coverage: https://www.sqlite.org/testing.html#coverage_testing_of_defensive_code https://www.sqlite.org/testing.html#coverage_testing_of_defe...
jonhohle 4y ago
What is fair in testing and what is being tested? If I can stub `secure_hash` and `bar`, that’s a pretty easy test to write. Assuming the symbols are visible at test time, stubbing those is relatively straight forward in C, Java, Bash, and Ruby, JavaScript, etc. Even in the above examples, a linter should point out that the second `p == null` is inaccessible. I’m very pro test coverage, but static analysis shouldn’t be thrown out just because unit tests are available.

adrianN 4y ago

How much do you charge? Because adding assertions to tests that give MC/DC coverage is a lot cheaper than writing them. I know a couple of big companies who might be interested.

chasil 4y ago

Compliance with the DO-178B standard was not easy for the SQLite maintainers. It required several years to achieve. By adhering to the DO-178B standard, the testing is confirmed extensive and satisfies requirements for use in avionics. "Rockwell Collins... introduced me to this concept of DO-178B. It’s a quality standard for safety-critical aviation products, and unlike so many quality standards, this one’s actually readable... one of the key things that they push is, they want 100% MCDC test coverage. "Your tests have to cause each branch operation in the resulting binary code to be taken and to fall through at least once. "At the machine code level... actually, MCDC’s a little stricter than that. Let’s not dwell on the details, but I had this idea, I’m going to write tests to bring SQLite up to the quality of 100% MCDC, and that took a year of 60 hour weeks. That was hard, hard work. I was putting in 12 hour days every single day. I was just getting so tired of this because with this sort of thing, it’s the old joke of, you get 95% of the functionality with the first 95% of your budget, and the last 5% on the second 95% of your budget. It’s kind of the same thing. It’s pretty easy to get up to 90 or 95% test coverage. Getting that last 5% is really, really hard and it took about a year for me to get there, but once we got to that point, we stopped getting bug reports from Android." https://corecursive.com/066-sqlite-with-richard-hipp/ https://corecursive.com/066-sqlite-with-richard-hipp/ Indeed, SQLite is used for flight systems on the Airbus A350. No other major database vendor is certified to do so. https://www.sqlite.org/famous.html https://www.sqlite.org/famous.html

deleted 4y ago

[deleted]