3 ms·

> Not if you're writing your unit tests correctly. Thought should go into what you're testing. I hear this all the time. “If you just write your tests correctl

> Not if you're writing your unit tests correctly. Thought should go into what you're testing. I hear this all the time. “If you just write your tests correctly, you will not have any bugs.” A statement like this can only be made with a huge misunderstanding of computation and programming. Try telling all the formal verification researchers out there that the perfect set of unit tests can be determined for any program at all. It is not possible. That has been the holy grail of the verification community for 50 years at this point. See this paper from 1975, Towards a Theory of Test Data Selection: https://www.researchgate.net/profile/John-Goodenough/publication/220071437_Toward_a_Theory_of_Test_Data_Selection/links/545794080cf26d5090ab498c/Toward-a-Theory-of-Test-Data-Selection.pdf?origin=publication_detail https://www.researchgate.net/profile/John-Goodenough/publica.... In 1988, the Category Partition method of test case generation was introduced in this paper: https://www.cc.gatech.edu/~harrold/6340/cs6340_fall2009/Readings/ostrandCategoryPartition88.pdf https://www.cc.gatech.edu/~harrold/6340/cs6340_fall2009/Read.... In it, a much bigger sampling of the parameter input space is selected from for test cases, but it can result in thousands of test cases to write. This method has at least one industrial case study where a team applied it on financial software for Freddie Mac with 0 reported defects after release: https://cs.gmu.edu/~offutt/rsrch/papers/calcengine.pdf https://cs.gmu.edu/~offutt/rsrch/papers/calcengine.pdf. Really, the mother of all testing techniques is the DO-178C certification: https://en.m.wikipedia.org/wiki/DO-178C https://en.m.wikipedia.org/wiki/DO-178C. This is what companies use when developing avionics software. The highest level of verification requires 100% modified condition / decision coverage: https://en.m.wikipedia.org/wiki/Modified_condition/decision_coverage https://en.m.wikipedia.org/wiki/Modified_condition/decision_.... Given that most people don’t know what MC/DC or the category partition method are, I doubt that there are many actually effective test suites out there. Which makes the frequency of these statements pretty confusing.

thraxil 5y ago
SQLite has a DO-178B "inspired" test suite and apparently has achieved 100% MC/DC [1]. Unfortunately, that test suite isn't public. I'd love to see more non-trivial examples like that made public. 1: https://news.ycombinator.com/item?id=18039992 https://news.ycombinator.com/item?id=18039992
- amw-zero 5y ago
  From everything I’ve heard about SQLite testing, it is absolutely an exception to the ‘normal’ state! I hadn’t heard that they took direct inspiration from DO-178B, so thanks for sharing that. That’s a great data point :)
Traster 5y ago
You're not really addressing what I've said. Sure, formal verification and ensuring code coverage is a higher standard of verification than unit testing. But that doesn't mean your unit testing provides literally the same probability of finding the bug as completely random test inputs. If you're writing decent tests your chance of finding that bug isn't 1%, it's way higher. In fact I think your entire appraoch to this is wrong. >Given that most people don’t know what MC/DC or the category partition method are, I doubt that there are many actually effective test suites out there. Given that most people aren't writing software to go into safety critical applications, it doesn't make sense for them to be designing their test suites to that standard. What I'm saying is that this method of testing boils down to hoping that the master is correct (which it probably isn't) and then trusting someone else's tool to provide coverage for you. This is worse than writing unit tests because the tool you've handed responsibility to for coverage doesn't know anything about your device under test and therefore can't direct it's tests to maximize the likelihood of finding errors. At the very least writing a simple model to test against would be more effective because you'd expose bugs in the detail-orientated implementation.

thraxil 5y ago

SQLite has a DO-178B "inspired" test suite and apparently has achieved 100% MC/DC [1]. Unfortunately, that test suite isn't public. I'd love to see more non-trivial examples like that made public. 1: https://news.ycombinator.com/item?id=18039992 https://news.ycombinator.com/item?id=18039992

amw-zero 5y ago
From everything I’ve heard about SQLite testing, it is absolutely an exception to the ‘normal’ state! I hadn’t heard that they took direct inspiration from DO-178B, so thanks for sharing that. That’s a great data point :)

Traster 5y ago

You're not really addressing what I've said. Sure, formal verification and ensuring code coverage is a higher standard of verification than unit testing. But that doesn't mean your unit testing provides literally the same probability of finding the bug as completely random test inputs. If you're writing decent tests your chance of finding that bug isn't 1%, it's way higher. In fact I think your entire appraoch to this is wrong. >Given that most people don’t know what MC/DC or the category partition method are, I doubt that there are many actually effective test suites out there. Given that most people aren't writing software to go into safety critical applications, it doesn't make sense for them to be designing their test suites to that standard. What I'm saying is that this method of testing boils down to hoping that the master is correct (which it probably isn't) and then trusting someone else's tool to provide coverage for you. This is worse than writing unit tests because the tool you've handed responsibility to for coverage doesn't know anything about your device under test and therefore can't direct it's tests to maximize the likelihood of finding errors. At the very least writing a simple model to test against would be more effective because you'd expose bugs in the detail-orientated implementation.