5 ms·

>Unit testing. Problem is that if the error is uniformly distributed then each unit test only has a 1%-ish chance of finding the bug. We need to test a lot more

>Unit testing. Problem is that if the error is uniformly distributed then each unit test only has a 1%-ish chance of finding the bug. We need to test a lot more inputs. Not if you're writing your unit tests correctly. Thought should go into what you're testing. It might be true that you're going to find an error in 1% of cases, but you shouldn't be shooting random shit into your test. Or in other words you should "constrain" your "random test". I find it difficult to think of a situation where you actually don't know any helpful mathematical properties of your code - since you've obviosuly written the code with an intention to perform some function. There may be good reason for doing this stupid rotations and powers etc. But at the end of the day there was some non-code motviation for writing this code and that is what you should be thinking about when writing your unit tests. What the cross-branch testing gives you is an assurance that your code is as broken as when you started. Now, think about what is happening here when you test across branches, you have your golden reference model (which might or might not be right) and you have some python packages that you hope are doing a better job than you at constrained random testing. In fact, their odds of hitting that 1% are no better than your unit tests. I actually think to some extent testing against your previous implementation has value above the unit test - because my experience is that the first person to write the code probably thoroughly manually tested it and didn't convert all those tests into unit tests, and therefore your confidence in the original implementation is higher than revisions, but the lesson to learn from that is to make sure when you first implement something you push as much of your verification into unit tests.

amw-zero 5y ago
> Not if you're writing your unit tests correctly. Thought should go into what you're testing. I hear this all the time. “If you just write your tests correctly, you will not have any bugs.” A statement like this can only be made with a huge misunderstanding of computation and programming. Try telling all the formal verification researchers out there that the perfect set of unit tests can be determined for any program at all. It is not possible. That has been the holy grail of the verification community for 50 years at this point. See this paper from 1975, Towards a Theory of Test Data Selection: https://www.researchgate.net/profile/John-Goodenough/publication/220071437_Toward_a_Theory_of_Test_Data_Selection/links/545794080cf26d5090ab498c/Toward-a-Theory-of-Test-Data-Selection.pdf?origin=publication_detail https://www.researchgate.net/profile/John-Goodenough/publica.... In 1988, the Category Partition method of test case generation was introduced in this paper: https://www.cc.gatech.edu/~harrold/6340/cs6340_fall2009/Readings/ostrandCategoryPartition88.pdf https://www.cc.gatech.edu/~harrold/6340/cs6340_fall2009/Read.... In it, a much bigger sampling of the parameter input space is selected from for test cases, but it can result in thousands of test cases to write. This method has at least one industrial case study where a team applied it on financial software for Freddie Mac with 0 reported defects after release: https://cs.gmu.edu/~offutt/rsrch/papers/calcengine.pdf https://cs.gmu.edu/~offutt/rsrch/papers/calcengine.pdf. Really, the mother of all testing techniques is the DO-178C certification: https://en.m.wikipedia.org/wiki/DO-178C https://en.m.wikipedia.org/wiki/DO-178C. This is what companies use when developing avionics software. The highest level of verification requires 100% modified condition / decision coverage: https://en.m.wikipedia.org/wiki/Modified_condition/decision_coverage https://en.m.wikipedia.org/wiki/Modified_condition/decision_.... Given that most people don’t know what MC/DC or the category partition method are, I doubt that there are many actually effective test suites out there. Which makes the frequency of these statements pretty confusing.
- thraxil 5y ago
  SQLite has a DO-178B "inspired" test suite and apparently has achieved 100% MC/DC [1]. Unfortunately, that test suite isn't public. I'd love to see more non-trivial examples like that made public. 1: https://news.ycombinator.com/item?id=18039992 https://news.ycombinator.com/item?id=18039992
  - amw-zero 5y ago
    From everything I’ve heard about SQLite testing, it is absolutely an exception to the ‘normal’ state! I hadn’t heard that they took direct inspiration from DO-178B, so thanks for sharing that. That’s a great data point :)
- Traster 5y ago
  You're not really addressing what I've said. Sure, formal verification and ensuring code coverage is a higher standard of verification than unit testing. But that doesn't mean your unit testing provides literally the same probability of finding the bug as completely random test inputs. If you're writing decent tests your chance of finding that bug isn't 1%, it's way higher. In fact I think your entire appraoch to this is wrong. >Given that most people don’t know what MC/DC or the category partition method are, I doubt that there are many actually effective test suites out there. Given that most people aren't writing software to go into safety critical applications, it doesn't make sense for them to be designing their test suites to that standard. What I'm saying is that this method of testing boils down to hoping that the master is correct (which it probably isn't) and then trusting someone else's tool to provide coverage for you. This is worse than writing unit tests because the tool you've handed responsibility to for coverage doesn't know anything about your device under test and therefore can't direct it's tests to maximize the likelihood of finding errors. At the very least writing a simple model to test against would be more effective because you'd expose bugs in the detail-orientated implementation.

amw-zero 5y ago

> Not if you're writing your unit tests correctly. Thought should go into what you're testing. I hear this all the time. “If you just write your tests correctly, you will not have any bugs.” A statement like this can only be made with a huge misunderstanding of computation and programming. Try telling all the formal verification researchers out there that the perfect set of unit tests can be determined for any program at all. It is not possible. That has been the holy grail of the verification community for 50 years at this point. See this paper from 1975, Towards a Theory of Test Data Selection: https://www.researchgate.net/profile/John-Goodenough/publication/220071437_Toward_a_Theory_of_Test_Data_Selection/links/545794080cf26d5090ab498c/Toward-a-Theory-of-Test-Data-Selection.pdf?origin=publication_detail https://www.researchgate.net/profile/John-Goodenough/publica.... In 1988, the Category Partition method of test case generation was introduced in this paper: https://www.cc.gatech.edu/~harrold/6340/cs6340_fall2009/Readings/ostrandCategoryPartition88.pdf https://www.cc.gatech.edu/~harrold/6340/cs6340_fall2009/Read.... In it, a much bigger sampling of the parameter input space is selected from for test cases, but it can result in thousands of test cases to write. This method has at least one industrial case study where a team applied it on financial software for Freddie Mac with 0 reported defects after release: https://cs.gmu.edu/~offutt/rsrch/papers/calcengine.pdf https://cs.gmu.edu/~offutt/rsrch/papers/calcengine.pdf. Really, the mother of all testing techniques is the DO-178C certification: https://en.m.wikipedia.org/wiki/DO-178C https://en.m.wikipedia.org/wiki/DO-178C. This is what companies use when developing avionics software. The highest level of verification requires 100% modified condition / decision coverage: https://en.m.wikipedia.org/wiki/Modified_condition/decision_coverage https://en.m.wikipedia.org/wiki/Modified_condition/decision_.... Given that most people don’t know what MC/DC or the category partition method are, I doubt that there are many actually effective test suites out there. Which makes the frequency of these statements pretty confusing.

thraxil 5y ago
SQLite has a DO-178B "inspired" test suite and apparently has achieved 100% MC/DC [1]. Unfortunately, that test suite isn't public. I'd love to see more non-trivial examples like that made public. 1: https://news.ycombinator.com/item?id=18039992 https://news.ycombinator.com/item?id=18039992
- amw-zero 5y ago
  From everything I’ve heard about SQLite testing, it is absolutely an exception to the ‘normal’ state! I hadn’t heard that they took direct inspiration from DO-178B, so thanks for sharing that. That’s a great data point :)
Traster 5y ago
You're not really addressing what I've said. Sure, formal verification and ensuring code coverage is a higher standard of verification than unit testing. But that doesn't mean your unit testing provides literally the same probability of finding the bug as completely random test inputs. If you're writing decent tests your chance of finding that bug isn't 1%, it's way higher. In fact I think your entire appraoch to this is wrong. >Given that most people don’t know what MC/DC or the category partition method are, I doubt that there are many actually effective test suites out there. Given that most people aren't writing software to go into safety critical applications, it doesn't make sense for them to be designing their test suites to that standard. What I'm saying is that this method of testing boils down to hoping that the master is correct (which it probably isn't) and then trusting someone else's tool to provide coverage for you. This is worse than writing unit tests because the tool you've handed responsibility to for coverage doesn't know anything about your device under test and therefore can't direct it's tests to maximize the likelihood of finding errors. At the very least writing a simple model to test against would be more effective because you'd expose bugs in the detail-orientated implementation.