4 ms·
No kidding. This has been known for a long time but it's good to see empirical evidence. The empirical evidence of 70's-90's showed the best ways to reduce defe
by nickpsecurity 11y ago
No kidding. This has been known for a long time but it's good to see empirical evidence. The empirical evidence of 70's-90's showed the best ways to reduce defects were the following:
1. Unambiguous specifications of what the system will do.
2. Implementation in safe languages or subsets with plenty of interface checks.
3. Human review of design, implementation, and configuration looking for errors. Best results were here.
4. Usage-based testing to eliminate most errors user will experience in practice.
5. Thorough functional and unit testing.
(Later on...)
6. Fuzz testing.
7. Static analysis.
Practices 1-4 were used in Cleanroom, some major commercial projects, and high assurance security work. Every time, they produced results in preventing serious defects from entering the system that often might have slipped by testing. So, those are where the majority of our verification work should go in. Supporting evidence is from those that do a subset of that from OpenBSD team to Microsoft's SDL along with substantial defect reduction that followed.
Note: 6 and 7 showed their value later on. No 7 should definitely be done where possible with 6 done to a degree. I haven't researched 6 enough to give advice on most efficient approach, though.
So, internal testing and coverage are the weakest forms of verification. They have many benefits for sure but people put way too much effort and expectations into them where it pays off more elsewhere. Do enough testing for its key benefits then just cut it off from there. And don't even do most of that until you've done code reviews, etc which pay off more.
- technion 11y agoI'm not going to dispute that #7 earns a place lower down the chain than the features you mentioned in practical terms of identifying bugs. That said, it's only overly feasible when run from early on in design. If a project currently produces clean output, and I make a commit that causes clang's analyser to complain, it's easy enough to fix, whether it's a "legitimate" error or not. I've tried throwing static analyzers at long established projects like OpenSSL and.. there's just no point. Noone is going to review a few thousand errors and make hundreds of commits that don't actually fix anything just to get perfectly clean static analysis output. It's more likely I would break something than help. It's really not that hard to implement a "no commit unless it passes static analysis" rule from the start, and even if the bugs removed are low, I usually argue they are non-zero.
- nickpsecurity 11y agoI did it in chronological order of discovery and significant application. I agree it should be used as early as possible. Plenty of benefit to that. Further, the Orange Book era systems discovered a heuristic that supports your theory about old codebases: things must be designed for such verification or security from Day 1. Usually impossible to retrofit such properties into an existing codebase whose methods were quite different.