5 ms·
Counting the number of CVEs to measure the security of a piece of software makes as much sense as counting lines of code to measure a developer’s performance.
by laumars 5y ago
Counting the number of CVEs to measure the security of a piece of software makes as much sense as counting lines of code to measure a developer’s performance.
- okwubodu 5y agoSomeone breaking into my house 10 times by jiggling the same doorknob a contractor told me they “fixed” would be very concerning.
- CharlesW 5y agoA door would need thousands of doorknobs for that analogy to make sense.
- okwubodu 5y agoNo, you would need to admit your current approach is ineffective and replace the door and/or the repairman.
- laumars 5y agoThe GPs point is that writing software is a thousand times more complicated than installing a door knob. And frankly I think they’re being generous to you; writing a frame buffer is even more complicated than their estimation.
- hk__2 5y ago> Someone breaking into my house 10 times by jiggling the same doorknob a contractor told me they “fixed” would be very concerning. The complexity is not the same. My grandfather is not questionning my ability to fix his computer that “breaks” once in a while just because I already fixed it the last time.
- malwarebytess 5y agoI don't agree with this analogy. It may very well point to a fundamental problem with the design of the software, or in methodologies employed in its development. In this case a reasonable inference could be that Apple's memory safety design principles need some re-examination. Hard to tell without being on the inside.
- laumars 5y agohttps://news.ycombinator.com/item?id=27969587 https://news.ycombinator.com/item?id=27969587
- gorgoiler 5y agoMaybe. Quantifying your counter argument would help. If one developer writes 90% of the code there is signal in the statistics.
- laumars 5y agoCounting CVEs is a trivially easy argument to debunk. It’s a topic well written about by many open source contributors and security researchers. It’s a topic discussed on here frequently too. And it’s a topic that common sense alone should be able to debunk, should one spend a few minutes thinking about it rationally. However I have written a more in-depth reply about why counting CVEs is meaningless here: https://news.ycombinator.com/item?id=27969587 https://news.ycombinator.com/item?id=27969587
- joebob42 5y agoI think the "counterpoint" here is that that makes an amount of sense >0. While it's far from a complete story, and there are plenty of edge cases where it can be misleading, it's also a metric that will tend to bias in the right direction and has a decent relation to what we are trying to measure.
- laumars 5y agoSome researchers get paid per CVE and thus are encouraged to release hundreds of frivolous reports. Some companies publish frivolous ones in order to raise their own profile. Also the software in question is a complicated subsystem to write on a platform that’s guaranteed to get many eyeballs on. So the fact that those CVEs span roughly once per year might also be a demonstration that there are relatively few bugs (if we knew the number of people and time spent researching vs CVEs published then maybe we’d have a more meaningful statistic). CVEs do also demonstrate that active research is happening. There are plenty of common libraries out there that never get audited. Does fewer CVEs mean they’re more secure? Or does it just mean that nobody has checked? Thus on its own, that data is pretty meaningless in terms of deriving a trend. The reason the LoC comparison was made is because measuring lines of code doesn’t tell you how long a developer has spent debugging, reading documentation, or doing other research required. It doesn’t tell you how secure, performant, or even buggy the code is. All it tells you is the number of lines written and literally nothing more can be derived from that figure. Likewise with CVEs submitted.