3 ms·

> nearly 6 times as many lines of code SQLite has 711 times more lines of tests as it does of code. Does that make it a worse software product?

arghwhat 8y ago
No, but if it has more lines of code (not tests) than a competing product, then it is more complex, and has a higher probability of defects. In security, defects have serious consequences, so it is in best interest of everyone to have the lowest possible complexity, much more so than for other types of software. The only "stricter" category would be software related to operation of equipment whose failure could cause direct physical harm.
nickpsecurity 8y ago
Based on empirical observation, the number of bugs goes up with lines of code on average. There's also some point where something is too big for the human mind to understand in its entirety, either developer or reviewers. To say something is secure, you have to understand everything it will and won't do. Those analysis grow exponentially with size due to input/output ranges and combinations of paths (i.e. combinatorial explosion). So, smaller is better. Ideally, small like Wireguard so the most thorough analysis possible. Lets look at your counter example. It's smaller than most databases as I advise. Following the other rule, it was untrustworthy by default with developers adding piles of tests to uncover bugs, increase confidence, and make changes with less breakage.. Nonetheless, CompSci papers I read on static analysis and fuzzing often test SQLite finding bugs. Even such a well-tested application still had plenty of bugs over time due to its complexity. Most of which might be just intrinsic to the kind of features they're developing. Back to VPN's, you want to know it will maintain security policy (esp correct info flows) under all inputs in all states, normal and failure. And if no progress can be made, you want it to fail-safe. So, making it six times larger without a need to is definitely worse than not making it six times larger. The larger product will, based on empirical data like SQLite, have more bugs with more code injections or data leaks following from that. I advise minimal, careful, rigorously-evaluated implementation of a formally-verified protocol. Wireguard is closest to that right now.

arghwhat 8y ago

No, but if it has more lines of code (not tests) than a competing product, then it is more complex, and has a higher probability of defects. In security, defects have serious consequences, so it is in best interest of everyone to have the lowest possible complexity, much more so than for other types of software. The only "stricter" category would be software related to operation of equipment whose failure could cause direct physical harm.

nickpsecurity 8y ago

Based on empirical observation, the number of bugs goes up with lines of code on average. There's also some point where something is too big for the human mind to understand in its entirety, either developer or reviewers. To say something is secure, you have to understand everything it will and won't do. Those analysis grow exponentially with size due to input/output ranges and combinations of paths (i.e. combinatorial explosion). So, smaller is better. Ideally, small like Wireguard so the most thorough analysis possible. Lets look at your counter example. It's smaller than most databases as I advise. Following the other rule, it was untrustworthy by default with developers adding piles of tests to uncover bugs, increase confidence, and make changes with less breakage.. Nonetheless, CompSci papers I read on static analysis and fuzzing often test SQLite finding bugs. Even such a well-tested application still had plenty of bugs over time due to its complexity. Most of which might be just intrinsic to the kind of features they're developing. Back to VPN's, you want to know it will maintain security policy (esp correct info flows) under all inputs in all states, normal and failure. And if no progress can be made, you want it to fail-safe. So, making it six times larger without a need to is definitely worse than not making it six times larger. The larger product will, based on empirical data like SQLite, have more bugs with more code injections or data leaks following from that. I advise minimal, careful, rigorously-evaluated implementation of a formally-verified protocol. Wireguard is closest to that right now.