6 ms·

I'm a little uneasy about the conclusions being drawn here as the obvious counterpoint isn't being raised - what if older code isn't being looked at as hard and

I'm a little uneasy about the conclusions being drawn here as the obvious counterpoint isn't being raised - what if older code isn't being looked at as hard and therefore vulnerabilities aren't being discovered? It's far more common to look at recent commit logs than it is to look at some library that hasn't changed for 20 years.

SoylentOrange 2y ago
I don’t understand this point. The project under scrutiny is Android and people are detecting vulnerabilities both manually and automatically based on source code/binary, not over commit logs. Why would the commit logs be relevant at all to finding bugs? The commits are just used for attribution. If there was some old lib that hasn’t been changed in 20 years that’s passed fuzzing and manual code inspection for 20 years without updates, chances are it’s solid.
- saagarjha 2y ago
  Exploit authors look at commit logs because new features have bugs in them, and it's easier to follow that to find vulnerabilities than dive into the codebase to find what's already there.
e28eta 2y ago
I wasn’t entirely satisfied with the assertion that older code has fewer vulnerabilities either. It feels like there could be explanations other than age for the discrepancy. For example: maybe the engineers over the last several years have focused on rewriting the riskiest parts in a MSL, and were less likely to change the lower risk old code. Or… maybe there was a process or personnel change that led to more defects. With that said, it does seem plausible to me that any given bug has a probability of detection per unit of time, and as time passes fewer defects remain to be found. And as long as your maintainers fix more vulnerabilities than they introduce, sure, older code will have fewer and the ones that remain are probably hard to find.
MBCook 2y ago
> what if older code isn't being looked at as hard and therefore vulnerabilities aren't being discovered? It wasn’t being look at as hard before either. I don’t think that’s changed. They don’t give a theory for why older code has fewer bugs, but I’ve got one: they’ve been found. If we assumed that any piece of code has a fixed amount of unknown bugs per 1000 lines, it stands to reason that overtime the sheer number of times the code is run with different inputs in prod makes it more and more likely they will be discovered. Between fixing them and the code reviews while fixing them the hope would be that on average things are being made better. So overtime, there are fewer bugs per thousand lines in existing code. It’s been battle tested. As the post says, if you continue introducing new bugs at the same rate you’re not going to make progress. But if using a memory safe language means you’re introducing fewer bugs in new features then overtime the total number of bugs should be going down.
- pacaro 2y ago
  I've always thought of this as being equivalent to "work hardening" My concern with it is more about legitimately old code (android is 20ish years old, so reasonably falls into this category) which was written using standards and tools of the time (necessarily) It requires a constant engineering effort to keep such code up to date. And the older code is, typically, less well understood. In addition older code (particularly in systems programming) is often associated with older requirements, some of which may have become niche over time. That long tail of old, less frequently exercised, code feels like it may well have a sting in its tail. The halflife/work-hardening model depends on the code being stressed to find bugs
  - kernal 2y ago
    Android was released on September 23, 2008, so it just had its sweet 16.
    - pacaro 2y ago
      I believe that they started writing it in 2003. It's hard to precisely age code unless you cut it down and count the number of rings
mccr8 2y ago
Their concern is not with theoretical vulnerabilities, but actual ones that are being exploited. If an attacker never tries to find a vulnerability in some code, then it might as well not have it.

SoylentOrange 2y ago

I don’t understand this point. The project under scrutiny is Android and people are detecting vulnerabilities both manually and automatically based on source code/binary, not over commit logs. Why would the commit logs be relevant at all to finding bugs? The commits are just used for attribution. If there was some old lib that hasn’t been changed in 20 years that’s passed fuzzing and manual code inspection for 20 years without updates, chances are it’s solid.

saagarjha 2y ago
Exploit authors look at commit logs because new features have bugs in them, and it's easier to follow that to find vulnerabilities than dive into the codebase to find what's already there.

e28eta 2y ago

I wasn’t entirely satisfied with the assertion that older code has fewer vulnerabilities either. It feels like there could be explanations other than age for the discrepancy. For example: maybe the engineers over the last several years have focused on rewriting the riskiest parts in a MSL, and were less likely to change the lower risk old code. Or… maybe there was a process or personnel change that led to more defects. With that said, it does seem plausible to me that any given bug has a probability of detection per unit of time, and as time passes fewer defects remain to be found. And as long as your maintainers fix more vulnerabilities than they introduce, sure, older code will have fewer and the ones that remain are probably hard to find.

MBCook 2y ago

> what if older code isn't being looked at as hard and therefore vulnerabilities aren't being discovered? It wasn’t being look at as hard before either. I don’t think that’s changed. They don’t give a theory for why older code has fewer bugs, but I’ve got one: they’ve been found. If we assumed that any piece of code has a fixed amount of unknown bugs per 1000 lines, it stands to reason that overtime the sheer number of times the code is run with different inputs in prod makes it more and more likely they will be discovered. Between fixing them and the code reviews while fixing them the hope would be that on average things are being made better. So overtime, there are fewer bugs per thousand lines in existing code. It’s been battle tested. As the post says, if you continue introducing new bugs at the same rate you’re not going to make progress. But if using a memory safe language means you’re introducing fewer bugs in new features then overtime the total number of bugs should be going down.

pacaro 2y ago
I've always thought of this as being equivalent to "work hardening" My concern with it is more about legitimately old code (android is 20ish years old, so reasonably falls into this category) which was written using standards and tools of the time (necessarily) It requires a constant engineering effort to keep such code up to date. And the older code is, typically, less well understood. In addition older code (particularly in systems programming) is often associated with older requirements, some of which may have become niche over time. That long tail of old, less frequently exercised, code feels like it may well have a sting in its tail. The halflife/work-hardening model depends on the code being stressed to find bugs
- kernal 2y ago
  Android was released on September 23, 2008, so it just had its sweet 16.
  - pacaro 2y ago
    I believe that they started writing it in 2003. It's hard to precisely age code unless you cut it down and count the number of rings

mccr8 2y ago

Their concern is not with theoretical vulnerabilities, but actual ones that are being exploited. If an attacker never tries to find a vulnerability in some code, then it might as well not have it.