4 ms·

There is a correlation between new code and memory vulnerabilities (a possible explanation is given in the blog post, that vulnerabilities have a half-life that

by gortok 2y ago

There is a correlation between new code and memory vulnerabilities (a possible explanation is given in the blog post, that vulnerabilities have a half-life that decays rapidly), but why does the blog post indicate causation between the two factors? There is more than one possible and reasonable explanation for this correlation: 1. New code often relates to new features, and folks focus on new features for vulnerabilities. 2. Older code has been through more real life usage, which can exercise those edge cases where memory vulnerabilities reside. I’m just not comfortable saying new code causes memory vulnerabilities and that vulnerabilities have a half-life that decays rapidly. That may —- may be true in sheer number count, but doesn’t seem to be true in impact, thinking back to the high-impact vulnerabilities in OSS like the heartbleed bug, and the cache-invalidation bugs for CPUs.

anymouse123456 2y ago
I'm here with you for the downvotes. This essay takes some interesting data from very specific (and unusual) projects and languages from a very specific (and unusual) culture and stridently extrapolates hard numeric values to all code without qualification. > For example, based on the average vulnerability lifetimes, 5-year-old code has a 3.4x (using lifetimes from the study) to 7.4x (using lifetimes observed in Android and Chromium) lower vulnerability density than new code. Given this conclusion, I can write a defect-filled chunk of C code and just let it marinate for 5 years offline in order for it become safe? I'm pretty sure there are important data in this research and there is truth underneath what is being shared, but the unsupported confidence and overreach of the writing is too distracting for me.
- throwaway2037 2y ago
  Why is this downvoted? It raises some interesting and important issues. I saw a big bank's back office settlement system once. 30+ years old with almost no unit tests. It changes very little now and is virtually bug free because people have been fixing bugs in it for 30+ years! When they need to make changes these days, they first write unit tests for existing behaviour, then fix the bug. It is an example of how code can mature with a very low defect rate with limited unit tests.
  - steveklabnik 2y ago
    I didn't downvote it, but, while I agree that there's reason to be skeptical that this research generalizes, the framing is aggressive. > stridently extrapolates hard numeric values to all code without qualification. The sentence they quote as evidence of this directly qualifies that this is from Android and Chromium.
    - anymouse123456 2y ago
      Please read the quotation more carefully. I appreciate the author calls out the source of the data, but the claims remain overly strong, broad and unqualified. I concede this may not be the strongest example, but in my opinion, the language throughout the article, starting with the title, makes stronger claims than the evidence provided supports. I agree with the author, that these are useful projects to use for research. I'm struggling with the lack of qualification when it comes to the conclusions. Perhaps I missed it, but I also didn't see information about trade-offs experienced in the transition to Rust on these projects. Was there any change related to other kinds of vulnerabilities or defects? How did the transition to Rust impact the number of features introduced over a given time period? Were the engineers able to move as quickly in this (presumably) new-to-them language? I'm under the impression that it can take many engineers multiple years to begin to feel productive in Rust, is there any measure of throughput (even qualitative) that could be compared before, during and after that period? I'm hung up on what reads as a sales pitch that implies broad and deep benefits to any software project of any scope, scale or purpose and makes no mention of trade offs or disadvantages in exchange for this incredible benefit.
      - steveklabnik 2y ago
        As I said, I think you're fine to be skeptical, and there's surely a lot more stuff to be researched in the future, including these questions. I was just trying to speculate on maybe why you got downvotes.
      - throwaway2037 2y ago
        Wow, this an excellent follow-up. You shared so many interesting questions. I too want to know the answers! How about you post these as a comment to their blog post? It might spur them to write a follow-up and dive into more details and nuance.
- hoten 2y ago
  I guess the article could have made it clearer, but software that is not used or where security issues are not actively fixed, they obviously don't apply to the findings. But that kind of code is uninteresting to talk about in the first place.
- bluGill 2y ago
  The idea is you write bug filled code but someone notices and fixes some of those bugs and soethe code counts as newer but it also has less bugs and eventualay enough bugs are fixed that nobody has to touch it and then the now bug free code gets old.
  - fn-mote 2y ago
    I actually don't think this is right. Bug fixes in the old code aren't in the "new, memory safe" code. You're fixing the old code as it gets older, reducing the defects.
    - bluGill 2y ago
      but now the old code has less memory defects
nitwit005 2y ago
I'd just assume the oldest code is written by the best people, and under less time pressure. My current company is an example of this. Early code written by founders. Some new code written by contractors under a tight deadline.

by gortok 2y ago

anymouse123456 2y ago

I'm here with you for the downvotes. This essay takes some interesting data from very specific (and unusual) projects and languages from a very specific (and unusual) culture and stridently extrapolates hard numeric values to all code without qualification. > For example, based on the average vulnerability lifetimes, 5-year-old code has a 3.4x (using lifetimes from the study) to 7.4x (using lifetimes observed in Android and Chromium) lower vulnerability density than new code. Given this conclusion, I can write a defect-filled chunk of C code and just let it marinate for 5 years offline in order for it become safe? I'm pretty sure there are important data in this research and there is truth underneath what is being shared, but the unsupported confidence and overreach of the writing is too distracting for me.

throwaway2037 2y ago
Why is this downvoted? It raises some interesting and important issues. I saw a big bank's back office settlement system once. 30+ years old with almost no unit tests. It changes very little now and is virtually bug free because people have been fixing bugs in it for 30+ years! When they need to make changes these days, they first write unit tests for existing behaviour, then fix the bug. It is an example of how code can mature with a very low defect rate with limited unit tests.
- steveklabnik 2y ago
  I didn't downvote it, but, while I agree that there's reason to be skeptical that this research generalizes, the framing is aggressive. > stridently extrapolates hard numeric values to all code without qualification. The sentence they quote as evidence of this directly qualifies that this is from Android and Chromium.
  - anymouse123456 2y ago
    Please read the quotation more carefully. I appreciate the author calls out the source of the data, but the claims remain overly strong, broad and unqualified. I concede this may not be the strongest example, but in my opinion, the language throughout the article, starting with the title, makes stronger claims than the evidence provided supports. I agree with the author, that these are useful projects to use for research. I'm struggling with the lack of qualification when it comes to the conclusions. Perhaps I missed it, but I also didn't see information about trade-offs experienced in the transition to Rust on these projects. Was there any change related to other kinds of vulnerabilities or defects? How did the transition to Rust impact the number of features introduced over a given time period? Were the engineers able to move as quickly in this (presumably) new-to-them language? I'm under the impression that it can take many engineers multiple years to begin to feel productive in Rust, is there any measure of throughput (even qualitative) that could be compared before, during and after that period? I'm hung up on what reads as a sales pitch that implies broad and deep benefits to any software project of any scope, scale or purpose and makes no mention of trade offs or disadvantages in exchange for this incredible benefit.
    - steveklabnik 2y ago
      As I said, I think you're fine to be skeptical, and there's surely a lot more stuff to be researched in the future, including these questions. I was just trying to speculate on maybe why you got downvotes.
    - throwaway2037 2y ago
      Wow, this an excellent follow-up. You shared so many interesting questions. I too want to know the answers! How about you post these as a comment to their blog post? It might spur them to write a follow-up and dive into more details and nuance.
hoten 2y ago
I guess the article could have made it clearer, but software that is not used or where security issues are not actively fixed, they obviously don't apply to the findings. But that kind of code is uninteresting to talk about in the first place.
bluGill 2y ago
The idea is you write bug filled code but someone notices and fixes some of those bugs and soethe code counts as newer but it also has less bugs and eventualay enough bugs are fixed that nobody has to touch it and then the now bug free code gets old.
- fn-mote 2y ago
  I actually don't think this is right. Bug fixes in the old code aren't in the "new, memory safe" code. You're fixing the old code as it gets older, reducing the defects.
  - bluGill 2y ago
    but now the old code has less memory defects

nitwit005 2y ago

I'd just assume the oldest code is written by the best people, and under less time pressure. My current company is an example of this. Early code written by founders. Some new code written by contractors under a tight deadline.