4 ms·

I'm here with you for the downvotes. This essay takes some interesting data from very specific (and unusual) projects and languages from a very specific (and u

I'm here with you for the downvotes. This essay takes some interesting data from very specific (and unusual) projects and languages from a very specific (and unusual) culture and stridently extrapolates hard numeric values to all code without qualification. > For example, based on the average vulnerability lifetimes, 5-year-old code has a 3.4x (using lifetimes from the study) to 7.4x (using lifetimes observed in Android and Chromium) lower vulnerability density than new code. Given this conclusion, I can write a defect-filled chunk of C code and just let it marinate for 5 years offline in order for it become safe? I'm pretty sure there are important data in this research and there is truth underneath what is being shared, but the unsupported confidence and overreach of the writing is too distracting for me.

throwaway2037 2y ago
Why is this downvoted? It raises some interesting and important issues. I saw a big bank's back office settlement system once. 30+ years old with almost no unit tests. It changes very little now and is virtually bug free because people have been fixing bugs in it for 30+ years! When they need to make changes these days, they first write unit tests for existing behaviour, then fix the bug. It is an example of how code can mature with a very low defect rate with limited unit tests.
- steveklabnik 2y ago
  I didn't downvote it, but, while I agree that there's reason to be skeptical that this research generalizes, the framing is aggressive. > stridently extrapolates hard numeric values to all code without qualification. The sentence they quote as evidence of this directly qualifies that this is from Android and Chromium.
  - anymouse123456 2y ago
    Please read the quotation more carefully. I appreciate the author calls out the source of the data, but the claims remain overly strong, broad and unqualified. I concede this may not be the strongest example, but in my opinion, the language throughout the article, starting with the title, makes stronger claims than the evidence provided supports. I agree with the author, that these are useful projects to use for research. I'm struggling with the lack of qualification when it comes to the conclusions. Perhaps I missed it, but I also didn't see information about trade-offs experienced in the transition to Rust on these projects. Was there any change related to other kinds of vulnerabilities or defects? How did the transition to Rust impact the number of features introduced over a given time period? Were the engineers able to move as quickly in this (presumably) new-to-them language? I'm under the impression that it can take many engineers multiple years to begin to feel productive in Rust, is there any measure of throughput (even qualitative) that could be compared before, during and after that period? I'm hung up on what reads as a sales pitch that implies broad and deep benefits to any software project of any scope, scale or purpose and makes no mention of trade offs or disadvantages in exchange for this incredible benefit.
    - steveklabnik 2y ago
      As I said, I think you're fine to be skeptical, and there's surely a lot more stuff to be researched in the future, including these questions. I was just trying to speculate on maybe why you got downvotes.
    - throwaway2037 2y ago
      Wow, this an excellent follow-up. You shared so many interesting questions. I too want to know the answers! How about you post these as a comment to their blog post? It might spur them to write a follow-up and dive into more details and nuance.
    - steveklabnik 2y ago
      Oh, a day later, I realized that there have been some of these questions of yours answered, but previously: > I also didn't see information about trade-offs experienced in the transition to Rust on these projects. Yeah, I mean that's just not the topic of this particular post. But they have talked about it. Specifically this, from 2023: https://opensource.googleblog.com/2023/06/rust-fact-vs-fiction-5-insights-from-googles-rust-journey-2022.html https://opensource.googleblog.com/2023/06/rust-fact-vs-ficti... > I'm under the impression that it can take many engineers multiple years to begin to feel productive in Rust, is there any measure of throughput (even qualitative) that could be compared before, during and after that period? That is something that people say, but Google found differently. As that post says, "more than 2/3 of respondents are confident in contributing to a Rust codebase within two months or less when learning Rust. Further, a third of respondents become as productive using Rust as other languages in two months or less." On this one: > Were the engineers able to move as quickly in this (presumably) new-to-them language? Regarding "presumably," 13% had Rust experience, but the rest did not. They say "a third of respondents become as productive using Rust as other languages in two months or less" and "we’ve seen no data to indicate that there is any productivity penalty for Rust relative to any other language these developers previously used at Google."
      - anymouse123456 2y ago
        Thanks for such a detailed response and especially the new link, it's really helpful.
hoten 2y ago
I guess the article could have made it clearer, but software that is not used or where security issues are not actively fixed, they obviously don't apply to the findings. But that kind of code is uninteresting to talk about in the first place.
bluGill 2y ago
The idea is you write bug filled code but someone notices and fixes some of those bugs and soethe code counts as newer but it also has less bugs and eventualay enough bugs are fixed that nobody has to touch it and then the now bug free code gets old.
- fn-mote 2y ago
  I actually don't think this is right. Bug fixes in the old code aren't in the "new, memory safe" code. You're fixing the old code as it gets older, reducing the defects.
  - bluGill 2y ago
    but now the old code has less memory defects

throwaway2037 2y ago

Why is this downvoted? It raises some interesting and important issues. I saw a big bank's back office settlement system once. 30+ years old with almost no unit tests. It changes very little now and is virtually bug free because people have been fixing bugs in it for 30+ years! When they need to make changes these days, they first write unit tests for existing behaviour, then fix the bug. It is an example of how code can mature with a very low defect rate with limited unit tests.

steveklabnik 2y ago
I didn't downvote it, but, while I agree that there's reason to be skeptical that this research generalizes, the framing is aggressive. > stridently extrapolates hard numeric values to all code without qualification. The sentence they quote as evidence of this directly qualifies that this is from Android and Chromium.
- anymouse123456 2y ago
  Please read the quotation more carefully. I appreciate the author calls out the source of the data, but the claims remain overly strong, broad and unqualified. I concede this may not be the strongest example, but in my opinion, the language throughout the article, starting with the title, makes stronger claims than the evidence provided supports. I agree with the author, that these are useful projects to use for research. I'm struggling with the lack of qualification when it comes to the conclusions. Perhaps I missed it, but I also didn't see information about trade-offs experienced in the transition to Rust on these projects. Was there any change related to other kinds of vulnerabilities or defects? How did the transition to Rust impact the number of features introduced over a given time period? Were the engineers able to move as quickly in this (presumably) new-to-them language? I'm under the impression that it can take many engineers multiple years to begin to feel productive in Rust, is there any measure of throughput (even qualitative) that could be compared before, during and after that period? I'm hung up on what reads as a sales pitch that implies broad and deep benefits to any software project of any scope, scale or purpose and makes no mention of trade offs or disadvantages in exchange for this incredible benefit.
  - steveklabnik 2y ago
    As I said, I think you're fine to be skeptical, and there's surely a lot more stuff to be researched in the future, including these questions. I was just trying to speculate on maybe why you got downvotes.
  - throwaway2037 2y ago
    Wow, this an excellent follow-up. You shared so many interesting questions. I too want to know the answers! How about you post these as a comment to their blog post? It might spur them to write a follow-up and dive into more details and nuance.
  - steveklabnik 2y ago
    Oh, a day later, I realized that there have been some of these questions of yours answered, but previously: > I also didn't see information about trade-offs experienced in the transition to Rust on these projects. Yeah, I mean that's just not the topic of this particular post. But they have talked about it. Specifically this, from 2023: https://opensource.googleblog.com/2023/06/rust-fact-vs-fiction-5-insights-from-googles-rust-journey-2022.html https://opensource.googleblog.com/2023/06/rust-fact-vs-ficti... > I'm under the impression that it can take many engineers multiple years to begin to feel productive in Rust, is there any measure of throughput (even qualitative) that could be compared before, during and after that period? That is something that people say, but Google found differently. As that post says, "more than 2/3 of respondents are confident in contributing to a Rust codebase within two months or less when learning Rust. Further, a third of respondents become as productive using Rust as other languages in two months or less." On this one: > Were the engineers able to move as quickly in this (presumably) new-to-them language? Regarding "presumably," 13% had Rust experience, but the rest did not. They say "a third of respondents become as productive using Rust as other languages in two months or less" and "we’ve seen no data to indicate that there is any productivity penalty for Rust relative to any other language these developers previously used at Google."
    - anymouse123456 2y ago
      Thanks for such a detailed response and especially the new link, it's really helpful.

hoten 2y ago

I guess the article could have made it clearer, but software that is not used or where security issues are not actively fixed, they obviously don't apply to the findings. But that kind of code is uninteresting to talk about in the first place.

bluGill 2y ago

The idea is you write bug filled code but someone notices and fixes some of those bugs and soethe code counts as newer but it also has less bugs and eventualay enough bugs are fixed that nobody has to touch it and then the now bug free code gets old.

fn-mote 2y ago
I actually don't think this is right. Bug fixes in the old code aren't in the "new, memory safe" code. You're fixing the old code as it gets older, reducing the defects.
- bluGill 2y ago
  but now the old code has less memory defects