10 ms·

The flip side of getting great performance is all the exposure to security vulnerabilities. Git seems to have an excellent security track record: https://www.c

The flip side of getting great performance is all the exposure to security vulnerabilities. Git seems to have an excellent security track record: https://www.cvedetails.com/vulnerability-list/vendor_id-4008/GIT.html https://www.cvedetails.com/vulnerability-list/vendor_id-4008... However, I think the right claim here is, "be very sure you have the chops to benefit from C's speed without compromising security."

YPPH 5y ago
Is that necessarily the case? From what I've heard, Rust offers great performance and safety.
- II2II 5y ago
  From my understanding, Rust performs more compile time checks for safety. In those cases, you may be able to maintain the performance of C without sacrificing safety. On the other hand, you may still get away with writing more performant C code since C allows the developer to play fast and loose with the rules. I'm not saying that is a good thing, but git can get away with it since it since it probably attracts very competent developers. The other thing mentioned in the post was eliminating overhead. My limited experience with Rust leads me to believe it is closer to C++ than C in this respect. That is to say there is a lot of higher level functionality that you can use and it is optimized for performance in the general case. On the other hand, that functionality is still for the general case, so a competent developer who has a thorough knowledge of what the program is trying to accomplish will be able to write more performant code.
  - School-Cotton 5y ago
    > C allows the developer to play fast and loose with the rules. So does Rust, in unsafe blocks. Is there anything that can be done in C that can’t be done in unsafe Rust?
    - vvanders 5y ago
      Only thing that comes to mind immediately is VLA/alloca but I'd have to check. Most of the gnarly pointer things are fair game and work just as well as their C counterparts.
      - bbatha 5y ago
        Rust supports VLAs, Box<[T]> is pretty common in fact. It just doesn’t have a native way to alloc them on the stack.
      - pjmlp 5y ago
        VLA are thankfully no longer required for ISO C compliance, who had thought it was a good idea in first place? Google even sponsored the work to clean the Linux kernel from them.
    - cesarb 5y ago
      > Is there anything that can be done in C that can’t be done in unsafe Rust? Duff's device <https://en.wikipedia.org/wiki/Duff's_device https://en.wikipedia.org/wiki/Duff's_device>.
      - josephcsible 5y ago
        You're mixing up a thing you can do with a way that you can do something. You can write Rust code with the same observable behavior as Duff's device in C.
        kaba0 5y ago
        Also, Duff’s device is pretty terrible and a modern compiler will output much better code.
        LAC-Tech 5y ago
        > You're mixing up a thing you can do with a way that you can do something. I don't think they are. > You can write Rust code with the same observable behavior as Duff's device in C. Yes, turing complete languages can do that.
  - nine_k 5y ago
    That C allows developers to play fast and loose is actually one of the reasons why C is slower than Fortran in many cases. Aliasing does not exist in Fortran, and array dimensions are well-defined, and this alone allows for much more aggressive optimizations, impossible to do safely in C. (Is Fortran still relevant? Try compiling Tensorflow, to say nothing of serious numerical modeling stuff. And yes, I know about __restrict; it's more of a band-aid.)
    - therealcamino 5y ago
      > Aliasing does not exist in Fortran Ha, I have some disgusting use of COMMON blocks from the 80's to show you. But that doesn't affect the optimizer's ability to assume that there's no aliasing, so you're totally correct about that.
    - EVa5I7bHFq9mnYK 5y ago
      The version of Fortran I worked with had no stack nor heap. Needless to say, it was very fast.
  - ekidd 5y ago
    > On the other hand, you may still get away with writing more performant C code since C allows the developer to play fast and loose with the rules. I have some thoughts on how Rust performance compares with C! I spent yesterday optimizing a sort routine in Rust. I need to sort records where the key length is known at compile-time, and the payload length is different every time sort is called. I expect the average sort call to involve at least 250 million rows and over 400 columns. Speed matters. Here's what I've learned so far, about how Rust versus C question: - Rust has generics with monomorphization. This makes it easy to compile two different versions of the sort routine to run on different length keys. - The very fastest version of my recursive base case, a 20-item insertion sort, currently uses unsafe code. This allows me to eliminate a couple of bounds checks that LLVM isn't eliminating automatically. I may figure out how to do this with safe code before shipping. (In 5+ years of production Rust, I've never needed unsafe for performance. This may be the first time!) - Performance often comes down to cache locality. Both C and Rust give me the fine-grained memory layout control that's essential. Also, this means that "detached key" sorts are almost always a bad choice, even when the alternative is repeatly moving large record payloads around. - The Rust "criterion" benchmarking library makes it super-easy to run valid performance tests. - The Rust "proptest" allows me to generate large amounts of test data and verify properties like "reckless_sort always produces output matching std sort." - "cargo fuzz" will be useful when I try to break the finished code. Seriously, it's a super-nice fuzzer workflow. - Rust's standard quicksort contains all sorts of funky optimizations: It detects semi-sorted arrays, it breaks patterns, it falls back to merge sort when things go wrong. It even marks "cold" paths for LLVM to improve instruction cache usage. - If I get my sort working, the "rayon" library will make it utterly painless and safe to split up the recursive calls over multiple CPUs. Overall, this has been a pretty fun experience. I'm not sure whether the finished product will use unsafe Rust in the inner loop. But the combination of generics, rayon, and tightly integrated test/bench/fuzz tooling has made this a very pleasant experience. Verdict: I am entirely happy using Rust for hot loops. (Even if my current fastest inner loop does use "unsafe" to bypass a few bounds checks I'm not clever enough to eliminate otherwise.)
chii 5y ago
> getting great performance is all the exposure to security vulnerabilities. ironic to post this right after java's biggest vuln to date (log4shell).
- winrid 5y ago
  Which had nothing to do with Java or how it manages memory. You could have the same vuln in NodeJS or Python.
  - ajross 5y ago
    Which argues that maybe language choice just doesn't matter that much for security. It's true that C allows a class of mistakes that don't exist in higher level runtimes. It's equally true that this class of mistakes represents an increasingly vanishing share of real world exploits. Static analysis tools and runtime hardening techniques don't "fix" C, exactly. But in practice they work well enough to push C's foibles down into the noise floor. But at the same time, C remains, and will probably always remain, the easiest language on which to tune and optimize. It's not going anywhere. Our grandkids will still be using systems with C firmware at their core.
    - School-Cotton 5y ago
      > It's equally true that this class of mistakes represents an increasingly vanishing share of real world exploits. Is this true? It seems implausible to me.
    - friedman23 5y ago
      > Which argues that maybe language choice just doesn't matter that much for security No it doesn't. There is actually no logical way to infer from the parent statement that language choice doesn't matter for security. > But in practice they work well enough to push C's foibles down into the noise floor. This goes against findings from research that has been done into the sources of security vulnerabilities. Microsoft and the chrome dev teams have published that 70% of their security bugs are a result of memory safety. If C is used in 100 years it will only be because of inertia. Today there are better choices in the domain of low level systems programming languages.
throwawaylinux 5y ago
There is a single memory error CVE there, the last one. Assuming a higher level language would not allow that without introducing any other security problems, and assuming the CVEs roughly represent the secureness of the software, then using a higher level language would be a pretty small marginal improvement in security.
a-dub 5y ago
it's kinda funny really. what you gain by eliminating memory bugs you lose in boundless complexity sprawl that is basically encouraged by the design of the language and its major libraries.
- throwaway894345 5y ago
  The complexity isn’t new, it’s just moved from manually managing dependencies or reinventing wheels into more fruitful, higher level applications. Of course, some of these are frivolous, but probably no worse than spending one’s complexity budget wading through dependency hell.
jmspring 5y ago
C can be written in a way to minimize vulnerabilities. You use things like critical thinking and sanitize/validate inputs to functions and before you call libraries. The log4j vulnerabilities and others like it where input validation specific to what you expect would not happen in properly paranoid systems.
- adrianN 5y ago
  There are approximately zero codebases of nontrivial size that are written in memory unsafe languages and free of serious security holes that would be impossible in safer languages. That is pretty good evidence that the measures you suggest are not very effective.
  - oynqr 5y ago
    Doesn't OpenBSD have a decent track record?
  - bear8642 5y ago
    Still approx. zero, but seL4 kernel's written in C and proven correct preventing many security holes memory control prevents
    - iamtedd 5y ago
      seL4 was released as open-source in 2014. I know of the helicopter drone; have any other projects made use of seL4? If not, why not? My naive guess is that it's too hard to build on the project without introducing vulnerabilities and therefore negating the benefits of the kernel, but I'd love to be shown otherwise.
    - erik_seaberg 5y ago
      Wasn’t that generated from Haskell source rather than handwritten, after years of formal verification of not only the source but also the C code generator and the final machine code?
      - 5y ago
1vuio0pswjnm7 5y ago
There are other factors. Not every application of C has the same amount of exposure to security vulnerabilities. On Github I see people rewriting simple UNIX utilties, e.g, cat, in some "safe" language, e.g., Rust. Yes, it is possible there could be some exploit based on an error in cat, maybe triggered by some malicious input that the user fails to detect, but this would not be a concern to the same degreee as errors in large, complex applications with network and other privileged access that are relatively new. For example, using C to write cat versus using C to write git. Exposure to security vulnerabilities depends not only on the language used, but also when, where and how the application is used.
- throwaway894345 5y ago
  Security is only one reason to rewrite utilities in another language. In particular for these foundational utilities, it means more of your dependency tree can be reproducibly built without the help of specialist package maintainers. Other reasons also include “fun” and “learning”.
  - kovac 5y ago
    Considering that package mgmt is a well solved problem, rewriting these tools for that in return for slow, bloated alternative sounds like a huge tradeoff to me. Fun and learning on the other hand, sure.
    - adrianN 5y ago
      I'm not so sure that package management is really a well solved problem. Sure, there are some solutions, but as far as I know all leave some things to be desired.
      - nineteen999 5y ago
        The obvious solution, of course, is to write yet another package manager.
    - 5y ago

YPPH 5y ago

Is that necessarily the case? From what I've heard, Rust offers great performance and safety.

II2II 5y ago
From my understanding, Rust performs more compile time checks for safety. In those cases, you may be able to maintain the performance of C without sacrificing safety. On the other hand, you may still get away with writing more performant C code since C allows the developer to play fast and loose with the rules. I'm not saying that is a good thing, but git can get away with it since it since it probably attracts very competent developers. The other thing mentioned in the post was eliminating overhead. My limited experience with Rust leads me to believe it is closer to C++ than C in this respect. That is to say there is a lot of higher level functionality that you can use and it is optimized for performance in the general case. On the other hand, that functionality is still for the general case, so a competent developer who has a thorough knowledge of what the program is trying to accomplish will be able to write more performant code.
- School-Cotton 5y ago
  > C allows the developer to play fast and loose with the rules. So does Rust, in unsafe blocks. Is there anything that can be done in C that can’t be done in unsafe Rust?
  - vvanders 5y ago
    Only thing that comes to mind immediately is VLA/alloca but I'd have to check. Most of the gnarly pointer things are fair game and work just as well as their C counterparts.
    - bbatha 5y ago
      Rust supports VLAs, Box<[T]> is pretty common in fact. It just doesn’t have a native way to alloc them on the stack.
    - pjmlp 5y ago
      VLA are thankfully no longer required for ISO C compliance, who had thought it was a good idea in first place? Google even sponsored the work to clean the Linux kernel from them.
  - cesarb 5y ago
    > Is there anything that can be done in C that can’t be done in unsafe Rust? Duff's device <https://en.wikipedia.org/wiki/Duff's_device https://en.wikipedia.org/wiki/Duff's_device>.
    - josephcsible 5y ago
      You're mixing up a thing you can do with a way that you can do something. You can write Rust code with the same observable behavior as Duff's device in C.
      - kaba0 5y ago
        Also, Duff’s device is pretty terrible and a modern compiler will output much better code.
      - LAC-Tech 5y ago
        > You're mixing up a thing you can do with a way that you can do something. I don't think they are. > You can write Rust code with the same observable behavior as Duff's device in C. Yes, turing complete languages can do that.
- nine_k 5y ago
  That C allows developers to play fast and loose is actually one of the reasons why C is slower than Fortran in many cases. Aliasing does not exist in Fortran, and array dimensions are well-defined, and this alone allows for much more aggressive optimizations, impossible to do safely in C. (Is Fortran still relevant? Try compiling Tensorflow, to say nothing of serious numerical modeling stuff. And yes, I know about __restrict; it's more of a band-aid.)
  - therealcamino 5y ago
    > Aliasing does not exist in Fortran Ha, I have some disgusting use of COMMON blocks from the 80's to show you. But that doesn't affect the optimizer's ability to assume that there's no aliasing, so you're totally correct about that.
  - EVa5I7bHFq9mnYK 5y ago
    The version of Fortran I worked with had no stack nor heap. Needless to say, it was very fast.
- ekidd 5y ago
  > On the other hand, you may still get away with writing more performant C code since C allows the developer to play fast and loose with the rules. I have some thoughts on how Rust performance compares with C! I spent yesterday optimizing a sort routine in Rust. I need to sort records where the key length is known at compile-time, and the payload length is different every time sort is called. I expect the average sort call to involve at least 250 million rows and over 400 columns. Speed matters. Here's what I've learned so far, about how Rust versus C question: - Rust has generics with monomorphization. This makes it easy to compile two different versions of the sort routine to run on different length keys. - The very fastest version of my recursive base case, a 20-item insertion sort, currently uses unsafe code. This allows me to eliminate a couple of bounds checks that LLVM isn't eliminating automatically. I may figure out how to do this with safe code before shipping. (In 5+ years of production Rust, I've never needed unsafe for performance. This may be the first time!) - Performance often comes down to cache locality. Both C and Rust give me the fine-grained memory layout control that's essential. Also, this means that "detached key" sorts are almost always a bad choice, even when the alternative is repeatly moving large record payloads around. - The Rust "criterion" benchmarking library makes it super-easy to run valid performance tests. - The Rust "proptest" allows me to generate large amounts of test data and verify properties like "reckless_sort always produces output matching std sort." - "cargo fuzz" will be useful when I try to break the finished code. Seriously, it's a super-nice fuzzer workflow. - Rust's standard quicksort contains all sorts of funky optimizations: It detects semi-sorted arrays, it breaks patterns, it falls back to merge sort when things go wrong. It even marks "cold" paths for LLVM to improve instruction cache usage. - If I get my sort working, the "rayon" library will make it utterly painless and safe to split up the recursive calls over multiple CPUs. Overall, this has been a pretty fun experience. I'm not sure whether the finished product will use unsafe Rust in the inner loop. But the combination of generics, rayon, and tightly integrated test/bench/fuzz tooling has made this a very pleasant experience. Verdict: I am entirely happy using Rust for hot loops. (Even if my current fastest inner loop does use "unsafe" to bypass a few bounds checks I'm not clever enough to eliminate otherwise.)

chii 5y ago

> getting great performance is all the exposure to security vulnerabilities. ironic to post this right after java's biggest vuln to date (log4shell).

winrid 5y ago
Which had nothing to do with Java or how it manages memory. You could have the same vuln in NodeJS or Python.
- ajross 5y ago
  Which argues that maybe language choice just doesn't matter that much for security. It's true that C allows a class of mistakes that don't exist in higher level runtimes. It's equally true that this class of mistakes represents an increasingly vanishing share of real world exploits. Static analysis tools and runtime hardening techniques don't "fix" C, exactly. But in practice they work well enough to push C's foibles down into the noise floor. But at the same time, C remains, and will probably always remain, the easiest language on which to tune and optimize. It's not going anywhere. Our grandkids will still be using systems with C firmware at their core.
  - School-Cotton 5y ago
    > It's equally true that this class of mistakes represents an increasingly vanishing share of real world exploits. Is this true? It seems implausible to me.
  - friedman23 5y ago
    > Which argues that maybe language choice just doesn't matter that much for security No it doesn't. There is actually no logical way to infer from the parent statement that language choice doesn't matter for security. > But in practice they work well enough to push C's foibles down into the noise floor. This goes against findings from research that has been done into the sources of security vulnerabilities. Microsoft and the chrome dev teams have published that 70% of their security bugs are a result of memory safety. If C is used in 100 years it will only be because of inertia. Today there are better choices in the domain of low level systems programming languages.

throwawaylinux 5y ago

There is a single memory error CVE there, the last one. Assuming a higher level language would not allow that without introducing any other security problems, and assuming the CVEs roughly represent the secureness of the software, then using a higher level language would be a pretty small marginal improvement in security.

a-dub 5y ago

it's kinda funny really. what you gain by eliminating memory bugs you lose in boundless complexity sprawl that is basically encouraged by the design of the language and its major libraries.

throwaway894345 5y ago
The complexity isn’t new, it’s just moved from manually managing dependencies or reinventing wheels into more fruitful, higher level applications. Of course, some of these are frivolous, but probably no worse than spending one’s complexity budget wading through dependency hell.

jmspring 5y ago

C can be written in a way to minimize vulnerabilities. You use things like critical thinking and sanitize/validate inputs to functions and before you call libraries. The log4j vulnerabilities and others like it where input validation specific to what you expect would not happen in properly paranoid systems.

adrianN 5y ago
There are approximately zero codebases of nontrivial size that are written in memory unsafe languages and free of serious security holes that would be impossible in safer languages. That is pretty good evidence that the measures you suggest are not very effective.
- oynqr 5y ago
  Doesn't OpenBSD have a decent track record?
- bear8642 5y ago
  Still approx. zero, but seL4 kernel's written in C and proven correct preventing many security holes memory control prevents
  - iamtedd 5y ago
    seL4 was released as open-source in 2014. I know of the helicopter drone; have any other projects made use of seL4? If not, why not? My naive guess is that it's too hard to build on the project without introducing vulnerabilities and therefore negating the benefits of the kernel, but I'd love to be shown otherwise.
  - erik_seaberg 5y ago
    Wasn’t that generated from Haskell source rather than handwritten, after years of formal verification of not only the source but also the C code generator and the final machine code?
    - 5y ago

1vuio0pswjnm7 5y ago

There are other factors. Not every application of C has the same amount of exposure to security vulnerabilities. On Github I see people rewriting simple UNIX utilties, e.g, cat, in some "safe" language, e.g., Rust. Yes, it is possible there could be some exploit based on an error in cat, maybe triggered by some malicious input that the user fails to detect, but this would not be a concern to the same degreee as errors in large, complex applications with network and other privileged access that are relatively new. For example, using C to write cat versus using C to write git. Exposure to security vulnerabilities depends not only on the language used, but also when, where and how the application is used.

throwaway894345 5y ago
Security is only one reason to rewrite utilities in another language. In particular for these foundational utilities, it means more of your dependency tree can be reproducibly built without the help of specialist package maintainers. Other reasons also include “fun” and “learning”.
- kovac 5y ago
  Considering that package mgmt is a well solved problem, rewriting these tools for that in return for slow, bloated alternative sounds like a huge tradeoff to me. Fun and learning on the other hand, sure.
  - adrianN 5y ago
    I'm not so sure that package management is really a well solved problem. Sure, there are some solutions, but as far as I know all leave some things to be desired.
    - nineteen999 5y ago
      The obvious solution, of course, is to write yet another package manager.
  - 5y ago