4 ms·

The thing about C that's always baffled me is the resistance to abstraction that makes its lack of memory safety worse than it has to be. Nothing stops you, eve

The thing about C that's always baffled me is the resistance to abstraction that makes its lack of memory safety worse than it has to be. Nothing stops you, even in C, from defining a "string" data type that maintains an internal buffer size and read position. You can even give such a thing an elegant API, as many people have when they've implemented things like this. But all over the place in C code, you see people open-coding all sorts of fiddly nonsense over and over again. You see people manually manipulate buffer positions and lengths; they'll laboriously type "!strcmp(a, b)" instead of defining a tiny little "streq" function. It's silly. However unsafe C-the-language's memory safety is, the problem is exacerbated by C-the-culture's resistance to using the abstraction features of the language. Why so little imagination? Maybe it's a kind of self-selection thing: maybe if you're the kind of person comfortable with abstraction, you switch to C++, which is much better at it than C and just as efficient, leaving only the open-code-all-string-manipulation people writing raw C.

pjmlp 7y ago
Exactly that, that is why every attempt to fix C security holes has failed, the culture just isn't willing to accept it. Even C++ community is aware of the flaws inherited from C, hence why standard library supports arrays, strings with bounds checking enabled in debug builds and eventually in release as well, if one so wishes. Papers are going forward to reduce amount of UB in C++. Meanwhile, on C's side Annex K was moved into optional in C11, instead of trying to fix whatever implementation issues it might have had on C99.
- loeg 7y ago
  Annex K has never been non-optional. No one has ever implemented a full compliant Annex K, and it's a terrible abstraction. It predates basic static analysis and suffers from more or less the same problems as non-Annex K variants. E.g., if you pass in the wrong buffer length to the K variant, you still get buffer overflows. How do you tell if the wrong length is passed? Static analysis. The same analyzers work with existing libc functions. I do think C or at least POSIX should have adopted strlcpy and strlcat, though; strncpy doesn't do what most people think it does. Source, and more info: http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1967.htm http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1967.htm
  - pjmlp 7y ago
    I got that wrong, thanks for correcting me. I do agree that Annex K doesn't provide a proper solution, as data and length keep being separated. However what I disagree is the way it was left to rotten without adoption, instead of actually fixing it. Static analysis is not a solution, given that all surveys point out that only a minority cares to actually make use of them, or works in environments where they are welcomed, with supportive toolchains. Also they don't work across third party binary dependencies, heavily used in the corporate environments. So unwillingness to improve C language security in the standard shows how little those driving WG 14 care about security. At least C11 made VLAs optional, which never were a good idea to start with.
    - loeg 7y ago
      I'm not saying static analysis is a complete solution, just that Annex K doesn't actually function without it. I.e., any criticism you want to leverage at static analysis also applies to Annex K. (The compiler is one form of static analysis that users can't really get away from, though. Users can turn all the warnings off, but only the extremely novice or foolhardy do so.) I agree letting Annex K hang out optionally in the standard isn't especially useful; in its current form it should be removed. It isn't clear how it would be "fixed" without being something completely different from Annex K. Unfortunately, C hasn't had an actual language update since C11; C18 was just a clarification and errata incorporation update. The C2x committee is in progress but hasn't yet made any binding decisions on formal language changes, AFAIK. I don't know that 3rd party binary dependencies are heavily used in corporate environments. I find that a little difficult to believe. I work in an enterprise environment with a huge codebase and we have no 3rd party binary libraries. We have a number of 3rd party open source or NDA source libraries, but we can and do use static analysis on them. It would be impossible to get to the kind of reliability numbers essential to our business with unauditable, unfixable 3rd party code in our product. WG 14 don't view bounded string copying as some core part of the language, I think. The C standard is about providing a minimal definition of a portable abstract machine; this leaves implementors or more concrete standards, like POSIX, free to provide their own safe abstractions. Quite a lot of the C standard library is sort of legacy accidents that might not be put in the standard language again if reinvented today. But there is obviously a lot of motivation to retain backwards compatibility.
- roca 7y ago
  Unfortunately C++ created new problems like https://github.com/isocpp/CppCoreGuidelines/issues/1038 https://github.com/isocpp/CppCoreGuidelines/issues/1038. Lambdas in particular are a minefield.
  - pjmlp 7y ago
    True, but the answer lies on the latest comments. > So I think that after the lengthy discussion here, this issue can now be closed. In summary: the use-after-free potential with string_view is accounted for in the Lifetime profile, even if there may be bugs or limitations in preliminary implementations that mean it is not diagnosed. This is the big difference across both language communities, being willing to improve instead of being stuck with what was born alongside UNIX.
    - roca 7y ago
      C++ is certainly a better boat to be in thanks to RAII and other things, but static analysis is not a panacea here. > even if there may be bugs or limitations in preliminary implementations that mean it is not diagnosed. The huge assumption there is that finding all or even most lifetime bugs involving string_view and other dangerous C++ idioms via static analysis is tractable, that it's just a matter of "overcoming bugs in preliminary implementations of the analyzer". There is no reason to believe that; in fact there are good reasons to disbelieve that, including: * Languages with static lifetime checking, i.e. Rust, found it necessary to add lifetime annotations, which the C++ lifetime guidelines don't have. * Just after that Github discussion, Herb Sutter released version 1.0 of the C++ Lifetime Guidelines with dramatically scaled-down ambitions. See https://robert.ocallahan.org/2018/09/more-realistic-goals-for-c-lifetimes-10.html https://robert.ocallahan.org/2018/09/more-realistic-goals-fo.... Anyone relying on the unrealistic promises of version 0.9, probably including the commenters in that Github discussion, would have had their hopes dashed. Relying on the emergence of some crazy-powerful static analysis to save us reminds me of the Itanium fiasco, where poor performance was portrayed as a temporary problem while we wait for compilers to "catch up", which of course they never did.
      - 7y ago
jcranmer 7y ago
One of the issues I see with C is that there is a core set of C developers that sees C as the only systems language, being "just" a portable assembler, and if you're not good enough to avoid these pitfalls in C, you shouldn't be programming it. And, as you allude to, I suspect that many of the people with different viewpoints have moved on from the language, leaving this conservative C core in charge of language evolution design.
- quotemstr 7y ago
  I'm reminded of hand-washing in medicine. Even after it became indisputable that hand-washing improved patient outcomes, doctors didn't want to do it: they felt insulted by the implication that their hands weren't already clean. The truth is a doctor could be the most hygienic person in the world and still walk into a surgery with germ-coated hands. A hand-washing requirement isn't a personal insult: it's just a reflection of reality. But pride gets in the way. Likewise, even if you're the best programmer in the world, if you write a large C program, sooner or later, you're going to introduce a security vulnerability. Switching to a safe language isn't an admission of some personal inadequacy: it just reflects how the world works. But pride gets in the way.
  - coretx 7y ago
    People should also not forget that the C culture comes from a age when the philosophy of letting your executable be as simple as possible, do only one thing and do it perfectly & interoperable was so prevalent it did not need discussion. It's a much different ecosystem compared to today; thousands of abstractions, tons of includes, unknown functions etc. In one environment it's "ok" to make "a" mistake, in another it ain't.
roca 7y ago
It's actually pretty hard to write an elegant API for C strings and have it perform well and be safe. Either you do a lot of copying, or you pass pointers around and risk use-after-free. But perhaps the bigger problem is that consuming third-party libraries in C is so painful. It's not too bad if you stick to shipping on Linux distributions which you're sure package the libraries you depend on. Otherwise, you basically have to vendor the library manually. It is always more convenient to just write "strcpy" in a few (more) places.
- quotemstr 7y ago
  > It's actually pretty hard to write an elegant API for C strings and have it perform well and be safe. Either you do a lot of copying, or you pass pointers around and risk use-after-free. Reference-counted heap-allocated strings are fine in practice for almost all programs, and it's easy to write an API that implements this model.
  - roca 7y ago
    Thread-safe refcounts or not? Do you have separate types for "mutable owned string" and "immutable ref-counted string"? Small-string optimizations? You may think there's a single obvious answer for all these decisions but major libraries for other languages have made different decisions for each one. Refcounted strings are actually very nasty to use in C because you have no RAII or other automatic memory management. You have to manually deref every string value when it goes out of scope, or get leaks --- and if you accidentally ever double-deref along some path, you get probably-exploitable use-after-free. Once you've picked some tradeoffs for your program, you have to live with the fact any foreign C you call isn't using your library so you'll be converting to/from their types (typically const char*) all over the place anyway, so the temptation to "just use chars" will be ever-present. Given those issues, I'm not surprised I actually haven't seen any C programs use this model.
    - quotemstr 7y ago
      Yes, these are all questions you need to answer. Thing is, no matter how you answer these questions, what you produce is still far more convenient and much safer than random char buffers everywhere. If sudo had a 200 line internal string library, this vulnerability wouldn't have happened. There's no excuse for open coding lots of random string stuff.

pjmlp 7y ago

Exactly that, that is why every attempt to fix C security holes has failed, the culture just isn't willing to accept it. Even C++ community is aware of the flaws inherited from C, hence why standard library supports arrays, strings with bounds checking enabled in debug builds and eventually in release as well, if one so wishes. Papers are going forward to reduce amount of UB in C++. Meanwhile, on C's side Annex K was moved into optional in C11, instead of trying to fix whatever implementation issues it might have had on C99.

loeg 7y ago
Annex K has never been non-optional. No one has ever implemented a full compliant Annex K, and it's a terrible abstraction. It predates basic static analysis and suffers from more or less the same problems as non-Annex K variants. E.g., if you pass in the wrong buffer length to the K variant, you still get buffer overflows. How do you tell if the wrong length is passed? Static analysis. The same analyzers work with existing libc functions. I do think C or at least POSIX should have adopted strlcpy and strlcat, though; strncpy doesn't do what most people think it does. Source, and more info: http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1967.htm http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1967.htm
- pjmlp 7y ago
  I got that wrong, thanks for correcting me. I do agree that Annex K doesn't provide a proper solution, as data and length keep being separated. However what I disagree is the way it was left to rotten without adoption, instead of actually fixing it. Static analysis is not a solution, given that all surveys point out that only a minority cares to actually make use of them, or works in environments where they are welcomed, with supportive toolchains. Also they don't work across third party binary dependencies, heavily used in the corporate environments. So unwillingness to improve C language security in the standard shows how little those driving WG 14 care about security. At least C11 made VLAs optional, which never were a good idea to start with.
  - loeg 7y ago
    I'm not saying static analysis is a complete solution, just that Annex K doesn't actually function without it. I.e., any criticism you want to leverage at static analysis also applies to Annex K. (The compiler is one form of static analysis that users can't really get away from, though. Users can turn all the warnings off, but only the extremely novice or foolhardy do so.) I agree letting Annex K hang out optionally in the standard isn't especially useful; in its current form it should be removed. It isn't clear how it would be "fixed" without being something completely different from Annex K. Unfortunately, C hasn't had an actual language update since C11; C18 was just a clarification and errata incorporation update. The C2x committee is in progress but hasn't yet made any binding decisions on formal language changes, AFAIK. I don't know that 3rd party binary dependencies are heavily used in corporate environments. I find that a little difficult to believe. I work in an enterprise environment with a huge codebase and we have no 3rd party binary libraries. We have a number of 3rd party open source or NDA source libraries, but we can and do use static analysis on them. It would be impossible to get to the kind of reliability numbers essential to our business with unauditable, unfixable 3rd party code in our product. WG 14 don't view bounded string copying as some core part of the language, I think. The C standard is about providing a minimal definition of a portable abstract machine; this leaves implementors or more concrete standards, like POSIX, free to provide their own safe abstractions. Quite a lot of the C standard library is sort of legacy accidents that might not be put in the standard language again if reinvented today. But there is obviously a lot of motivation to retain backwards compatibility.
roca 7y ago
Unfortunately C++ created new problems like https://github.com/isocpp/CppCoreGuidelines/issues/1038 https://github.com/isocpp/CppCoreGuidelines/issues/1038. Lambdas in particular are a minefield.
- pjmlp 7y ago
  True, but the answer lies on the latest comments. > So I think that after the lengthy discussion here, this issue can now be closed. In summary: the use-after-free potential with string_view is accounted for in the Lifetime profile, even if there may be bugs or limitations in preliminary implementations that mean it is not diagnosed. This is the big difference across both language communities, being willing to improve instead of being stuck with what was born alongside UNIX.
  - roca 7y ago
    C++ is certainly a better boat to be in thanks to RAII and other things, but static analysis is not a panacea here. > even if there may be bugs or limitations in preliminary implementations that mean it is not diagnosed. The huge assumption there is that finding all or even most lifetime bugs involving string_view and other dangerous C++ idioms via static analysis is tractable, that it's just a matter of "overcoming bugs in preliminary implementations of the analyzer". There is no reason to believe that; in fact there are good reasons to disbelieve that, including: * Languages with static lifetime checking, i.e. Rust, found it necessary to add lifetime annotations, which the C++ lifetime guidelines don't have. * Just after that Github discussion, Herb Sutter released version 1.0 of the C++ Lifetime Guidelines with dramatically scaled-down ambitions. See https://robert.ocallahan.org/2018/09/more-realistic-goals-for-c-lifetimes-10.html https://robert.ocallahan.org/2018/09/more-realistic-goals-fo.... Anyone relying on the unrealistic promises of version 0.9, probably including the commenters in that Github discussion, would have had their hopes dashed. Relying on the emergence of some crazy-powerful static analysis to save us reminds me of the Itanium fiasco, where poor performance was portrayed as a temporary problem while we wait for compilers to "catch up", which of course they never did.
    - 7y ago

jcranmer 7y ago

One of the issues I see with C is that there is a core set of C developers that sees C as the only systems language, being "just" a portable assembler, and if you're not good enough to avoid these pitfalls in C, you shouldn't be programming it. And, as you allude to, I suspect that many of the people with different viewpoints have moved on from the language, leaving this conservative C core in charge of language evolution design.

quotemstr 7y ago
I'm reminded of hand-washing in medicine. Even after it became indisputable that hand-washing improved patient outcomes, doctors didn't want to do it: they felt insulted by the implication that their hands weren't already clean. The truth is a doctor could be the most hygienic person in the world and still walk into a surgery with germ-coated hands. A hand-washing requirement isn't a personal insult: it's just a reflection of reality. But pride gets in the way. Likewise, even if you're the best programmer in the world, if you write a large C program, sooner or later, you're going to introduce a security vulnerability. Switching to a safe language isn't an admission of some personal inadequacy: it just reflects how the world works. But pride gets in the way.
- coretx 7y ago
  People should also not forget that the C culture comes from a age when the philosophy of letting your executable be as simple as possible, do only one thing and do it perfectly & interoperable was so prevalent it did not need discussion. It's a much different ecosystem compared to today; thousands of abstractions, tons of includes, unknown functions etc. In one environment it's "ok" to make "a" mistake, in another it ain't.

roca 7y ago

It's actually pretty hard to write an elegant API for C strings and have it perform well and be safe. Either you do a lot of copying, or you pass pointers around and risk use-after-free. But perhaps the bigger problem is that consuming third-party libraries in C is so painful. It's not too bad if you stick to shipping on Linux distributions which you're sure package the libraries you depend on. Otherwise, you basically have to vendor the library manually. It is always more convenient to just write "strcpy" in a few (more) places.

quotemstr 7y ago
> It's actually pretty hard to write an elegant API for C strings and have it perform well and be safe. Either you do a lot of copying, or you pass pointers around and risk use-after-free. Reference-counted heap-allocated strings are fine in practice for almost all programs, and it's easy to write an API that implements this model.
- roca 7y ago
  Thread-safe refcounts or not? Do you have separate types for "mutable owned string" and "immutable ref-counted string"? Small-string optimizations? You may think there's a single obvious answer for all these decisions but major libraries for other languages have made different decisions for each one. Refcounted strings are actually very nasty to use in C because you have no RAII or other automatic memory management. You have to manually deref every string value when it goes out of scope, or get leaks --- and if you accidentally ever double-deref along some path, you get probably-exploitable use-after-free. Once you've picked some tradeoffs for your program, you have to live with the fact any foreign C you call isn't using your library so you'll be converting to/from their types (typically const char*) all over the place anyway, so the temptation to "just use chars" will be ever-present. Given those issues, I'm not surprised I actually haven't seen any C programs use this model.
  - quotemstr 7y ago
    Yes, these are all questions you need to answer. Thing is, no matter how you answer these questions, what you produce is still far more convenient and much safer than random char buffers everywhere. If sudo had a 200 line internal string library, this vulnerability wouldn't have happened. There's no excuse for open coding lots of random string stuff.