6 ms·

Into the Depths of C: Elaborating the De Facto Standards [pdf]

115 points by jsnell 10y ago

kerneis 10y ago
If you think you know C quite well, here is one of the studies the authors ran to elaborate their semantics on corner cases of the language: http://www.cl.cam.ac.uk/~pes20/cerberus/notes50-survey-discussion.html http://www.cl.cam.ac.uk/~pes20/cerberus/notes50-survey-discu... > If you zero all bytes of a struct and then write some of its members, do reads of the padding return zero? (e.g. for a bytewise CAS or hash of the struct, or to know that no security-relevant data has leaked into them.) (and 14 other questions) Webpage of the project: http://www.cl.cam.ac.uk/~pes20/cerberus/ http://www.cl.cam.ac.uk/~pes20/cerberus/
- blastrat 10y ago
  I knew C quite well. Haven't written any for years. The statements "zero all bytes of a struct" and "reads of the padding" contain enough ambiguity that it answers the question. Not to mention the ambiguity in the words "read" and "write" as they pertain to C, since they already have a "std" meaning that's not the same as lvalue or rvalue, so what exactly do they mean here? And if you think you can answer the question without resolving the ambiguities, that answers some other questions.
  - kerneis 10y ago
    I believe the questions in this study (I did not write it, only know the authors) were deliberately open-ended, allowing for comments on the specifics. A previous, much longer version used to contain code examples to comment, but it proved too detailed for people to complete. Moreover, the study was explicitly not about ISO C: "We were not asking what the ISO C standard permits, which is often more restrictive, or about obsolete or obscure hardware or compilers. We focussed on the behaviour of memory and pointers. This is a step towards an unambiguous and mathematically precise definition of the de facto standards: the C dialects that are actually used by systems programmers and implemented by mainstream compilers." Here is an actual example of a comment to this question: I would expect this code to work: struct foo { char a; double b; }; foo p; foo q; memset( &p, 0, sizeof( p ) ); memset( &q, 0, sizeof( q ) ); p.a = 1; q.a = 1; assert( memcmp( &p, &q, sizeof( foo ) ) == 0 );
    - psewell 10y ago
      Indeed. The long version is at [pdf] http://www.cl.cam.ac.uk/~pes20/cerberus/notes30-full.pdf; http://www.cl.cam.ac.uk/~pes20/cerberus/notes30-full.pdf; it has 85 questions supported by concrete code examples and experimental data, e.g. (one of several questions that refine the above): Q64. After an explicit write of zero to a padding byte followed by a write to adjacent members of the structure, does the padding byte hold a well-defined zero value? (not an unspecified value) #include <stdio.h> #include <stddef.h> typedef struct { char c; float f; int i; } st; int main() { // check there is a padding byte between c and f size_t offset_padding = offsetof(st,c)+sizeof(char); if (offsetof(st,f)>offset_padding) { st s; unsigned char *p = ((unsigned char*)(&s)) + offset_padding; *p = 0; s.c = 'A'; s.f = 1.0; s.i = 42; unsigned char c3 = *p; // does c3 hold 0, not an unspecified value? printf("c3=0x%x\n",c3); } return 0; } Some of the questions have clear answers with respect to either the ISO or de facto standards, but many do not - that's the point.
    - Too 10y ago
      Is memset and memcmp compatible with strict aliasing? Intuitively it seems like that would be a gap into the aliasing rules. Altough void* is allowed to alias anything so maybe it works through that. Ive never seen memset, memcpy or memcmp on anything but char* in production code.
      - gpderetta 10y ago
        The aliasing rules only talk about dereferencing pointers. Void* can't be dereferenced so it has no interaction with the aliasing rules. You might be thinking about char, and yes, you are allowed to dereference char to inspect the bytes of an object, which is what the various mem* functions do under the hood.
- wahern 10y ago
  Here's a similar, real-world case involving GCC 6's more aggressive SRA and alias-analysis optimizations: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=71120 https://gcc.gnu.org/bugzilla/show_bug.cgi?id=71120
paxcoder 10y ago
To write portable code, I wouldn't study de-facto definitions of de-jure undefined behavior, except to see if I could cover every possible one and only if all alternatives were inferior.
- pm215 10y ago
  Indeed not, but enquiring into what the in-the-wild de-facto beliefs about behaviour are might help in deciding what the de-jure rules should be changed to, or what a compiler implementation ought to do if it cares about what it does on the vast mass of code out there that does commit undefined behaviour, wittingly or otherwise...
  - paxcoder 10y ago
    Undefined behavior has a purpose: Not specifying implementation details makes it easier to write new implementations and for a wider variety of platforms. "De facto standards" take away this freedom, so ideally you'd want to reject reliance on UD, but I see your (second) point about that not always being practical. I guess "be conservative in what you do, be liberal in what you accept from others". Just make sure that your foundations are strong (pun) or the whole house will be an EcmaScript.
TickleSteve 10y ago
I've been using C for over 20 years and I'm sure I would be caught out by these... ...but... The fact these curiosities are not an issue in day-to-day work and C is (one of) the most popular languages around today mean that they aren't too serious. When you have a knowledge of the hardware and are working at that level day-in day-out then issues like this really don't bother you that much. (I do realise that this is a slightly contrarian view these days, but there is an awful lot of unjustified C-bashing around currently).
- pkhuong 10y ago
  http://cacm.acm.org/magazines/2016/3/198849-a-differential-approach-to-undefined-behavior-detection/fulltext http://cacm.acm.org/magazines/2016/3/198849-a-differential-a... 8575 C or C++ packages in Wheezy. This tool found definite UB bugs in 40% of them. How would you know that your code is sometimes misbehaving because of UB?
  - TickleSteve 10y ago
    not denying there are issues... just saying that C isn't the only widespread language with issues. at least C's problems are widely known.
    - D-Coder 10y ago
      The Potzrebie car isn't the only car with issues. At least the Potzrebie's doors-falling-off problem is widely known.
      - Gibbon1 10y ago
        If the compiler writers union decides that the spec allows for 'format hard drive off on signed overflow' we can always change the spec to 'don't format hard drive on signed overflow'
        pcwalton 10y ago

115 points by jsnell 10y ago

If you think you know C quite well, here is one of the studies the authors ran to elaborate their semantics on corner cases of the language: http://www.cl.cam.ac.uk/~pes20/cerberus/notes50-survey-discussion.html http://www.cl.cam.ac.uk/~pes20/cerberus/notes50-survey-discu... > If you zero all bytes of a struct and then write some of its members, do reads of the padding return zero? (e.g. for a bytewise CAS or hash of the struct, or to know that no security-relevant data has leaked into them.) (and 14 other questions) Webpage of the project: http://www.cl.cam.ac.uk/~pes20/cerberus/ http://www.cl.cam.ac.uk/~pes20/cerberus/

blastrat 10y ago
I knew C quite well. Haven't written any for years. The statements "zero all bytes of a struct" and "reads of the padding" contain enough ambiguity that it answers the question. Not to mention the ambiguity in the words "read" and "write" as they pertain to C, since they already have a "std" meaning that's not the same as lvalue or rvalue, so what exactly do they mean here? And if you think you can answer the question without resolving the ambiguities, that answers some other questions.
- kerneis 10y ago
  I believe the questions in this study (I did not write it, only know the authors) were deliberately open-ended, allowing for comments on the specifics. A previous, much longer version used to contain code examples to comment, but it proved too detailed for people to complete. Moreover, the study was explicitly not about ISO C: "We were not asking what the ISO C standard permits, which is often more restrictive, or about obsolete or obscure hardware or compilers. We focussed on the behaviour of memory and pointers. This is a step towards an unambiguous and mathematically precise definition of the de facto standards: the C dialects that are actually used by systems programmers and implemented by mainstream compilers." Here is an actual example of a comment to this question: I would expect this code to work: struct foo { char a; double b; }; foo p; foo q; memset( &p, 0, sizeof( p ) ); memset( &q, 0, sizeof( q ) ); p.a = 1; q.a = 1; assert( memcmp( &p, &q, sizeof( foo ) ) == 0 );
  - psewell 10y ago
    Indeed. The long version is at [pdf] http://www.cl.cam.ac.uk/~pes20/cerberus/notes30-full.pdf; http://www.cl.cam.ac.uk/~pes20/cerberus/notes30-full.pdf; it has 85 questions supported by concrete code examples and experimental data, e.g. (one of several questions that refine the above): Q64. After an explicit write of zero to a padding byte followed by a write to adjacent members of the structure, does the padding byte hold a well-defined zero value? (not an unspecified value) #include <stdio.h> #include <stddef.h> typedef struct { char c; float f; int i; } st; int main() { // check there is a padding byte between c and f size_t offset_padding = offsetof(st,c)+sizeof(char); if (offsetof(st,f)>offset_padding) { st s; unsigned char *p = ((unsigned char*)(&s)) + offset_padding; *p = 0; s.c = 'A'; s.f = 1.0; s.i = 42; unsigned char c3 = *p; // does c3 hold 0, not an unspecified value? printf("c3=0x%x\n",c3); } return 0; } Some of the questions have clear answers with respect to either the ISO or de facto standards, but many do not - that's the point.
  - Too 10y ago
    Is memset and memcmp compatible with strict aliasing? Intuitively it seems like that would be a gap into the aliasing rules. Altough void* is allowed to alias anything so maybe it works through that. Ive never seen memset, memcpy or memcmp on anything but char* in production code.
    - gpderetta 10y ago
      The aliasing rules only talk about dereferencing pointers. Void* can't be dereferenced so it has no interaction with the aliasing rules. You might be thinking about char, and yes, you are allowed to dereference char to inspect the bytes of an object, which is what the various mem* functions do under the hood.
wahern 10y ago
Here's a similar, real-world case involving GCC 6's more aggressive SRA and alias-analysis optimizations: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=71120 https://gcc.gnu.org/bugzilla/show_bug.cgi?id=71120

paxcoder 10y ago

To write portable code, I wouldn't study de-facto definitions of de-jure undefined behavior, except to see if I could cover every possible one and only if all alternatives were inferior.

pm215 10y ago
Indeed not, but enquiring into what the in-the-wild de-facto beliefs about behaviour are might help in deciding what the de-jure rules should be changed to, or what a compiler implementation ought to do if it cares about what it does on the vast mass of code out there that does commit undefined behaviour, wittingly or otherwise...
- paxcoder 10y ago
  Undefined behavior has a purpose: Not specifying implementation details makes it easier to write new implementations and for a wider variety of platforms. "De facto standards" take away this freedom, so ideally you'd want to reject reliance on UD, but I see your (second) point about that not always being practical. I guess "be conservative in what you do, be liberal in what you accept from others". Just make sure that your foundations are strong (pun) or the whole house will be an EcmaScript.

TickleSteve 10y ago

I've been using C for over 20 years and I'm sure I would be caught out by these... ...but... The fact these curiosities are not an issue in day-to-day work and C is (one of) the most popular languages around today mean that they aren't too serious. When you have a knowledge of the hardware and are working at that level day-in day-out then issues like this really don't bother you that much. (I do realise that this is a slightly contrarian view these days, but there is an awful lot of unjustified C-bashing around currently).

pkhuong 10y ago
http://cacm.acm.org/magazines/2016/3/198849-a-differential-approach-to-undefined-behavior-detection/fulltext http://cacm.acm.org/magazines/2016/3/198849-a-differential-a... 8575 C or C++ packages in Wheezy. This tool found definite UB bugs in 40% of them. How would you know that your code is sometimes misbehaving because of UB?
- TickleSteve 10y ago
  not denying there are issues... just saying that C isn't the only widespread language with issues. at least C's problems are widely known.
  - D-Coder 10y ago
    The Potzrebie car isn't the only car with issues. At least the Potzrebie's doors-falling-off problem is widely known.
    - Gibbon1 10y ago
      If the compiler writers union decides that the spec allows for 'format hard drive off on signed overflow' we can always change the spec to 'don't format hard drive on signed overflow'
      - pcwalton 10y ago