10 ms·

I think we are always looking at ways to "clean up C" but that this has to be done very carefully not to break existing code. For example, the committee recentl

I think we are always looking at ways to "clean up C" but that this has to be done very carefully not to break existing code. For example, the committee recently voted to remove support for function definitions with identifier lists from C2x http://www.open-std.org/jtc1/sc22/wg14/www/docs/n2432.pdf http://www.open-std.org/jtc1/sc22/wg14/www/docs/n2432.pdf At least one vendor was not very happy with this decision. Undefined behaviors tend to be undefined for a reason and shouldn't be thought of as defects in the standard. In my years on the committee, I have always argued to define as much behavior as possible and to as narrowly define undefined behaviors as possible. We also had a recent discussion about adding additional name spaces (when discussing reserved identifiers), but it didn't gain much traction.

bumblebritches5 7y ago
Looks like that proposal is dropping support for K&R function declarations, is that right?
- rseacord 7y ago
  yes, that is correct.
3JPLW 7y ago
Does it concern you how aggressively compiler teams are exploiting UB?
- Spivak 7y ago
  You do have to understand that compiler teams aren't saying something like "this triggers UB, quick just replace it with noop." It's just something that naturally happens when you need to reason about code. For example, consider a very simple statement. let array[10]; let i = some_function(); print(array[i]); The function might not even be known to the compiler at compilation time if it was from a DLL or something. But the compiler is like "hey! you used the result of this function as an index for this array! i must be in the range [0, 10)! I can use that information!"
  - msebor 7y ago
    This is a good example. Let me flesh it out a bit more to illustrate a specific instance of this problem: int a[2][2]; int f (int i, int j) { int t = a[1][j]; a[0][i] = 0; // cannot change a[1] return a[1][j] - t; // can be folded to zero } The language says that elements of the matrix a must only be accessed by indices that are valid for each bound, so compilers can and some do optimize code based on that requirement (see https://godbolt.org/z/spSF8e https://godbolt.org/z/spSF8e). But when a program breaks that requirement (say, by calling f(2, 0)) the function will likely return an unexpected value.
    - Spivak 7y ago
      But I don't know what you want to happen in this case? If you actually call f(2,0) then the program makes no sense. How can you have an expected value for a function call that violates its preconditions?
      - userbinator 7y ago
        Based on the memory layout of arrays, which AFAIK is defined rather strictly by the standard, a[0][2] will be the same as a[1][0].
  - gwd 7y ago
    > But the compiler is like "hey! you used the result of this function as an index for this array! i must be in the range [0, 10)! I can use that information!" As a developer who has seen lots of developers (including himself) make really dumb mistakes, this seems like a very strange statement. Imagine if you hired a security guard to stand outside your house. One day, he sees you leave the house and forget to lock the door. So he reasons, "Oh, nothing important inside the house today -- guess I can take the day off", and walks off. That's what a lot of these "I can infer X must be true" reasonings sounds like to me: they assume that developers don't make mistakes; and that all unwanted behavior is exactly the same. So suppose we have code that does this: int array[10]; int i = some_function(); /* Lots of stuff */ if ( i > 10 ) { return -EINVAL; } array[i] = newval; And then someone decides to add some optional debug logging, and forgets that `i` hasn't been sanitized yet: int array[10]; int i = some_function(); logf("old value: %d\n", array[i]); /* Lots of stuff */ if ( i > 10 ) { return -EINVAL; } array[i] = newval; Now reading `array[i]` if `i` > 10 is certainly UB; but in a lot of cases, it will be harmless; and in the worst case it will crash with a segfault. But suppose a clever compiler says, "We've accessed array[i], so I can infer that i < 10, and get rid of the check entirely!" Now we've changed an out-of-bounds read into an out-of-bounds write, which has changed worst-case a DoS into a privilege escalation! I don't know whether anything like this has ever happened, but 1) it's certainly the kind of thing allowed by the spec, 2) it makes C a much more dangerous language to deal with.
- rseacord 7y ago
  I would say that there is a lot of concern in the committee about how compilers are optimizing based on pointer providence. There has been a study group looking at this. It now appears that they are likely to publish their proposal as a Technical Report.
  - revertts 7y ago
    What's the best way to keep an eye out for that TR? Periodically checking http://www.open-std.org/jtc1/sc22/wg14/ http://www.open-std.org/jtc1/sc22/wg14/ ? I can't ever tell if I'm looking in the right place. :)
    - AaronBallman 7y ago
      If you're interested in the final TR, I would imagine we'd list it on that page you linked. If you're interested in following the drafts before it becomes published, you'd fine them on http://www.open-std.org/jtc1/sc22/wg14/www/wg14_document_log.htm http://www.open-std.org/jtc1/sc22/wg14/www/wg14_document_log... (A draft has yet to be posted, though, so you won't find one there yet.)
  - _kst_ 7y ago
    "based on pointer providence" I think you meant "provenance" (mentioning it for the sake of anyone who wants to search for it).
    - rseacord 7y ago
      Yes, my mistake--I was thinking of Rhode Island. I wrote a short bit about this at https://www.nccgroup.trust/us/about-us/newsroom-and-events/blog/2019/june/pointer-provenance/ https://www.nccgroup.trust/us/about-us/newsroom-and-events/b... if anyone is interested.
      - flatfinger 7y ago
        What makes pointer provenance really great is that clang and gcc will treat that pointers that are observed to have the same address as freely interchangeable, even if their provenance is different. Clang sometimes even goes so far with that concept that even uintptr_t comparisons won't help. extern int x[],y[]; int test(int i) { y[0] = 1; if ((uintptr_t)(x+5) == (uintptr_t)(y+i)) y[i] = 2; return y[0]; } If this function is invoked with i==0, it should be possible for y[0] and the return value to both be 1, or both be 2. If x has five elements, however, and y immediately follows it, clang's generated code will set y[0] to 2 and yet return 1. Cool, eh?
- msebor 7y ago
  This is a common misconception (or poor way of phrasing it, sorry). Compiler implementers don't go looking for instances of undefined behavior in a program with the goal of optimizing it in some way. There is little value in optimizing invalid code. The opposite is the case. But we must write code that relies on the same rules and requirements that programs are held to (and vice versa). When either party breaks those rules, either accidentally or deliberately, bad things happen. What sometimes happens is that code written years or decades ago relies on the absence of an explicit guarantee in the language suddenly stops working because a compiler change depends on the assumption that code doesn't rely on the absence of the guarantee. That can happen as a result of improving optimizations, which is often but not not necessarily always motivated by improving the efficiency of programs. Better analysis can also help find bugs in code or avoid issuing warnings for safe code.
  - ori_b 7y ago
    There are rules and requirements documented in the spec, and there are de-facto rules and requirements that programs expect. Not only that, but when they do exploit these rules, often the code generated is obviously incorrect, and could have been flagged at compile time. Right now, it seems like compiler vendors are playing a game of chicken with their users.
    - saagarjha 7y ago
      I think the issue is that many of these "obviously incorrect" things are not obvious at the level that the optimizations are taking place. Perhaps it would be worth considering adding higher-level passes in the compiler that can detect these kinds of surprising changes and warn about them.
      - a1369209993 7y ago
        Well, no, the issue is that the compiler writers refuse to acknowledge the these obviously incorrect things are incorrect in the first place and tend to blame users for tripping over compiler bugs. If it were just that they didn't know how to fix said bugs, that would be a qualitatively different and much less severe problem.
ximeng 7y ago
Why would a vendor be unhappy about that? They have a large library using this deprecated syntax? Or many customers? It seems like a relatively easy fix to existing code.
- AaronBallman 7y ago
  The usual argument is: once you've verified some piece of code is correct, changing it (even when there should be no functional change in the semantics) carries risk. Some customers have C89-era code that compiles in C17 mode and they don't want to change that code because of these risks (perhaps the cost of testing is prohibitively expensive, there may be contractual obligations that kick in when changing that code, etc).
  - rmind 7y ago
    Well, one argument is that the vendors should not compile C89 code as C17. If you write C89, then stick with -std=c89 (or upgrade to the latest officially compatible revision). It makes sense to preserve language compatibility within several language revisions, gradually sunsetting some features, but why do that for the eternity? Gradual de-supporting would push the problem to the compilers, but while it is no fun supporting, let's say, C89 and a hypothetical incompatible language C3X, this is where the effort should go (after all, companies with the old codebases can stick with older compilers). There is a great value in paving a way for a more fundamental C language simplifications and clean ups.
    - apotheon 7y ago
      These are all good points, and I don't see a legitimate, technical reason to avoid deprecating and eliminating identifier list syntax in new C standards (but then, I'm not as much of an expert as some people, so I might be missing something important). That having been said, a compiler vendor has, almost by definition as its first priority, an undeniable interest in keeping customers happy while, at the same time, ensuring strong reasons to see value in a version upgrade. When dealing with corporate enterprise customers, that often means offering new features without deprecating old features, because the customers want the new features but don't want to have to rewrite anything just because of a compiler upgrade. They'll want C17 (and C32, for that matter) hot new features, but they will not want to pay a developer to "rewrite code that already works" (in the view of middle managers). That's why I think they'd most likely complain. Their concerns about removing identifier lists likely have nothing at all to do with good technical sense. Ideally, if you don't want to rewrite your rickety old bit-rotting shit code, you should just continue compiling it with an old compiler, and if you want new language features you should use them in new language standard code, period, but business (for pathological, perhaps, but not really upstream-curable reasons) doesn't generally work that way.
tropo 7y ago
C has strayed very far from the original intent because compiler authors prioritized benchmark results at the expense of real-world use cases. This bad trend needs to be reversed. Consider signed integer overflow. The intent wasn't that the compiler could generate nonsense code if the programmer overflowed an integer. The intent was the the programmer could determine what would happen by reading the hardware manual. You'd wrap around if the hardware naturally would do so. On some other hardware you might get saturation or an exception. In other words, all modern computers should wrap. That includes x86, ARM, Power, Alpha, Itanium, SPARC, and just about everything else. I don't believe you can even buy non-wrapping hardware with a C99 or newer compiler. Since this is likely to remain true, there is no longer any justification for retaining undefined behavior that is getting abused to the detriment of C users.
- favorited 7y ago
  If most C developers wanted to trade the performance they get from the compiler being able to assume `n+1 > n` for signed integer n, it would happen.
  - flatfinger 7y ago
    Most of the useful optimizations that could be facilitated by treating integer overflow as jump the rails optimization could be facilitated just as well by allowing implementations to behave as though integers may sometimes, non-deterministically, be capable of holding values outside their range. If integer computations are guaranteed never to have side effects beyond yielding "weird" values, programs that exploit that guarantee may be processed to more efficient machine code than those which must avoid integer overflow at all costs.
    - saagarjha 7y ago
      How is this better behavior?
      - whelming_wave 7y ago

bumblebritches5 7y ago

Looks like that proposal is dropping support for K&R function declarations, is that right?

rseacord 7y ago
yes, that is correct.

3JPLW 7y ago

Does it concern you how aggressively compiler teams are exploiting UB?

Spivak 7y ago
You do have to understand that compiler teams aren't saying something like "this triggers UB, quick just replace it with noop." It's just something that naturally happens when you need to reason about code. For example, consider a very simple statement. let array[10]; let i = some_function(); print(array[i]); The function might not even be known to the compiler at compilation time if it was from a DLL or something. But the compiler is like "hey! you used the result of this function as an index for this array! i must be in the range [0, 10)! I can use that information!"
- msebor 7y ago
  This is a good example. Let me flesh it out a bit more to illustrate a specific instance of this problem: int a[2][2]; int f (int i, int j) { int t = a[1][j]; a[0][i] = 0; // cannot change a[1] return a[1][j] - t; // can be folded to zero } The language says that elements of the matrix a must only be accessed by indices that are valid for each bound, so compilers can and some do optimize code based on that requirement (see https://godbolt.org/z/spSF8e https://godbolt.org/z/spSF8e). But when a program breaks that requirement (say, by calling f(2, 0)) the function will likely return an unexpected value.
  - Spivak 7y ago
    But I don't know what you want to happen in this case? If you actually call f(2,0) then the program makes no sense. How can you have an expected value for a function call that violates its preconditions?
    - userbinator 7y ago
      Based on the memory layout of arrays, which AFAIK is defined rather strictly by the standard, a[0][2] will be the same as a[1][0].
- gwd 7y ago
  > But the compiler is like "hey! you used the result of this function as an index for this array! i must be in the range [0, 10)! I can use that information!" As a developer who has seen lots of developers (including himself) make really dumb mistakes, this seems like a very strange statement. Imagine if you hired a security guard to stand outside your house. One day, he sees you leave the house and forget to lock the door. So he reasons, "Oh, nothing important inside the house today -- guess I can take the day off", and walks off. That's what a lot of these "I can infer X must be true" reasonings sounds like to me: they assume that developers don't make mistakes; and that all unwanted behavior is exactly the same. So suppose we have code that does this: int array[10]; int i = some_function(); /* Lots of stuff */ if ( i > 10 ) { return -EINVAL; } array[i] = newval; And then someone decides to add some optional debug logging, and forgets that `i` hasn't been sanitized yet: int array[10]; int i = some_function(); logf("old value: %d\n", array[i]); /* Lots of stuff */ if ( i > 10 ) { return -EINVAL; } array[i] = newval; Now reading `array[i]` if `i` > 10 is certainly UB; but in a lot of cases, it will be harmless; and in the worst case it will crash with a segfault. But suppose a clever compiler says, "We've accessed array[i], so I can infer that i < 10, and get rid of the check entirely!" Now we've changed an out-of-bounds read into an out-of-bounds write, which has changed worst-case a DoS into a privilege escalation! I don't know whether anything like this has ever happened, but 1) it's certainly the kind of thing allowed by the spec, 2) it makes C a much more dangerous language to deal with.
rseacord 7y ago
I would say that there is a lot of concern in the committee about how compilers are optimizing based on pointer providence. There has been a study group looking at this. It now appears that they are likely to publish their proposal as a Technical Report.
- revertts 7y ago
  What's the best way to keep an eye out for that TR? Periodically checking http://www.open-std.org/jtc1/sc22/wg14/ http://www.open-std.org/jtc1/sc22/wg14/ ? I can't ever tell if I'm looking in the right place. :)
  - AaronBallman 7y ago
    If you're interested in the final TR, I would imagine we'd list it on that page you linked. If you're interested in following the drafts before it becomes published, you'd fine them on http://www.open-std.org/jtc1/sc22/wg14/www/wg14_document_log.htm http://www.open-std.org/jtc1/sc22/wg14/www/wg14_document_log... (A draft has yet to be posted, though, so you won't find one there yet.)
- _kst_ 7y ago
  "based on pointer providence" I think you meant "provenance" (mentioning it for the sake of anyone who wants to search for it).
  - rseacord 7y ago
    Yes, my mistake--I was thinking of Rhode Island. I wrote a short bit about this at https://www.nccgroup.trust/us/about-us/newsroom-and-events/blog/2019/june/pointer-provenance/ https://www.nccgroup.trust/us/about-us/newsroom-and-events/b... if anyone is interested.
    - flatfinger 7y ago
      What makes pointer provenance really great is that clang and gcc will treat that pointers that are observed to have the same address as freely interchangeable, even if their provenance is different. Clang sometimes even goes so far with that concept that even uintptr_t comparisons won't help. extern int x[],y[]; int test(int i) { y[0] = 1; if ((uintptr_t)(x+5) == (uintptr_t)(y+i)) y[i] = 2; return y[0]; } If this function is invoked with i==0, it should be possible for y[0] and the return value to both be 1, or both be 2. If x has five elements, however, and y immediately follows it, clang's generated code will set y[0] to 2 and yet return 1. Cool, eh?
msebor 7y ago
This is a common misconception (or poor way of phrasing it, sorry). Compiler implementers don't go looking for instances of undefined behavior in a program with the goal of optimizing it in some way. There is little value in optimizing invalid code. The opposite is the case. But we must write code that relies on the same rules and requirements that programs are held to (and vice versa). When either party breaks those rules, either accidentally or deliberately, bad things happen. What sometimes happens is that code written years or decades ago relies on the absence of an explicit guarantee in the language suddenly stops working because a compiler change depends on the assumption that code doesn't rely on the absence of the guarantee. That can happen as a result of improving optimizations, which is often but not not necessarily always motivated by improving the efficiency of programs. Better analysis can also help find bugs in code or avoid issuing warnings for safe code.
- ori_b 7y ago
  There are rules and requirements documented in the spec, and there are de-facto rules and requirements that programs expect. Not only that, but when they do exploit these rules, often the code generated is obviously incorrect, and could have been flagged at compile time. Right now, it seems like compiler vendors are playing a game of chicken with their users.
  - saagarjha 7y ago
    I think the issue is that many of these "obviously incorrect" things are not obvious at the level that the optimizations are taking place. Perhaps it would be worth considering adding higher-level passes in the compiler that can detect these kinds of surprising changes and warn about them.
    - a1369209993 7y ago
      Well, no, the issue is that the compiler writers refuse to acknowledge the these obviously incorrect things are incorrect in the first place and tend to blame users for tripping over compiler bugs. If it were just that they didn't know how to fix said bugs, that would be a qualitatively different and much less severe problem.

ximeng 7y ago

Why would a vendor be unhappy about that? They have a large library using this deprecated syntax? Or many customers? It seems like a relatively easy fix to existing code.

AaronBallman 7y ago
The usual argument is: once you've verified some piece of code is correct, changing it (even when there should be no functional change in the semantics) carries risk. Some customers have C89-era code that compiles in C17 mode and they don't want to change that code because of these risks (perhaps the cost of testing is prohibitively expensive, there may be contractual obligations that kick in when changing that code, etc).
- rmind 7y ago
  Well, one argument is that the vendors should not compile C89 code as C17. If you write C89, then stick with -std=c89 (or upgrade to the latest officially compatible revision). It makes sense to preserve language compatibility within several language revisions, gradually sunsetting some features, but why do that for the eternity? Gradual de-supporting would push the problem to the compilers, but while it is no fun supporting, let's say, C89 and a hypothetical incompatible language C3X, this is where the effort should go (after all, companies with the old codebases can stick with older compilers). There is a great value in paving a way for a more fundamental C language simplifications and clean ups.
  - apotheon 7y ago
    These are all good points, and I don't see a legitimate, technical reason to avoid deprecating and eliminating identifier list syntax in new C standards (but then, I'm not as much of an expert as some people, so I might be missing something important). That having been said, a compiler vendor has, almost by definition as its first priority, an undeniable interest in keeping customers happy while, at the same time, ensuring strong reasons to see value in a version upgrade. When dealing with corporate enterprise customers, that often means offering new features without deprecating old features, because the customers want the new features but don't want to have to rewrite anything just because of a compiler upgrade. They'll want C17 (and C32, for that matter) hot new features, but they will not want to pay a developer to "rewrite code that already works" (in the view of middle managers). That's why I think they'd most likely complain. Their concerns about removing identifier lists likely have nothing at all to do with good technical sense. Ideally, if you don't want to rewrite your rickety old bit-rotting shit code, you should just continue compiling it with an old compiler, and if you want new language features you should use them in new language standard code, period, but business (for pathological, perhaps, but not really upstream-curable reasons) doesn't generally work that way.

tropo 7y ago

C has strayed very far from the original intent because compiler authors prioritized benchmark results at the expense of real-world use cases. This bad trend needs to be reversed. Consider signed integer overflow. The intent wasn't that the compiler could generate nonsense code if the programmer overflowed an integer. The intent was the the programmer could determine what would happen by reading the hardware manual. You'd wrap around if the hardware naturally would do so. On some other hardware you might get saturation or an exception. In other words, all modern computers should wrap. That includes x86, ARM, Power, Alpha, Itanium, SPARC, and just about everything else. I don't believe you can even buy non-wrapping hardware with a C99 or newer compiler. Since this is likely to remain true, there is no longer any justification for retaining undefined behavior that is getting abused to the detriment of C users.

favorited 7y ago
If most C developers wanted to trade the performance they get from the compiler being able to assume `n+1 > n` for signed integer n, it would happen.
- flatfinger 7y ago
  Most of the useful optimizations that could be facilitated by treating integer overflow as jump the rails optimization could be facilitated just as well by allowing implementations to behave as though integers may sometimes, non-deterministically, be capable of holding values outside their range. If integer computations are guaranteed never to have side effects beyond yielding "weird" values, programs that exploit that guarantee may be processed to more efficient machine code than those which must avoid integer overflow at all costs.
  - saagarjha 7y ago
    How is this better behavior?
    - whelming_wave 7y ago