7 ms·

> But I also think a lot of discussions of this topic caricaturize compiler writers to a ridiculous degree. The inflamed backlash should tell you just how dama

> But I also think a lot of discussions of this topic caricaturize compiler writers to a ridiculous degree. The inflamed backlash should tell you just how damaging it is to impose silent failure on meticulously written, previously fine programs.

jcranmer 5y ago
> meticulously written, previously fine programs With relatively few exceptions, if your program hits undefined behavior, then your program was already doing something pretty wrong to begin with. Signed overflow is a poignant example: in how many contexts is INT_MAX + 1 overflowing to INT_MIN actually sane semantics? Unless you're immediately attempting to check the result to see if it overflowed (which is extremely rare in the code I see), this overflow is almost certain to be unexpected, and a program which would have permitted this was not "previously fine" nor "meticulously written." I feel compelled right now to point out that software development is a field where it is routine to tell users that it's their fault for expecting our products to work (that's what the big all-caps block of every software license and EULA says, translated into simple English).
- anarazel 5y ago
  > in how many contexts is INT_MAX + 1 overflowing to INT_MIN actually sane semantics? Unless you're immediately attempting to check the result to see if it overflowed (which is extremely rare in the code I see) It's not that rare - I know that postgres got bit by particularly that issue, and several other other projects as well. Particularly painful because that obviously can cause security issues.
- rocqua 5y ago
  I recall a bug-report discussion that I sadly have never been able to find. It contains a pretty bad side-effect of this. It had code like: int *p; // lots of code if (p != NULL) return 1; // use p Then a later refactor wrongly added a single line before the if statement: int *p; // lots of code int a = *p; if (p != NULL) return 1; // use p This meant the null check was optimized away, since de referencing a null pointer is undefined behavior, so the if-statement can be assumed to be always false. This then lead to actual errors (perhaps even an exploit, I do not recall) arising from the removed null-check. I think in general the "sanity check" cases are the worst. It is hard to determine whether an expression causes undefined behavior if you cannot try and evaluate it. Perhaps a compiler intrinsic that checks (at runtime) whether an expression causes undefined behavior could be useful here. Though I can imagine such an intrinsic being essentially impossible to implement.
  - tom_mellior 5y ago
    This was in the Linux kernel, which is compiled with special kernel flags which make dereferencing null pointers legal. In the context of that code, dereferencing a pointer and later checking it for null was absoutely meaningful. Optimizing the later null check was a compiler bug that was acknowledged and fixed. The compiler here didn't respect the semantics it had promised to kernel code. This was entirely uncontroversial; it was not a case of "unwanted optimization based on undefined behavior", it was a case of "compiler bug breaking well-defined code". Again, all this in a kernel context. In user code GCC will still happily remove the null check because in user code this is an actual bug in the user's code.
  - nwallin 5y ago
    https://lwn.net/Articles/342330/ https://lwn.net/Articles/342330/ https://lwn.net/Articles/342420/ https://lwn.net/Articles/342420/
  - a1369209993 5y ago
    Actually, there's a even worse version: struct foo { ...; bar_t bar[NBAR]; }; struct foo* p = ...; bar_t* q = &p->bar[0]; // add rq, rp, #foo_bar_offs // other declarations if(!p) return NOPE; // optimized out // use p and q Not even any dereferencing, just pointer arithmetic.
    - schlupa 5y ago
      Yes, that one is especially nasty as &p->bar[0] is a constant expression completely solvable at compile time. It's equivalent to the offsetof() macro of stddef.h
    - tom_mellior 5y ago
      Could you post a complete example please and tell us which compiler optimizes out this check? Clang and GCC don't: https://gcc.godbolt.org/z/dbTadEcra https://gcc.godbolt.org/z/dbTadEcra
- lixtra 5y ago
  > With relatively few exceptions, if your program hits undefined behavior, then your program was already doing something pretty wrong to begin with. The author claims: “We have the absurd situation that C, specifically constructed to write the UNIX kernel, cannot be used to write operating systems. In fact, Linux and other operating systems are written in an unstable dialect of C that is produced by using a number of special flags that turn off compiler transformations based on undefined behavior (with no guarantees about future “optimizations”). The Postgres database also needs some of these flags as does the libsodium encryption library and even the machine learning tensor-flow package.” So basically the programers of the most used C programs consider the C standard so broken that they force the compiler to deviate from it. (Or they are not able to do things right?)
  - nytgop77 5y ago
    Somewhat similar to sql situation. (difference being, i am not sure if anybody implements ansi-sql fully) Standard is just common denominator agreed by actual competitors. (Standard takes into account the chipset where incrementing max_int produces a beep instead of min_int). If user wants to use more advantages of the product (dbms/compiler/hardware), one must sacrifice portability and use cnonstandard extensions.
- rectang 5y ago
  > extremely rare in the code I see Well, I learned of the change in compiler behavior some years back because I had written loop code with a sanity check which depended on signed integer overflow wrapping, along with a test case to prove that the sanity check worked, and that test case started failing: not ok 2 - catch overflow in token position calculation Failed test 'catch overflow in token position calculation' at t/152-inversion.t line 70. To the extent I can be, I'm done with C. I leave it to people who think that silently optimizing away previously functional sanity checks is an acceptable engineering tradeoff, and who disparage those of us who have been bitten.
  - a1369209993 5y ago
    > people who think that silently optimizing away previously functional sanity checks is an acceptable engineering tradeoff To be fair, as several people and TFA have pointed out, this isn't a problem with C, but with defective/malicous C compilers. Admittedly, that's not much help if you can't find a compiler that isn't defective/malicous, though, so I can only wish you the best of luck.
    - hnjst 5y ago
      I don't get what you or the comment you're responding to are wishing for. Do you want compilers to stop adding optimizations while staying within the bounds defined by the spec? That they somehow guess that a given piece of code that may trigger UB is too important for them to optimize it based on the assumption that the developer knew what she was doing and ensured that it wouldn't? The case of a compiler update breaking the test sounds like desirable to me. It pinpoints a critical piece of code that was relying on a specific implementation's behavior that is not specified. This could have been triggered by a compiler or architecture change. If this is something that is a hassle to fix immediately, you can temporarily downgrade to the version of the compiler used earlier or disable some optimizations. I fail to see why one would consider a compiler evolving while conforming to language specification defective or malicious (however, with that definition I fear that finding one that isn't may indeed be difficult).
      - 5y ago
- barrkel 5y ago
  Triggering signed overflow and testing whether it has occurred was how I protected the Delphi RTL memory allocation routines from signed overflow attacks, attacks which if not prevented, lead fairly directly to buffer overruns. The code was written in Pascal and assembly, though, so it was safe from a C compiler.
  - pjmlp 5y ago
    I am quite thankfull that I learned systems programing via BASIC, Pascal and Assembly before coming to C. As such my vision of low level coding isn't tainted by the ways of C.
- andreareina 5y ago
  Isn't checking for overflow depending on UB? i.e. in int a; // lots of code int b = a + 1; // check for overflow if (b <= a) abort(); the compiler is allowed to remove the check because in the absence of UB a + 1 > a, therefore the conditional is always false.
  - hnjst 5y ago
    As brilliantly said further in this thread by someone else, maybe check that something is OK before doing it. In that case, something like: [...] if (a > INT_MAX - 1) abort(); int b = a + 1; [...]
    - anarazel 5y ago
      > As brilliantly said further in this thread by someone else, maybe check that something is OK before doing it. Much easier said than done: https://github.com/postgres/postgres/blob/master/src/include/common/int.h#L231 https://github.com/postgres/postgres/blob/master/src/include...
      - Thorrez 5y ago
        Good point that checking for overflow of a multiplication beforehand is hard. But I don't think checking after the fact would be much easier. So thus a C modification to remove the undefined behavior probably wouldn't be very useful still.
        andreareina 5y ago
        The thing is that it's necessary, and (x86, not sure about arm) asm tells you via the d register if not the overflow flag. So why doesn't the C standard arguably provide a misuse-resistant way of doing so? Instead every project having to reinvent it and get it wrong if they do the "obvious" (but wrong) thing.
        Thorrez 5y ago
vinkelhake 5y ago
If the program was previously "fine" on version x.y.z of some compiler, then it is most likely still fine on it. That's the target that the program was written for. There's some disagreement on whether you can call a program "fine" that breaks after switching to a newer version, or a different compiler. I see a lot of programmers out there that unfortunately use the behavior of their code on whatever compiler they're using at the moment as a proxy for what the language actually guarantees.
- IcePic 5y ago
  Well, I can imagine a program having something where new C comments ( // ) makes it not divide, like: a = 5 //* junk here */ 2 ; I'm sure there are ioccc or underhanded C contest entries doing this to make code work differently on compilers based on if // is starting a comment line or not. Sure it is an ugly way of writing stuff and you'd be hard pressed to find lots of real world traps like this, but when/if you did have code that "suddenly" miscompiles you might actually think your old code with an old compiler did work, and a new compiler for "the same" language breaks your program. I don't think everyone code base should need full rewrites ever time a new compiler comes out.

jcranmer 5y ago

> meticulously written, previously fine programs With relatively few exceptions, if your program hits undefined behavior, then your program was already doing something pretty wrong to begin with. Signed overflow is a poignant example: in how many contexts is INT_MAX + 1 overflowing to INT_MIN actually sane semantics? Unless you're immediately attempting to check the result to see if it overflowed (which is extremely rare in the code I see), this overflow is almost certain to be unexpected, and a program which would have permitted this was not "previously fine" nor "meticulously written." I feel compelled right now to point out that software development is a field where it is routine to tell users that it's their fault for expecting our products to work (that's what the big all-caps block of every software license and EULA says, translated into simple English).

anarazel 5y ago
> in how many contexts is INT_MAX + 1 overflowing to INT_MIN actually sane semantics? Unless you're immediately attempting to check the result to see if it overflowed (which is extremely rare in the code I see) It's not that rare - I know that postgres got bit by particularly that issue, and several other other projects as well. Particularly painful because that obviously can cause security issues.
rocqua 5y ago
I recall a bug-report discussion that I sadly have never been able to find. It contains a pretty bad side-effect of this. It had code like: int *p; // lots of code if (p != NULL) return 1; // use p Then a later refactor wrongly added a single line before the if statement: int *p; // lots of code int a = *p; if (p != NULL) return 1; // use p This meant the null check was optimized away, since de referencing a null pointer is undefined behavior, so the if-statement can be assumed to be always false. This then lead to actual errors (perhaps even an exploit, I do not recall) arising from the removed null-check. I think in general the "sanity check" cases are the worst. It is hard to determine whether an expression causes undefined behavior if you cannot try and evaluate it. Perhaps a compiler intrinsic that checks (at runtime) whether an expression causes undefined behavior could be useful here. Though I can imagine such an intrinsic being essentially impossible to implement.
- tom_mellior 5y ago
  This was in the Linux kernel, which is compiled with special kernel flags which make dereferencing null pointers legal. In the context of that code, dereferencing a pointer and later checking it for null was absoutely meaningful. Optimizing the later null check was a compiler bug that was acknowledged and fixed. The compiler here didn't respect the semantics it had promised to kernel code. This was entirely uncontroversial; it was not a case of "unwanted optimization based on undefined behavior", it was a case of "compiler bug breaking well-defined code". Again, all this in a kernel context. In user code GCC will still happily remove the null check because in user code this is an actual bug in the user's code.
- nwallin 5y ago
  https://lwn.net/Articles/342330/ https://lwn.net/Articles/342330/ https://lwn.net/Articles/342420/ https://lwn.net/Articles/342420/
- a1369209993 5y ago
  Actually, there's a even worse version: struct foo { ...; bar_t bar[NBAR]; }; struct foo* p = ...; bar_t* q = &p->bar[0]; // add rq, rp, #foo_bar_offs // other declarations if(!p) return NOPE; // optimized out // use p and q Not even any dereferencing, just pointer arithmetic.
  - schlupa 5y ago
    Yes, that one is especially nasty as &p->bar[0] is a constant expression completely solvable at compile time. It's equivalent to the offsetof() macro of stddef.h
  - tom_mellior 5y ago
    Could you post a complete example please and tell us which compiler optimizes out this check? Clang and GCC don't: https://gcc.godbolt.org/z/dbTadEcra https://gcc.godbolt.org/z/dbTadEcra
lixtra 5y ago
> With relatively few exceptions, if your program hits undefined behavior, then your program was already doing something pretty wrong to begin with. The author claims: “We have the absurd situation that C, specifically constructed to write the UNIX kernel, cannot be used to write operating systems. In fact, Linux and other operating systems are written in an unstable dialect of C that is produced by using a number of special flags that turn off compiler transformations based on undefined behavior (with no guarantees about future “optimizations”). The Postgres database also needs some of these flags as does the libsodium encryption library and even the machine learning tensor-flow package.” So basically the programers of the most used C programs consider the C standard so broken that they force the compiler to deviate from it. (Or they are not able to do things right?)
- nytgop77 5y ago
  Somewhat similar to sql situation. (difference being, i am not sure if anybody implements ansi-sql fully) Standard is just common denominator agreed by actual competitors. (Standard takes into account the chipset where incrementing max_int produces a beep instead of min_int). If user wants to use more advantages of the product (dbms/compiler/hardware), one must sacrifice portability and use cnonstandard extensions.
rectang 5y ago
> extremely rare in the code I see Well, I learned of the change in compiler behavior some years back because I had written loop code with a sanity check which depended on signed integer overflow wrapping, along with a test case to prove that the sanity check worked, and that test case started failing: not ok 2 - catch overflow in token position calculation Failed test 'catch overflow in token position calculation' at t/152-inversion.t line 70. To the extent I can be, I'm done with C. I leave it to people who think that silently optimizing away previously functional sanity checks is an acceptable engineering tradeoff, and who disparage those of us who have been bitten.
- a1369209993 5y ago
  > people who think that silently optimizing away previously functional sanity checks is an acceptable engineering tradeoff To be fair, as several people and TFA have pointed out, this isn't a problem with C, but with defective/malicous C compilers. Admittedly, that's not much help if you can't find a compiler that isn't defective/malicous, though, so I can only wish you the best of luck.
  - hnjst 5y ago
    I don't get what you or the comment you're responding to are wishing for. Do you want compilers to stop adding optimizations while staying within the bounds defined by the spec? That they somehow guess that a given piece of code that may trigger UB is too important for them to optimize it based on the assumption that the developer knew what she was doing and ensured that it wouldn't? The case of a compiler update breaking the test sounds like desirable to me. It pinpoints a critical piece of code that was relying on a specific implementation's behavior that is not specified. This could have been triggered by a compiler or architecture change. If this is something that is a hassle to fix immediately, you can temporarily downgrade to the version of the compiler used earlier or disable some optimizations. I fail to see why one would consider a compiler evolving while conforming to language specification defective or malicious (however, with that definition I fear that finding one that isn't may indeed be difficult).
    - 5y ago
barrkel 5y ago
Triggering signed overflow and testing whether it has occurred was how I protected the Delphi RTL memory allocation routines from signed overflow attacks, attacks which if not prevented, lead fairly directly to buffer overruns. The code was written in Pascal and assembly, though, so it was safe from a C compiler.
- pjmlp 5y ago
  I am quite thankfull that I learned systems programing via BASIC, Pascal and Assembly before coming to C. As such my vision of low level coding isn't tainted by the ways of C.
andreareina 5y ago
Isn't checking for overflow depending on UB? i.e. in int a; // lots of code int b = a + 1; // check for overflow if (b <= a) abort(); the compiler is allowed to remove the check because in the absence of UB a + 1 > a, therefore the conditional is always false.
- hnjst 5y ago
  As brilliantly said further in this thread by someone else, maybe check that something is OK before doing it. In that case, something like: [...] if (a > INT_MAX - 1) abort(); int b = a + 1; [...]
  - anarazel 5y ago
    > As brilliantly said further in this thread by someone else, maybe check that something is OK before doing it. Much easier said than done: https://github.com/postgres/postgres/blob/master/src/include/common/int.h#L231 https://github.com/postgres/postgres/blob/master/src/include...
    - Thorrez 5y ago
      Good point that checking for overflow of a multiplication beforehand is hard. But I don't think checking after the fact would be much easier. So thus a C modification to remove the undefined behavior probably wouldn't be very useful still.
      - andreareina 5y ago
        The thing is that it's necessary, and (x86, not sure about arm) asm tells you via the d register if not the overflow flag. So why doesn't the C standard arguably provide a misuse-resistant way of doing so? Instead every project having to reinvent it and get it wrong if they do the "obvious" (but wrong) thing.
        Thorrez 5y ago

vinkelhake 5y ago

If the program was previously "fine" on version x.y.z of some compiler, then it is most likely still fine on it. That's the target that the program was written for. There's some disagreement on whether you can call a program "fine" that breaks after switching to a newer version, or a different compiler. I see a lot of programmers out there that unfortunately use the behavior of their code on whatever compiler they're using at the moment as a proxy for what the language actually guarantees.

IcePic 5y ago
Well, I can imagine a program having something where new C comments ( // ) makes it not divide, like: a = 5 //* junk here */ 2 ; I'm sure there are ioccc or underhanded C contest entries doing this to make code work differently on compilers based on if // is starting a comment line or not. Sure it is an ugly way of writing stuff and you'd be hard pressed to find lots of real world traps like this, but when/if you did have code that "suddenly" miscompiles you might actually think your old code with an old compiler did work, and a new compiler for "the same" language breaks your program. I don't think everyone code base should need full rewrites ever time a new compiler comes out.