5 ms·

> It’s a bug to assume a certain implementation has particular properties rather than testing. Adding further weight to the article, I think this quote from Ri

> It’s a bug to assume a certain implementation has particular properties rather than testing. Adding further weight to the article, I think this quote from Rich Felker is incredibly language-dependent, and categorically incorrect for languages with undefined behavior, such as C and C++. It's a bug to assume that properties of an implementation can be determined solely through testing. For example, suppose you want to determine what happens when signed integers overflow. You can write a test program that checks the behavior of signed integer overflow in a variety of cases. That only shows that signed integer overflow has the property you want in the cases you've checked for. Even if you were to somehow check every single result of "a + b" for all 2^128 possible combinations of (a,b), that still would only tell you that it works in that particular program, at that particular execution time. It could be that you just haven't hit the particular set of register usage or of allowed optimizations to run into an edge case. There is no possible test that can distinguish between an implementation that guarantees to have a particular property, and an implementation whose undefined behavior happens to match the property in the cases that have been tested. The only way to know is to have some additional piece of information, some additional communication channel to establish intent, rather than inferring intent from the observable behavior. In C, this is typically done with a preprocessor macro.

kevincox 5y ago
A middle ground could be more specific feature macros. For example instead of checking for __MUSL__ and assuming that it supports some function check __MUSL_SOME_FUNCTION or similar. This is basically what your configure script ends up doing (likely poorly) anyways.
skissane 5y ago
Do any C or C++ compilers behave like that in practice – have signed integer overflow which doesn't simply depend on data types and compilation options, but varies more dynamically, depending on the values, runtime execution mode, register allocations, optimisations, etc? In practice, the space of actual undefined behaviours is a lot smaller than the space of legally permitted undefined behaviours – it may be legal per the standard for a compiler in a case of undefined behaviour to make demons fly out of your nose, but I don't think any compiler has ever actually done that, and it would be a mighty strange occurrence if any ever did. Less jocularly, even if some particular type of undefined behaviour is technically permitted by the standard, if all major compilers don't do it, it can be a reasonable engineering decision to assume none ever will, and deal with that unlikely risk if and when it actually occurs.
- tyingq 5y ago
  Here's one example: https://stackoverflow.com/questions/54510094/why-how-does-gcc-compile-the-undefined-behaviour-in-this-signed-overflow-test-so https://stackoverflow.com/questions/54510094/why-how-does-gc... Worked as expected on X86, not as expected on ARM64. One of the answers shows the question author probably used a different GCC version, but it still feels like a good example.
- leni536 5y ago
  bool foo(int x) { return x+1 > x; } This compiles to a function that always returns true on most compilers with optimizations turned on. If the value of `x+1` were merely implementation defined for `x == INT_MAX`, then it would still need to return false then, as it can't be larger than `INT_MAX`. Yet it returns true. https://godbolt.org/z/hn3de1h49 https://godbolt.org/z/hn3de1h49 edit: correction, clang and gcc optimize this to always true, MSVC doesn't.
  - skissane 5y ago
    That’s still “static” in the sense that it is only dependent on compilation options and data types, not the actual value of those data types at runtime. There is nothing there which could not trivially be determined by executing a test program-which is what the comment to which I was originally replying was talking about. Imagine if there was a ‘set_thread_signed_overflow_behavior()’ API which dynamically set signed overflow behaviour at runtime. That would be an example of undefined behaviour which could not be easily detected by executing a test. But, although such an API is permitted by the standard, I’ve never heard of one actually being implemented-well, it exists for floating point (fesetexceptflag FE_OVERFLOW), but (to my knowledge) no mainstream platform has an equivalent for integers. It can be a reasonable engineering decision to ignore those (exceedingly?) rare platforms on which such an API exists for integers, and save thinking about them for the unlikely event you are asked to port to one of them - which will probably never happen, especially not any time soon.
sleepydog 5y ago
Why wouldn't you change the program being built so it doesn't rely on undefined behavior, then?
- hermitdev 5y ago
  Writing C or C++ completely without undefined behavior is a daunting task. Signed integer over/underflow in particular is so easy to get wrong. Doing it manually is a recipe for disaster. In C++, one can leverage "safe" or "checked" integer types that look and behave like regular signed ints, except they ensure no undefined behavior by checking potentially undefined operations before hand and appropriately handling the case where the checks fail by, e.g. throwing an exception or aborting. What makes this more insidious is that you can't just look at a function in isolation, you need to look at how it's invoked to consider cases for UB. Consider: `int add(int x, int y) { return x + y; }` Does this have UB? `add(1, 1)` is fine. But what about `add(1, INT_MAX)` or `add(-1, INT_MIN)`? Yup, these are both UB. Yes, developers should strive to eliminate UB. But, it's a hard problem. Tooling helps, but unfortunately the default settings for every compiler I'm aware of don't help. One really needs to use external tools and sanitizers, but let's be honest, the ergonomics of having to know about and use a separate tool, such as ubsan, isn't great. The tools are wonderful and need wider use, but they'd be of so much more value if they were integrated more tightly in the compiler/linker toolchains and on by default.
denotational 5y ago
I think you are confusing undefined behaviour with implementation-defined behaviour. The compiler is within its rights to optimise your program under the assumption that undefined behaviour is never invoked; this holds irrespective of the particular libc implementation you happen to be using. You cannot assume that the specific libc implementation makes it "safe" to invoke UB in general. As far as I'm aware, the specific example you give (signed integer overflow) has nothing to do with the libc implementation; it doesn't even have anything to do with the underlying architecture, since the compiler is still allowed to optimise under the assumption it never happens, even if it has well defined semantics on an x86. On the other hand it's perfectly valid to rely on implementation-defined behaviour, assuming you can determine the implementation, but signed integer overflow doesn't fall into this category.
- hedora 5y ago
  I'm reasonably sure signed integer overflow is implementation defined to mean twos complement overflow on x86. Of course, assuming two's complement is non-portable, and the compiler is free to add a flag that breaks or enforces my assumption. It's much better to add some casts (that the compiler will compile away on most architectures). However, that doesn't solve the problem of legacy code that assumes it is on an x86.
  - denotational 5y ago
    > I'm reasonably sure signed integer overflow is implementation defined to mean twos complement overflow on x86. I think you're mistaken, or are using implementation-defined in a colloquial sense; the term implementation-defined has a specific technical meaning in the context of C that isn't the same as how it might be used in common parlance. The distinction between implemention-defined behaviour and undefined behaviour is extremely important, because as soon as you invoke the latter your program "doesn't exist" and a standards-conforming compiler is allowed to do whatever it wants. Signed integer overflow is explicitly given as an example of UB by the C standard [1], in 3.4.3: > 3 EXAMPLE An example of undefined behavior is the behavior on integer overflow. By my reading, the specific part of the standard that makes signed overflow UB is in 6.5: > 5 If an exceptional condition occurs during the evaluation of an expression (that is, if the result is not mathematically defined or not in the range of representable values for its type), the behavior is undefined. This is completely unrelated to what actually happens when you issue an ADD instruction on an x86; the C standard doesn't care about this. [1] : http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1256.pdf http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1256.pdf
chasil 5y ago
Why is this insufficient for testing? $ strings /usr/local/musl/lib/libc.a | grep -i musl MUSL_LOCPATH As an aside, I love Rich Felker's POSIX shell tricks. http://www.etalabs.net/sh_tricks.html http://www.etalabs.net/sh_tricks.html
- R0b0t1 5y ago
  Because it may not test the features of the installed system. I may not have a / available for my target.
- andrewmcwatters 5y ago
  Thanks for sharing this link. As soon as you start needing to read POSIX.* docs, you suddenly realize you need answers to a lot of basic questions. Sufficiently advanced shell scripts should be moved to a programming language, but until you hit that threshold, whatever one defines as appropriate, you still need some of these answered.
  - chasil 5y ago
    If you need to coordinate activity between multiple systems in such a way that you are calling remote shells, then the shell is the place to run the show. I wrote an article on xargs parallel scripting some years ago that was popular here, and I can't imagine trying to do that in C. POSIX shells have great flaws. It is not LR-parsed language that can be implemented by a simple yacc grammar, and this leads to ambiguity. However, POSIX (as dash) is fast, portable, and small enough for embedded systems. Few of dash's peers are its equal if all of these are required.
    - andrewmcwatters 5y ago
      Totally. This is lost on a lot of developers. Even having a rudimentary knowledge of Shell Command Language saves a lot of otherwise wasted time.
- MereInterest 5y ago
  The general argument isn't so much about "testing" vs "macros", because #ifdef LIBRARY_VERSION is also a test of a macro's value, but what type of thing should be tested. Whether to test the functionality or the labels of it, whether to test the things that the computer runs or something that is part of a comment, whether you test the signifier or the signified. In this context, testing functionality would mean compiling a test program and observing the results. A macro or the existence of MUSL_LOCPATH would both be testing the labels and not the actual code.

kevincox 5y ago

A middle ground could be more specific feature macros. For example instead of checking for __MUSL__ and assuming that it supports some function check __MUSL_SOME_FUNCTION or similar. This is basically what your configure script ends up doing (likely poorly) anyways.

skissane 5y ago

Do any C or C++ compilers behave like that in practice – have signed integer overflow which doesn't simply depend on data types and compilation options, but varies more dynamically, depending on the values, runtime execution mode, register allocations, optimisations, etc? In practice, the space of actual undefined behaviours is a lot smaller than the space of legally permitted undefined behaviours – it may be legal per the standard for a compiler in a case of undefined behaviour to make demons fly out of your nose, but I don't think any compiler has ever actually done that, and it would be a mighty strange occurrence if any ever did. Less jocularly, even if some particular type of undefined behaviour is technically permitted by the standard, if all major compilers don't do it, it can be a reasonable engineering decision to assume none ever will, and deal with that unlikely risk if and when it actually occurs.

tyingq 5y ago
Here's one example: https://stackoverflow.com/questions/54510094/why-how-does-gcc-compile-the-undefined-behaviour-in-this-signed-overflow-test-so https://stackoverflow.com/questions/54510094/why-how-does-gc... Worked as expected on X86, not as expected on ARM64. One of the answers shows the question author probably used a different GCC version, but it still feels like a good example.
leni536 5y ago
bool foo(int x) { return x+1 > x; } This compiles to a function that always returns true on most compilers with optimizations turned on. If the value of `x+1` were merely implementation defined for `x == INT_MAX`, then it would still need to return false then, as it can't be larger than `INT_MAX`. Yet it returns true. https://godbolt.org/z/hn3de1h49 https://godbolt.org/z/hn3de1h49 edit: correction, clang and gcc optimize this to always true, MSVC doesn't.
- skissane 5y ago
  That’s still “static” in the sense that it is only dependent on compilation options and data types, not the actual value of those data types at runtime. There is nothing there which could not trivially be determined by executing a test program-which is what the comment to which I was originally replying was talking about. Imagine if there was a ‘set_thread_signed_overflow_behavior()’ API which dynamically set signed overflow behaviour at runtime. That would be an example of undefined behaviour which could not be easily detected by executing a test. But, although such an API is permitted by the standard, I’ve never heard of one actually being implemented-well, it exists for floating point (fesetexceptflag FE_OVERFLOW), but (to my knowledge) no mainstream platform has an equivalent for integers. It can be a reasonable engineering decision to ignore those (exceedingly?) rare platforms on which such an API exists for integers, and save thinking about them for the unlikely event you are asked to port to one of them - which will probably never happen, especially not any time soon.

sleepydog 5y ago

Why wouldn't you change the program being built so it doesn't rely on undefined behavior, then?

hermitdev 5y ago
Writing C or C++ completely without undefined behavior is a daunting task. Signed integer over/underflow in particular is so easy to get wrong. Doing it manually is a recipe for disaster. In C++, one can leverage "safe" or "checked" integer types that look and behave like regular signed ints, except they ensure no undefined behavior by checking potentially undefined operations before hand and appropriately handling the case where the checks fail by, e.g. throwing an exception or aborting. What makes this more insidious is that you can't just look at a function in isolation, you need to look at how it's invoked to consider cases for UB. Consider: `int add(int x, int y) { return x + y; }` Does this have UB? `add(1, 1)` is fine. But what about `add(1, INT_MAX)` or `add(-1, INT_MIN)`? Yup, these are both UB. Yes, developers should strive to eliminate UB. But, it's a hard problem. Tooling helps, but unfortunately the default settings for every compiler I'm aware of don't help. One really needs to use external tools and sanitizers, but let's be honest, the ergonomics of having to know about and use a separate tool, such as ubsan, isn't great. The tools are wonderful and need wider use, but they'd be of so much more value if they were integrated more tightly in the compiler/linker toolchains and on by default.

denotational 5y ago

I think you are confusing undefined behaviour with implementation-defined behaviour. The compiler is within its rights to optimise your program under the assumption that undefined behaviour is never invoked; this holds irrespective of the particular libc implementation you happen to be using. You cannot assume that the specific libc implementation makes it "safe" to invoke UB in general. As far as I'm aware, the specific example you give (signed integer overflow) has nothing to do with the libc implementation; it doesn't even have anything to do with the underlying architecture, since the compiler is still allowed to optimise under the assumption it never happens, even if it has well defined semantics on an x86. On the other hand it's perfectly valid to rely on implementation-defined behaviour, assuming you can determine the implementation, but signed integer overflow doesn't fall into this category.

hedora 5y ago
I'm reasonably sure signed integer overflow is implementation defined to mean twos complement overflow on x86. Of course, assuming two's complement is non-portable, and the compiler is free to add a flag that breaks or enforces my assumption. It's much better to add some casts (that the compiler will compile away on most architectures). However, that doesn't solve the problem of legacy code that assumes it is on an x86.
- denotational 5y ago
  > I'm reasonably sure signed integer overflow is implementation defined to mean twos complement overflow on x86. I think you're mistaken, or are using implementation-defined in a colloquial sense; the term implementation-defined has a specific technical meaning in the context of C that isn't the same as how it might be used in common parlance. The distinction between implemention-defined behaviour and undefined behaviour is extremely important, because as soon as you invoke the latter your program "doesn't exist" and a standards-conforming compiler is allowed to do whatever it wants. Signed integer overflow is explicitly given as an example of UB by the C standard [1], in 3.4.3: > 3 EXAMPLE An example of undefined behavior is the behavior on integer overflow. By my reading, the specific part of the standard that makes signed overflow UB is in 6.5: > 5 If an exceptional condition occurs during the evaluation of an expression (that is, if the result is not mathematically defined or not in the range of representable values for its type), the behavior is undefined. This is completely unrelated to what actually happens when you issue an ADD instruction on an x86; the C standard doesn't care about this. [1] : http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1256.pdf http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1256.pdf

chasil 5y ago

Why is this insufficient for testing? $ strings /usr/local/musl/lib/libc.a | grep -i musl MUSL_LOCPATH As an aside, I love Rich Felker's POSIX shell tricks. http://www.etalabs.net/sh_tricks.html http://www.etalabs.net/sh_tricks.html

R0b0t1 5y ago
Because it may not test the features of the installed system. I may not have a / available for my target.
andrewmcwatters 5y ago
Thanks for sharing this link. As soon as you start needing to read POSIX.* docs, you suddenly realize you need answers to a lot of basic questions. Sufficiently advanced shell scripts should be moved to a programming language, but until you hit that threshold, whatever one defines as appropriate, you still need some of these answered.
- chasil 5y ago
  If you need to coordinate activity between multiple systems in such a way that you are calling remote shells, then the shell is the place to run the show. I wrote an article on xargs parallel scripting some years ago that was popular here, and I can't imagine trying to do that in C. POSIX shells have great flaws. It is not LR-parsed language that can be implemented by a simple yacc grammar, and this leads to ambiguity. However, POSIX (as dash) is fast, portable, and small enough for embedded systems. Few of dash's peers are its equal if all of these are required.
  - andrewmcwatters 5y ago
    Totally. This is lost on a lot of developers. Even having a rudimentary knowledge of Shell Command Language saves a lot of otherwise wasted time.
MereInterest 5y ago
The general argument isn't so much about "testing" vs "macros", because #ifdef LIBRARY_VERSION is also a test of a macro's value, but what type of thing should be tested. Whether to test the functionality or the labels of it, whether to test the things that the computer runs or something that is part of a comment, whether you test the signifier or the signified. In this context, testing functionality would mean compiling a test program and observing the results. A macro or the existence of MUSL_LOCPATH would both be testing the labels and not the actual code.