9 ms·

Type Punning Functions in C

164 points by heliostatic 10y ago

kbenson 10y ago
Sir, while the gentlemen and ladies of this fine organization are infinitely pleased with your research into the black magic of the nether regions and the treatise you have presented from aforesaid research, we would ask you please be even more strident in your warnings about practicing this subject. It seems a number of neophytes have disappeared under odd circumstances in the weeks since you presented your work, and the interruption to the experiments they were assisting with has become quite burdensome. Additionally, we expect training and recruitment costs to be quite a bit higher than normal next year, and as you well know our coffers are not unlimited.
- catnaroek 10y ago
  > black magic of the nether regions Or plain abstraction violation. “Black magic” gives it an undeserved good reputation: “I must aspire to be clever enough to understand and actively make use of this.”
  - kbenson 10y ago
    Maybe. Given it relies on the calling convention of the OS you're on, the CPU architecture you're on, and knowledge of assembly, it could very well be both.
    - int_19h 10y ago
      It also relies on implementation-specific behavior in the compiler, since the latter is not actually required to follow established the convention for argument passing, so long as it can prove that the only place the function is called from is the same translation unit. In fact, so far as I can tell by looking at the first example, the compiler can circumvent this whole thing by noting that: 1. DoubleToTheInt is only ever called in the same unit, so it can be inlined - no need to generate the function. 2. Pointer to DoubleToTheInt is created, but is then immediately cast to an incompatible function type, which is undefined behavior - and so it can be substituted with any random value, say, whatever happens to be in EAX at that moment (any U.B. is substitutable for any other U.B.!), and the compiler can still avoid generating the actual function.
  - chriswarbo 10y ago
    > “Black magic” gives it an undeserved good reputation From http://catb.org/jargon/html/B/black-magic.html http://catb.org/jargon/html/B/black-magic.html > black magic: n. > [common] A technique that works, though nobody really understands why. More obscure than voodoo programming, which may be done by cookbook. Compare also black art, deep magic, and magic number (sense 2).
- ChuckMcM 10y ago
  Nicely done. The first time I came across the information of punning vs code was during the SPARC ABI standards work where the test code was working but the test case was supposed to fail. It happened to be a vendor compiler that optimized certain four parameter function calls in a similar way. I was rather perplexed how a compiler could "work" and "not work" at the same time.
- david-given 10y ago
  ...I once wrote a program to summon demons. It didn't work, mind. But I was really proud of my doubly-linked-list pentacle, each node of which contained a pointer to the summoning point in the centre. I thought that was really elegant. Incidentally, if anyone is thinking of trying this for yourself: always remember to mlock() the memory containing your summoning. The last thing you want is some demon of the pit getting swapped out to disk. Not only is swap notoriously insecure, but sulphur buildup on the disk platter will cause a disk crash. And for heaven's sake, make sure your pentacle is properly aligned. If it's split across two pages, they may end up not being next to each other in physical ram, and then it won't work! Ideally, use an embedded device where you know what every cycle does. Don't be put off by the low power. You'd be surprised how much evil you can do with an Arduino.
  - jschwartzi 10y ago
    I've found that my demon summoning programs work much better if I sacrifice the ram beforehand.
tathougies 10y ago
This is undefined behavior, so you're claim that this is 'in C' is blatantly false. In general, you cannot assume the underlying architecture will pass its arguments in any particular location.
- catnaroek 10y ago
  The problem is that determining whether an ostensibly C (syntactically valid and typable) program is actually written “in C” is undecidable.
  - adrianN 10y ago
    Almost any interesting property about programs is undecidable. Yet somehow programmers still write nontrivial programs. Undefined behavior in C sucks, but being its presence or absence being undecidable is not the main reason.
    - catnaroek 10y ago
      > Almost any interesting property about programs is undecidable. It's only undecidable if the language makes it undecidable. For example, subject reduction is decidable in a type-safe language (EDIT: with decidable type checking, oops!), because the very definition of type safety means that typability implies subject reduction. Of course, not everything of interest can be made decidable in a general-purpose language, but the integrity of a language's basic abstractions should never be in question.
    - haldean 10y ago
      The formalization of this idea is Rice's Theorem: https://en.wikipedia.org/wiki/Rice%27s_theorem https://en.wikipedia.org/wiki/Rice%27s_theorem
      - catnaroek 10y ago
        Rice's theorem is a statement about partial functions, not programs. There exist decidable properties of programs: Is it of size less than N (for any suitable measure)? Does it finish in less than N steps (for a given fixed input)? Two syntactically distinct programs may compute the same partial function.
bitwize 10y ago
I freaking hate you, OP. I really freaking hate you right now. This is an abomination in the sight of God.
pklausler 10y ago
If your C compiler doesn't complain bitterly to you about an incompatible pointer type on that assignment, get a better C compiler. If your C compiler does complain bitterly about it, heed those warnings or expect no sympathy. If you write this in production code, and don't get sent home that day in tears, your company's code review process has failed.
- akkartik 10y ago
  No warnings (-W -Wall) with gcc 4.2.1 on OSX :/ gcc 4.8.4 on Ubuntu 14.04 Trusty still gives no warnings, but gives a different result: $ gcc -W -Wall x.c -lm $ a.out (0.99)^100: 0.366032 (0.99)^100: inf clang 3.4 on Ubuntu 14.04: $ clang -W -Wall -fsanitize=undefined x.c -lm $ a.out (0.99)^100: 0.366032 (0.99)^100: inf :(
  - codemac 10y ago
    ; gcc -v |[2] grep version gcc version 6.1.1 20160802 (GCC) ; gcc -Wall -Werror ./something.c -lm ; ./a.out (0.99)^100: 0.366032 (0.99)^100: 0.000000 ;
  - pavanky 10y ago
    I get 0.366032 on gcc-5.4, gcc-6.1, and clang-3.8 on Linux (Archlinux)
- jcranmer 10y ago
  It's not practical to make this a warning because of, e.g., dlsym (which is actually worse because void * and void (*)() aren't necessarily compatible types). Another thing which makes this sort of thing difficult to warn in is the brokenness of autoconf. Autoconf checks that your system has memcpy (or anything that you check via AC_TRY_LINK) by compiling this program: void memcpy(); int main() { (void)memcpy(); }
  - jandrese 10y ago
    Wait, undefined parameters are guaranteed to be zero in C? That's something I wouldn't have expected. Certainly nothing I would rely on in my own code.
quotemstr 10y ago
Here's an example I'm rather proud of. It involves setting getpid(2) as a signal handler(2) via JNA. https://github.com/facebook/buck/commit/48f1e201a966b9e2adabdde4a7425d0f3989208f https://github.com/facebook/buck/commit/48f1e201a966b9e2adab...
- DblPlusUngood 10y ago
  Why is setting the signal handler for SIGHUP to getpid preferable to masking the signal?
  - swolchok 10y ago
    > But when we exec a child, the kernel resets any signal not set to `SIG_IGN` to `SIG_DFL`, so all our children, atomically, get their `SIGCHLD` handlers set to `SIG_DFL` at the moment of creation. Consequently, when either our process or the kernel sends `SIGHUP` to our process group, all our children die.
    - DblPlusUngood 10y ago
      SIG_IGN is not the same as the signal mask. The result is the same though; I forgot that signal mask is not reset on exec. Damn POSIX signals!
pavlov 10y ago
... we saw how register allocation and calling conventions — supposedly the exclusive concern of assembly-spinning compiler writers — occasionally pop their heads up in C ... I don't know about "occasionally". At least on Windows, calling conventions used to be a constant headache. The C language default was cdecl, but the Win32 API default (most of the time) was stdcall. Then there was a variant (fastcall?) that tried to make use of registers. Maybe it's all fixed now... But this being Win32, I rather expect they've managed to accumulate a few more "interesting" gotchas and edge cases as the platform expanded with 64-bit, WinRT, Universal Windows Platform and whatever.
- quotemstr 10y ago
  Fortunately, in the amd64 Windows world, there's only one calling convention. (Edit: as cremno points out, there's now a new vectorcall convention. Dammit.) The 64-bit Windows ABI is a big improvement over the old 32-bit one. Other improvements are use of unwind tables for SEH and the requirement that all code be unwindable.
  - pavlov 10y ago
    I'm happy to hear that! With a fixed ABI, C++ 1x language features and UWP, it sounds like doing C/C++ on Windows could actually be fun these days.
    - jheriko 10y ago
      don't get ahead of yourself... that ABI isn't there yet. :P
  - cremno 10y ago
    >Fortunately, in the amd64 Windows world, there's only one calling convention. Not anymore: https://blogs.msdn.microsoft.com/vcblog/2013/07/11/introducing-vector-calling-convention/ https://blogs.msdn.microsoft.com/vcblog/2013/07/11/introduci...
  - jheriko 10y ago
    > the requirement that all code be unwindable. i would not call that an improvement in all cases... the extra code produced by enabling exceptions, even for the latest compilers, produces enough bloat and slow down for basically everyone seriously concerned with real-time performance to turn them off. small simple functions are forced to have prologue/epilogue, use a frame pointer and to preserve the return address on the stack. doesn't sound like much, but if your function does anything simple (float f() { return 1; } bool g(float a, float b) { return a < b; } etc...) you spend all the time paying for exceptions instead of the intended functionality.
david-given 10y ago
Incidentally, the codebase I'm looking at right now contains gems like this: /*VARARGS2*/ int margin_printf (fp, a, b, c, d, e, f, g) FILE *fp; char *a; long b, c, d, e, f, g; { ind_printf (0, fp, a, b, c, d, e, f, g); } /* margin_printf */ Called (from a different source file) like this: margin_printf (outfile, length ? "/* %s */\n" : "\n", storage); Okay, so that's K&R C, and it's not actually compiled any more (because I've been slowly taking this stuff out and replacing it with things that actually work), but still --- the horror, the horror...
- caf 10y ago
  That kind of hackery was necessary because K&R C didn't give you any sanctioned way to write your own printf-like functions. ANSI C introduced <stdarg.h>.
  - david-given 10y ago
    Sure. Also, K&R C tended to target platforms with really simple ABIs, where parameters were always passed on the stack and everything was a machine word; so you could get away with this kind of thing. K&R C actually has a minimalist elegance to it that later versions of C lost. Not that I'm claiming that it's better, of course. But K&R C had a distinct philosophy to it that was definitely its own.
jheriko 10y ago
its a shame this only glosses over the subject, and starts with a highly misleading example and then continues with plenty more misleading information. unix/windows are not interchangeable with compilers targetting those platforms. the article doesn't make the distinction or highlight why it matters - the OS has nothing to do with calling conventions in this context, they are just very conveniently the same in this case. this example doesn't work with vs2010 out of the box because it targets win32 by default... for example. its the compilers, and only the compilers, which are deciding this. the standard calling convention for the platform is incidental, and it being identical to the default for the compiler is a wise design choice for compilers targetting x64, rather than a necessity - something which is much clearer in the old world of windows and x86 (32-bit) where cdecl was the default out of the box, and stdcall was what windows api calls expected.
- cbr 10y ago
  Doesn't the compiler at least need to use the platform-standard argument convention when calling OS functions?
  - inlined 10y ago
    Yes. Every Windows API is prefixed with WINAPI, which is a macro for the calling convention.
    - int_19h 10y ago
      The macro expands to nothing on Win64, though, because it has one standard calling convention. (At the ABI boundary, anyway. There's also __vectorcall now. And, of course, the compiler can use whatever it wants for functions that are not exposed at the boundary - even arbitrary register assignments based on where the function is actually called, to minimize the amount of work needed to save registers etc.)

164 points by heliostatic 10y ago

Sir, while the gentlemen and ladies of this fine organization are infinitely pleased with your research into the black magic of the nether regions and the treatise you have presented from aforesaid research, we would ask you please be even more strident in your warnings about practicing this subject. It seems a number of neophytes have disappeared under odd circumstances in the weeks since you presented your work, and the interruption to the experiments they were assisting with has become quite burdensome. Additionally, we expect training and recruitment costs to be quite a bit higher than normal next year, and as you well know our coffers are not unlimited.

catnaroek 10y ago
> black magic of the nether regions Or plain abstraction violation. “Black magic” gives it an undeserved good reputation: “I must aspire to be clever enough to understand and actively make use of this.”
- kbenson 10y ago
  Maybe. Given it relies on the calling convention of the OS you're on, the CPU architecture you're on, and knowledge of assembly, it could very well be both.
  - int_19h 10y ago
    It also relies on implementation-specific behavior in the compiler, since the latter is not actually required to follow established the convention for argument passing, so long as it can prove that the only place the function is called from is the same translation unit. In fact, so far as I can tell by looking at the first example, the compiler can circumvent this whole thing by noting that: 1. DoubleToTheInt is only ever called in the same unit, so it can be inlined - no need to generate the function. 2. Pointer to DoubleToTheInt is created, but is then immediately cast to an incompatible function type, which is undefined behavior - and so it can be substituted with any random value, say, whatever happens to be in EAX at that moment (any U.B. is substitutable for any other U.B.!), and the compiler can still avoid generating the actual function.
- chriswarbo 10y ago
  > “Black magic” gives it an undeserved good reputation From http://catb.org/jargon/html/B/black-magic.html http://catb.org/jargon/html/B/black-magic.html > black magic: n. > [common] A technique that works, though nobody really understands why. More obscure than voodoo programming, which may be done by cookbook. Compare also black art, deep magic, and magic number (sense 2).
ChuckMcM 10y ago
Nicely done. The first time I came across the information of punning vs code was during the SPARC ABI standards work where the test code was working but the test case was supposed to fail. It happened to be a vendor compiler that optimized certain four parameter function calls in a similar way. I was rather perplexed how a compiler could "work" and "not work" at the same time.
david-given 10y ago
...I once wrote a program to summon demons. It didn't work, mind. But I was really proud of my doubly-linked-list pentacle, each node of which contained a pointer to the summoning point in the centre. I thought that was really elegant. Incidentally, if anyone is thinking of trying this for yourself: always remember to mlock() the memory containing your summoning. The last thing you want is some demon of the pit getting swapped out to disk. Not only is swap notoriously insecure, but sulphur buildup on the disk platter will cause a disk crash. And for heaven's sake, make sure your pentacle is properly aligned. If it's split across two pages, they may end up not being next to each other in physical ram, and then it won't work! Ideally, use an embedded device where you know what every cycle does. Don't be put off by the low power. You'd be surprised how much evil you can do with an Arduino.
- jschwartzi 10y ago
  I've found that my demon summoning programs work much better if I sacrifice the ram beforehand.

tathougies 10y ago

This is undefined behavior, so you're claim that this is 'in C' is blatantly false. In general, you cannot assume the underlying architecture will pass its arguments in any particular location.

catnaroek 10y ago
The problem is that determining whether an ostensibly C (syntactically valid and typable) program is actually written “in C” is undecidable.
- adrianN 10y ago
  Almost any interesting property about programs is undecidable. Yet somehow programmers still write nontrivial programs. Undefined behavior in C sucks, but being its presence or absence being undecidable is not the main reason.
  - catnaroek 10y ago
    > Almost any interesting property about programs is undecidable. It's only undecidable if the language makes it undecidable. For example, subject reduction is decidable in a type-safe language (EDIT: with decidable type checking, oops!), because the very definition of type safety means that typability implies subject reduction. Of course, not everything of interest can be made decidable in a general-purpose language, but the integrity of a language's basic abstractions should never be in question.
  - haldean 10y ago
    The formalization of this idea is Rice's Theorem: https://en.wikipedia.org/wiki/Rice%27s_theorem https://en.wikipedia.org/wiki/Rice%27s_theorem
    - catnaroek 10y ago
      Rice's theorem is a statement about partial functions, not programs. There exist decidable properties of programs: Is it of size less than N (for any suitable measure)? Does it finish in less than N steps (for a given fixed input)? Two syntactically distinct programs may compute the same partial function.

bitwize 10y ago

I freaking hate you, OP. I really freaking hate you right now. This is an abomination in the sight of God.

pklausler 10y ago

If your C compiler doesn't complain bitterly to you about an incompatible pointer type on that assignment, get a better C compiler. If your C compiler does complain bitterly about it, heed those warnings or expect no sympathy. If you write this in production code, and don't get sent home that day in tears, your company's code review process has failed.

akkartik 10y ago
No warnings (-W -Wall) with gcc 4.2.1 on OSX :/ gcc 4.8.4 on Ubuntu 14.04 Trusty still gives no warnings, but gives a different result: $ gcc -W -Wall x.c -lm $ a.out (0.99)^100: 0.366032 (0.99)^100: inf clang 3.4 on Ubuntu 14.04: $ clang -W -Wall -fsanitize=undefined x.c -lm $ a.out (0.99)^100: 0.366032 (0.99)^100: inf :(
- codemac 10y ago
  ; gcc -v |[2] grep version gcc version 6.1.1 20160802 (GCC) ; gcc -Wall -Werror ./something.c -lm ; ./a.out (0.99)^100: 0.366032 (0.99)^100: 0.000000 ;
- pavanky 10y ago
  I get 0.366032 on gcc-5.4, gcc-6.1, and clang-3.8 on Linux (Archlinux)
jcranmer 10y ago
It's not practical to make this a warning because of, e.g., dlsym (which is actually worse because void * and void (*)() aren't necessarily compatible types). Another thing which makes this sort of thing difficult to warn in is the brokenness of autoconf. Autoconf checks that your system has memcpy (or anything that you check via AC_TRY_LINK) by compiling this program: void memcpy(); int main() { (void)memcpy(); }
- jandrese 10y ago
  Wait, undefined parameters are guaranteed to be zero in C? That's something I wouldn't have expected. Certainly nothing I would rely on in my own code.

quotemstr 10y ago

Here's an example I'm rather proud of. It involves setting getpid(2) as a signal handler(2) via JNA. https://github.com/facebook/buck/commit/48f1e201a966b9e2adabdde4a7425d0f3989208f https://github.com/facebook/buck/commit/48f1e201a966b9e2adab...

DblPlusUngood 10y ago
Why is setting the signal handler for SIGHUP to getpid preferable to masking the signal?
- swolchok 10y ago
  > But when we exec a child, the kernel resets any signal not set to `SIG_IGN` to `SIG_DFL`, so all our children, atomically, get their `SIGCHLD` handlers set to `SIG_DFL` at the moment of creation. Consequently, when either our process or the kernel sends `SIGHUP` to our process group, all our children die.
  - DblPlusUngood 10y ago
    SIG_IGN is not the same as the signal mask. The result is the same though; I forgot that signal mask is not reset on exec. Damn POSIX signals!

pavlov 10y ago

... we saw how register allocation and calling conventions — supposedly the exclusive concern of assembly-spinning compiler writers — occasionally pop their heads up in C ... I don't know about "occasionally". At least on Windows, calling conventions used to be a constant headache. The C language default was cdecl, but the Win32 API default (most of the time) was stdcall. Then there was a variant (fastcall?) that tried to make use of registers. Maybe it's all fixed now... But this being Win32, I rather expect they've managed to accumulate a few more "interesting" gotchas and edge cases as the platform expanded with 64-bit, WinRT, Universal Windows Platform and whatever.

quotemstr 10y ago
Fortunately, in the amd64 Windows world, there's only one calling convention. (Edit: as cremno points out, there's now a new vectorcall convention. Dammit.) The 64-bit Windows ABI is a big improvement over the old 32-bit one. Other improvements are use of unwind tables for SEH and the requirement that all code be unwindable.
- pavlov 10y ago
  I'm happy to hear that! With a fixed ABI, C++ 1x language features and UWP, it sounds like doing C/C++ on Windows could actually be fun these days.
  - jheriko 10y ago
    don't get ahead of yourself... that ABI isn't there yet. :P
- cremno 10y ago
  >Fortunately, in the amd64 Windows world, there's only one calling convention. Not anymore: https://blogs.msdn.microsoft.com/vcblog/2013/07/11/introducing-vector-calling-convention/ https://blogs.msdn.microsoft.com/vcblog/2013/07/11/introduci...
- jheriko 10y ago
  > the requirement that all code be unwindable. i would not call that an improvement in all cases... the extra code produced by enabling exceptions, even for the latest compilers, produces enough bloat and slow down for basically everyone seriously concerned with real-time performance to turn them off. small simple functions are forced to have prologue/epilogue, use a frame pointer and to preserve the return address on the stack. doesn't sound like much, but if your function does anything simple (float f() { return 1; } bool g(float a, float b) { return a < b; } etc...) you spend all the time paying for exceptions instead of the intended functionality.

david-given 10y ago

Incidentally, the codebase I'm looking at right now contains gems like this: /*VARARGS2*/ int margin_printf (fp, a, b, c, d, e, f, g) FILE *fp; char *a; long b, c, d, e, f, g; { ind_printf (0, fp, a, b, c, d, e, f, g); } /* margin_printf */ Called (from a different source file) like this: margin_printf (outfile, length ? "/* %s */\n" : "\n", storage); Okay, so that's K&R C, and it's not actually compiled any more (because I've been slowly taking this stuff out and replacing it with things that actually work), but still --- the horror, the horror...

caf 10y ago
That kind of hackery was necessary because K&R C didn't give you any sanctioned way to write your own printf-like functions. ANSI C introduced <stdarg.h>.
- david-given 10y ago
  Sure. Also, K&R C tended to target platforms with really simple ABIs, where parameters were always passed on the stack and everything was a machine word; so you could get away with this kind of thing. K&R C actually has a minimalist elegance to it that later versions of C lost. Not that I'm claiming that it's better, of course. But K&R C had a distinct philosophy to it that was definitely its own.

jheriko 10y ago

its a shame this only glosses over the subject, and starts with a highly misleading example and then continues with plenty more misleading information. unix/windows are not interchangeable with compilers targetting those platforms. the article doesn't make the distinction or highlight why it matters - the OS has nothing to do with calling conventions in this context, they are just very conveniently the same in this case. this example doesn't work with vs2010 out of the box because it targets win32 by default... for example. its the compilers, and only the compilers, which are deciding this. the standard calling convention for the platform is incidental, and it being identical to the default for the compiler is a wise design choice for compilers targetting x64, rather than a necessity - something which is much clearer in the old world of windows and x86 (32-bit) where cdecl was the default out of the box, and stdcall was what windows api calls expected.

cbr 10y ago
Doesn't the compiler at least need to use the platform-standard argument convention when calling OS functions?
- inlined 10y ago
  Yes. Every Windows API is prefixed with WINAPI, which is a macro for the calling convention.
  - int_19h 10y ago
    The macro expands to nothing on Win64, though, because it has one standard calling convention. (At the ABI boundary, anyway. There's also __vectorcall now. And, of course, the compiler can use whatever it wants for functions that are not exposed at the boundary - even arbitrary register assignments based on where the function is actually called, to minimize the amount of work needed to save registers etc.)