5 ms·

Scandalous Weird Old Things About The C Preprocessor (2015)

100 points by AndreyKarpov 9y ago

rwmj 9y ago
[The scandal isn't that we're still using it?] I actually used the fact that: #define return(r) ... only expands for: return (something); and not for: return something; in a (non-serious non-production) program I wrote. I felt slightly dirty, but it made sense in the context -- tracking stack frames in order to do precise garbage collection of C code.
- naasking 9y ago
  That sounds interesting. Any links or a further description?
  - rwmj 9y ago
    Not anything released yet, but the idea is not new. I stole it from OCaml (possibly it came from elsewhere before that): https://caml.inria.fr/pub/docs/manual-ocaml/intfc.html#sec423 https://caml.inria.fr/pub/docs/manual-ocaml/intfc.html#sec42...
    - naasking 9y ago
      Ah, I was thinking it might be a more complex series of tricks to integrate GC more easily with C. For instance, redefining some C keywords, like 'return', to inject a prologue for freeing tagged locals or something.
klodolph 9y ago
A nit: the comment about how the C preprocessor can't be described by a context-free grammar is followed by a comment about how # directives are sensitive to where they are in the line. That's actually a non-sequitur! It's totally possible for context-free grammars to be line-oriented, the "context" in this sense has a very technical meaning--it means that the left side of a production rule must have only one symbol (i.e. no context).
- yorwba 9y ago
  Yes, the real reason C preprocessor syntax can't be described by a context-free grammar is that parsing and execution are interleaved. This requires the parser to carry the set of current macro definitions for each parsing decisions, which implies non-context-freeness. However, see [0] for a way to parse C with preprocessor #ifdefs (but after expanding #includes and #defines). [0] "SuperC : Parsing All of C by Taming the Preprocessor" http://cs.nyu.edu/rgrimm/papers/pldi12.pdf http://cs.nyu.edu/rgrimm/papers/pldi12.pdf
- robertelder 9y ago
  Can you give me an example grammar to help visualize your statement? I can probably make a note or clarification. I think one other point I forgot to mention was that the production rule for the # directives would have to look something like include-directive: beginning-of-file # INCLUDE ... newline-from-previous-line # INCLUDE ... Since 'beginning-of-file' isn't really something you can reduce to a token I justified that it would be context sensitive.
  - xigency 9y ago
    The best explanation is that context free languages are a superset of regular languages, and it's totally possible to write a regular expression without any extensions for that case: (^|\n)#\s*INCLUDE.
  - kpxxx 9y ago
    > Since 'beginning-of-file' isn't really something you can reduce to a token... Of course you can, generate a token immediately upon opening a file. Nothing special about b-o-f or b-o-l. Similar to EOF.
  - chubot 9y ago
    FWIW, I haven't thought too much about the case of the preprocessor, but this series of blog posts might help: http://trevorjim.com/how-to-prove-that-a-programming-language-is-context-free/ http://trevorjim.com/how-to-prove-that-a-programming-languag... My initial thoughts are that the property of being context-free only refers to a parser, not to the lexer. A lexer can be line-oriented, as mentioned. The blog post mentions the lexer issue. The definition of a context-free language is fuzzy because you could have an infinitely powerful lexer, but he gives a precise definition.
komerdoor 9y ago
I once started experimenting with (GNU) C macros: https://gist.github.com/machuidel/d7cc099ddc4970c6ddf4 https://gist.github.com/machuidel/d7cc099ddc4970c6ddf4 It became an abomination consisting out of many C-preprocessor hacks and impossible to debug. I never put it online (and I never will). In the end if you want to use C at a more abstract level you may as well use Nim (not C, but it compiles to C89), C++ (of course) or write your own code generator (using libclang with annotations for ex.).
- deleted 9y ago
  [deleted]
- bachmeier 9y ago
  Or mix in a Scheme that compiles to C, like Gambit, Chicken, or Bigloo. You can leave the low level and performance sensitive stuff in C.
- FrozenVoid 9y ago
  D mixin templates/mixins are another alternative: You can include arbitrary strings from functions running at compile time. Theoretically they're more powerful than C macros, but type safety(template parameters) makes some constructs awkward - unless you create and parse strings as tokens manually(basically reading/writing many strings to achieve what C preprocessor does with symbolic token composition). https://dlang.org/mixin.html https://dlang.org/mixin.html https://dlang.org/spec/template-mixin.html https://dlang.org/spec/template-mixin.html Templates can also include mixins: template liter(String s){mixin(s);}
cryptonector 9y ago
https://news.ycombinator.com/item?id=10945552 https://news.ycombinator.com/item?id=10945552
FrozenVoid 9y ago
The most interesting things about C preprocessor is compile-time zero-overhead tuples and variadic macro overloading. e.g you can pass (a,b,c) as single argument to function(a) macros and selectively alter the tuple members and perform clever tricks which look like lisp code blocks unrolled at compile time(visible with gcc -E). Boost preprocessor and Order use it to create very powerful abstractions. see http://www.boost.org/doc/libs/1_64_0/libs/preprocessor/doc/index.html http://www.boost.org/doc/libs/1_64_0/libs/preprocessor/doc/i... https://github.com/rofl0r/order-pp https://github.com/rofl0r/order-pp Variadic overloading of function macros is technique from http://stackoverflow.com/questions/11761703/overloading-macro-on-number-of-arguments http://stackoverflow.com/questions/11761703/overloading-macr... combined with _Generic which overloads by type. It allows things like print("abc",1.2,3) to transform into printf("%s","abc"),printf("%f",1.2),printf("%d",3) without any overhead in the generated code. see https://www.reddit.com/r/frozenvoid/wiki/voidh https://www.reddit.com/r/frozenvoid/wiki/voidh Edit: and the tuple members are just symbolic tokens, they can contain anything from code to nested tuple trees. (code can be encapsulated in https://gcc.gnu.org/onlinedocs/gcc/Statement-Exprs.html https://gcc.gnu.org/onlinedocs/gcc/Statement-Exprs.html)
PeCaN 9y ago
For better or worse, I wrote https://github.com/alpha123/cpplib https://github.com/alpha123/cpplib which is actually used in a commercial product. The C preprocessor can be remarkably useful for metaprogramming.
raarts 9y ago
Well I like it because of all the things you can DO with it, not for all the things you can do wrong with it.
jandrese 9y ago
> The C99 standard is about 500 pages, but only 19 of them are dedicated to describing how the C preprocessor should work IMHO This is not a mistake. This is a subtle indication about how much you should be doing with the C preprocessor. People who try to do crazy clever tricks with the preprocessor are usually just creating a huge problem for themselves down the road, especially when porting over to a different OS or compiler. If you're considering doing complex metaprogramming with the preprocessor you need to step back and reconsider your approach and ask yourself who is going to maintain it once you've moved on.
- microcolonel 9y ago
  You don't need to get into "complex metaprogramming" to hit preprocessor bugs in many implementations. I've been finding c preprocessor bugs to iron out in glslang over the course of more than a year, and many of them are triggered perfectly reasonable typos. The C preprocessor has just enough power for most of what it exists for (smoothing over build-time oddities, code which should/must be manually inlined with a macro to perform as desired). Yes, you can abuse it and produce Bournegol, but there are many decent reasons to use most features of the C preprocessor, and the way it is defined is largely ergonomic, aside from the whitespace gotchas around function macros. Robert is fully entitled to lament the process of implementing the preprocessor. Far be it from me to disagree. However, none of these examples seem weird if you just follow the intended implementation pattern: execute each step separately (at least as a model of behaviour) and you will have the desired result. Even the "let me destroy your world with this example" example is straightforward as long as you follow the process. -2: Digraphs and trigraphs are expanded -1: Line continuations are collapsed 0: Comments are removed or ignored 1. A macro is defined, `foo` is `<stdio.h>` 2. The first non-whitespace character of a line is # 3. The first non-whitespace character, and the contiguous non-whitespace characters form "include", followed by a macro expansion token, `foo` which we expand to `<stdio.h>` 4. Splice in the included file, and continue preprocessing 5. We're done, really, it was quite straightforward. I think Robert's pain and confusion rests solely on his desire to define C as a context free grammar, when C parsers are really character-oriented state machines.

100 points by AndreyKarpov 9y ago

rwmj 9y ago

[The scandal isn't that we're still using it?] I actually used the fact that: #define return(r) ... only expands for: return (something); and not for: return something; in a (non-serious non-production) program I wrote. I felt slightly dirty, but it made sense in the context -- tracking stack frames in order to do precise garbage collection of C code.

naasking 9y ago
That sounds interesting. Any links or a further description?
- rwmj 9y ago
  Not anything released yet, but the idea is not new. I stole it from OCaml (possibly it came from elsewhere before that): https://caml.inria.fr/pub/docs/manual-ocaml/intfc.html#sec423 https://caml.inria.fr/pub/docs/manual-ocaml/intfc.html#sec42...
  - naasking 9y ago
    Ah, I was thinking it might be a more complex series of tricks to integrate GC more easily with C. For instance, redefining some C keywords, like 'return', to inject a prologue for freeing tagged locals or something.

klodolph 9y ago

A nit: the comment about how the C preprocessor can't be described by a context-free grammar is followed by a comment about how # directives are sensitive to where they are in the line. That's actually a non-sequitur! It's totally possible for context-free grammars to be line-oriented, the "context" in this sense has a very technical meaning--it means that the left side of a production rule must have only one symbol (i.e. no context).

yorwba 9y ago
Yes, the real reason C preprocessor syntax can't be described by a context-free grammar is that parsing and execution are interleaved. This requires the parser to carry the set of current macro definitions for each parsing decisions, which implies non-context-freeness. However, see [0] for a way to parse C with preprocessor #ifdefs (but after expanding #includes and #defines). [0] "SuperC : Parsing All of C by Taming the Preprocessor" http://cs.nyu.edu/rgrimm/papers/pldi12.pdf http://cs.nyu.edu/rgrimm/papers/pldi12.pdf
robertelder 9y ago
Can you give me an example grammar to help visualize your statement? I can probably make a note or clarification. I think one other point I forgot to mention was that the production rule for the # directives would have to look something like include-directive: beginning-of-file # INCLUDE ... newline-from-previous-line # INCLUDE ... Since 'beginning-of-file' isn't really something you can reduce to a token I justified that it would be context sensitive.
- xigency 9y ago
  The best explanation is that context free languages are a superset of regular languages, and it's totally possible to write a regular expression without any extensions for that case: (^|\n)#\s*INCLUDE.
- kpxxx 9y ago
  > Since 'beginning-of-file' isn't really something you can reduce to a token... Of course you can, generate a token immediately upon opening a file. Nothing special about b-o-f or b-o-l. Similar to EOF.
- chubot 9y ago
  FWIW, I haven't thought too much about the case of the preprocessor, but this series of blog posts might help: http://trevorjim.com/how-to-prove-that-a-programming-language-is-context-free/ http://trevorjim.com/how-to-prove-that-a-programming-languag... My initial thoughts are that the property of being context-free only refers to a parser, not to the lexer. A lexer can be line-oriented, as mentioned. The blog post mentions the lexer issue. The definition of a context-free language is fuzzy because you could have an infinitely powerful lexer, but he gives a precise definition.

komerdoor 9y ago

I once started experimenting with (GNU) C macros: https://gist.github.com/machuidel/d7cc099ddc4970c6ddf4 https://gist.github.com/machuidel/d7cc099ddc4970c6ddf4 It became an abomination consisting out of many C-preprocessor hacks and impossible to debug. I never put it online (and I never will). In the end if you want to use C at a more abstract level you may as well use Nim (not C, but it compiles to C89), C++ (of course) or write your own code generator (using libclang with annotations for ex.).

deleted 9y ago
[deleted]
bachmeier 9y ago
Or mix in a Scheme that compiles to C, like Gambit, Chicken, or Bigloo. You can leave the low level and performance sensitive stuff in C.
FrozenVoid 9y ago
D mixin templates/mixins are another alternative: You can include arbitrary strings from functions running at compile time. Theoretically they're more powerful than C macros, but type safety(template parameters) makes some constructs awkward - unless you create and parse strings as tokens manually(basically reading/writing many strings to achieve what C preprocessor does with symbolic token composition). https://dlang.org/mixin.html https://dlang.org/mixin.html https://dlang.org/spec/template-mixin.html https://dlang.org/spec/template-mixin.html Templates can also include mixins: template liter(String s){mixin(s);}

cryptonector 9y ago

https://news.ycombinator.com/item?id=10945552 https://news.ycombinator.com/item?id=10945552

FrozenVoid 9y ago

The most interesting things about C preprocessor is compile-time zero-overhead tuples and variadic macro overloading. e.g you can pass (a,b,c) as single argument to function(a) macros and selectively alter the tuple members and perform clever tricks which look like lisp code blocks unrolled at compile time(visible with gcc -E). Boost preprocessor and Order use it to create very powerful abstractions. see http://www.boost.org/doc/libs/1_64_0/libs/preprocessor/doc/index.html http://www.boost.org/doc/libs/1_64_0/libs/preprocessor/doc/i... https://github.com/rofl0r/order-pp https://github.com/rofl0r/order-pp Variadic overloading of function macros is technique from http://stackoverflow.com/questions/11761703/overloading-macro-on-number-of-arguments http://stackoverflow.com/questions/11761703/overloading-macr... combined with _Generic which overloads by type. It allows things like print("abc",1.2,3) to transform into printf("%s","abc"),printf("%f",1.2),printf("%d",3) without any overhead in the generated code. see https://www.reddit.com/r/frozenvoid/wiki/voidh https://www.reddit.com/r/frozenvoid/wiki/voidh Edit: and the tuple members are just symbolic tokens, they can contain anything from code to nested tuple trees. (code can be encapsulated in https://gcc.gnu.org/onlinedocs/gcc/Statement-Exprs.html https://gcc.gnu.org/onlinedocs/gcc/Statement-Exprs.html)

PeCaN 9y ago

For better or worse, I wrote https://github.com/alpha123/cpplib https://github.com/alpha123/cpplib which is actually used in a commercial product. The C preprocessor can be remarkably useful for metaprogramming.

raarts 9y ago

Well I like it because of all the things you can DO with it, not for all the things you can do wrong with it.

jandrese 9y ago

> The C99 standard is about 500 pages, but only 19 of them are dedicated to describing how the C preprocessor should work IMHO This is not a mistake. This is a subtle indication about how much you should be doing with the C preprocessor. People who try to do crazy clever tricks with the preprocessor are usually just creating a huge problem for themselves down the road, especially when porting over to a different OS or compiler. If you're considering doing complex metaprogramming with the preprocessor you need to step back and reconsider your approach and ask yourself who is going to maintain it once you've moved on.

microcolonel 9y ago
You don't need to get into "complex metaprogramming" to hit preprocessor bugs in many implementations. I've been finding c preprocessor bugs to iron out in glslang over the course of more than a year, and many of them are triggered perfectly reasonable typos. The C preprocessor has just enough power for most of what it exists for (smoothing over build-time oddities, code which should/must be manually inlined with a macro to perform as desired). Yes, you can abuse it and produce Bournegol, but there are many decent reasons to use most features of the C preprocessor, and the way it is defined is largely ergonomic, aside from the whitespace gotchas around function macros. Robert is fully entitled to lament the process of implementing the preprocessor. Far be it from me to disagree. However, none of these examples seem weird if you just follow the intended implementation pattern: execute each step separately (at least as a model of behaviour) and you will have the desired result. Even the "let me destroy your world with this example" example is straightforward as long as you follow the process. -2: Digraphs and trigraphs are expanded -1: Line continuations are collapsed 0: Comments are removed or ignored 1. A macro is defined, `foo` is `<stdio.h>` 2. The first non-whitespace character of a line is # 3. The first non-whitespace character, and the contiguous non-whitespace characters form "include", followed by a macro expansion token, `foo` which we expand to `<stdio.h>` 4. Splice in the included file, and continue preprocessing 5. We're done, really, it was quite straightforward. I think Robert's pain and confusion rests solely on his desire to define C as a context free grammar, when C parsers are really character-oriented state machines.