7 ms·
Recursive macros in C, demystified (once the ugly crying stops)
- hyperhello 1y agoCan I use this technique to expand MACRO(a,b,c,…) into something like F(a,b,c…); G(a,b,c…)?
- viega 1y agoThat's just: ``` #define MACRO(...) F(__VA_ARGS__); G(__VA_ARGS__) ``` The technique in the article is more often used to type check the individual parameters, or wrap a function call around them individually, etc.
- hyperhello 1y agoOk. How about into F(a,”a”);F(b,”b”);etc. The problem being automating enums and their names in one call. Like MACRO(a,b,c) and getting a map from a to “a”.
- viega 1y ago100%, that's definitely easy to do once you understand the technique.
- hyperhello 1y agoPlease?
- viega 1y agoI'm on my phone, but if you start with the top 8 lines in the code box under the ascii art, you'll get an implementation of `H4X0R_MAP()`; the bottom two lines are an example, and you can just write yourself a body that produces one term. Only thing you need to know beyond that is the stringify operator.
- viega 1y agoAnd I should say, if you want to apply the same transformation to arguments twice, but call F() separately from G() per your starting example, you'd just apply your map twice in your top-level macro.
- WalterBright 1y ago> if you want to apply the same transformation to arguments twice, but call F() > separately from G() per your starting example, you'd just apply your map twice in your top-level macro My brain just blew a fuse.
- viega 1y agoOkay, finally found some time to provide you with a fully annotated example of your original ask here, assuming you wanted to transform the arguments passed to F into IDs, and the arguments passed to G into strings (as seemed to be the case from the rest of the thread). https://c.godbolt.org/z/6zqx1dsn3 https://c.godbolt.org/z/6zqx1dsn3 I've fully annotated it, so it might seem like more than it is. About half the macro code is from the original article (the chunk at the top). And I do implement both transforms for you. Each one I think is only 6 lines of code by itself, despite the rediculous amount of exposition in the comments. If you have any questions about it, let me know.
- deleted 1y ago[deleted]
- dandersch 1y agoRelated: The Preprocessor Iceberg https://jadlevesque.github.io/PPMP-Iceberg/ https://jadlevesque.github.io/PPMP-Iceberg/ There you can find a recursive macro expansion implementation (as a gcc hack) that fits on a slide: #2""3 #define PRAGMA(...) _Pragma(#__VA_ARGS__) #define REVIVE(m) PRAGMA(push_macro(#m))PRAGMA(pop_macro(#m)) #define DEC(n,...) (__VA_ARGS__) #define FX(f,x) REVIVE(FX) f x #define HOW_MANY_ARGS(...) REVIVE(HOW_MANY_ARGS) \ __VA_OPT__(+1 FX(HOW_MANY_ARGS, DEC(__VA_ARGS__))) int main () { printf("%i", HOW_MANY_ARGS(1,2,3,4,5)); // 5 } It sounds like the one in the article works for more compilers, but there doesn't seem to be a copy-pasteable example anywhere to check for myself. Also, the "Our GitHub Org" link on the site just links to github.com.
- lifthrasiir 1y agoAsk and you'll get: https://c.godbolt.org/z/rKsWT5E9T https://c.godbolt.org/z/rKsWT5E9T It seems that MSVC doesn't like those macros, though.
- david2ndaccount 1y agoWorks with MSVC if you add /Zc:preprocessor (to get a standard compliant preprocessor instead of the legacy one).
- viega 1y agoAuthor of the article here. Absolutely, the code box under the ascii art is a complete implementation, just paste that in a C file, and then use `H4X0R_VA_COUNT(...)`. Or, you could follow the link the my typed variadic arguments article (from which this post forked off). The repo there is: https://codeberg.org/h4x0r/vargs https://codeberg.org/h4x0r/vargs
- viega 1y agoAnd yes, GCC extensions are often going to be adopted in clang, but generally not the broader world of C and C++ compilers. Everything in my article conforms to the standard.
- stevefan1999 1y agoI used to write a preprocessor until I noticed those kind of thing...I stopped writing it after that
- chirsz 1y agoThe behavior of C macros is actually described by a piece of pseudocode from Dave Prosser and it is not in the standard: * https://www.spinellis.gr/blog/20060626/ https://www.spinellis.gr/blog/20060626/ * https://www.spinellis.gr/pubs/jrnl/2006-DDJ-Finessing/html/Spi06j.htm https://www.spinellis.gr/pubs/jrnl/2006-DDJ-Finessing/html/S... * https://gcc.gnu.org/legacy-ml/gcc-prs/2001-q1/msg00495.html https://gcc.gnu.org/legacy-ml/gcc-prs/2001-q1/msg00495.html
- viega 1y agoWow, I'm not sure I've ever seen this (or if I did, it was 20 years ago). And I was definitely looking around for this kind of history when I was searching around when writing. Perhaps my google skills have decayed... or google... or both! Thanks very much.
- camel-cdr 1y agoThis is the best explanation I've seen so far: https://marc.info/?l=boost&m=118835769257658&w=2 https://marc.info/?l=boost&m=118835769257658&w=2
- deleted 1y ago[deleted]
- russfink 1y agoIs this a DoS risk - code that sends your build chain into an infinite loop?
- viega 1y agoNo. Other modern languages have strong compile-time execution capabilities, including Zig, Rust and C++. And my understanding is that C is looking to move in that direction, though as with C++, macros will not go away.
- saghm 1y agoWithout any specific implementation of a constraint it certainly can happen, although I'm not totally sure that it's something to be concerned about in terms of a DOS as much as a nuisance when writing code with a bug in it; if you're including malicious code, there's probably much worse things it could do if it actually builds properly instead of just spinning indefinitely. Rust's macros are recursive intentionally, and the compiler implements a recursion limit that IIRC defaults to 64, at which point it will error out and mention that you need to increase it with an attribute in the code if you need it to be higher. This isn't just for macros though, as I've seen it get triggered before with the compiler attempting to resolve deeply nested generics, so it seems plausible to me that C compilers might already have some sort of internal check for this. At the very least, C++ templates certainly can get pretty deeply nested, and given that the major C compilers are pretty closely related to their C++ counterparts, maybe this is something that exists in the shared part of the compiler logic.
- viega 1y agoC++ also has constexpr functions, which can be recursive. All code can have bugs, error out and die. There are lots of good reasons to run code at compile time, most commonly to generate code, especially tedious and error-prone code. If the language doesn't have good built-in facilities to do that, then people will write separate programs as part of the build, which adds system complexity, which is, in my experience, worse for C than for most other languages. If a language can remove that build complexity, and the semantics are clear enough to the average programmer (For example, Nim's macro system which originally were highly appealing (and easy) to me as a compiler guy, until I saw how other people find even simple examples completely opaque-- worse than C macros.
- pjsg 1y agoI wept when the author mentioned implementing SHA256 in macros.
- fuhsnn 1y agoI wonder if the author is aware of the __VA_TAIL__ proposal[1], it covered similar grounds and IMO very well thought out, but unfortunately not accepted into C2Y (judging from committee meeting minutes). [1] https://www.open-std.org/jtc1/sc22/wg14/www/docs/n3307.htm https://www.open-std.org/jtc1/sc22/wg14/www/docs/n3307.htm
- viega 1y agoYes, I know that it was not accepted, but do not have any color on why not. It's well thought out; but I do not think the semantics are self-evident to the average C programmer who already finds the preprocessor inscrutable.
- WalterBright 1y agoImagine trying to implement the C preprocessor. I had to write it from scratch 3 times before it worked 100%.
- viega 1y agoWow, you are a braver person than I. Well done.
- WalterBright 1y agoThank you. I am actually perversely proud of it.
- le-mark 1y agoThis was back in 80s when you were working on c compilers? That’s an interesting story (80s compiler scene and what you worked on) I’ve picked up bits and pieces over the years, have you written it up anywhere? Would be benefit for many I think.
- WalterBright 1y agohttps://github.com/DigitalMars/dmpp https://github.com/DigitalMars/dmpp It's Boost licensed. Open sourced. No need for anyone else to struggle to implement a preprocessor.
- camel-cdr 1y agomicrosoft fixed their broken c preprocessor implementation just a few years ago
- WalterBright 1y agoThey could have just asked me :-)
- 1y ago
- camel-cdr 1y agoI did the first week of AOC22 in the C preprocessor: https://github.com/camel-cdr/boline/tree/main/aoc22 https://github.com/camel-cdr/boline/tree/main/aoc22
- winocm 1y agoMildly related, sort of, one can prevent expansion of variadic macros as follows: #define printf(...) int (printf)(const char *, ...); I keep on seeing many random code bases just resort to #undef instead...
- pwdisswordfishy 1y agoDoesn't this trigger warnings?
- joriatsy 1y agoFunction like macros literally requires name( , i.e name followed directly by open paren, otherwise no macro substitution occurs. so (name)() will always suppress function like macros (but not non-function ones, i.e regular #define name xxx)
- MangoToupe 1y agoThe c pre processor. C doesn't have macros. It's fucking miserable. Anyone who uses it is a masochist
- procaryote 1y agoIt's a feature... macros allows people to change the language at will which is great when you're researching programming languages (like in lisp) but less good when you want maintainable and consistent code C++ has more powerful metaprogramming and look how that turned out
- MangoToupe 1y agoNo, it is not a feature. It prevents C from being parseable and precludes actual metaprocessing. Just use lisp if you want to write maintainable code Nobody wants to end up like C++ of course, you ain't wrong about that
- bluGill 1y agoAt least you can implement things like vector in C++. I challenge anyone to write the same for C macros. (this is semi-serious - if you get a full generic STL for C that is usable that would prevent a large number of bugs. The implementation can be as bad as you want, so long as the user API is not too difficult. Though note that I'm not a C programmer and so if there are better ways someone else is doing this I'm not aware of it)
- uecker 1y agoI wonder how you like my vector: https://godbolt.org/z/97YGrbP9s https://godbolt.org/z/97YGrbP9s (note, experimental library, but I see no fundamental issue).
- bluGill 1y agothat is amazing... I don't write C so I didn't dig too deep, but kudos for getting anything to work. Now you use need great documentation so we can figure out how to use it and what it supports without digging into the macros themselves. (tests would be good too, but maybe they are there and I didn't see them).
- procaryote 1y agoIn many ways being limited ends up being a feature. Even limited as it is, you get some crimes against humanity like the bourne shell source, but at least most people agree it is a bad idea If it allowed more unlimited metaprogramming, building big complex things as macros might well have become popular
- lpribis 1y agoThe CPP does allow quite advanced metaprogramming, it's just so obtuse to use and requires insane hacks so almost nobody does. See one of my favorite projects https://github.com/hirrolot/metalang99 https://github.com/hirrolot/metalang99
- MangoToupe 1y agoThe CPP allows “advanced metaprogramming” like throating a banana can sometimes sound like speech
- parados 1y agoA C preprocessor implemented in Python: https://github.com/paulross/cpip https://github.com/paulross/cpip
- kragen 1y agoI think the C preprocessor was designed after the GPM clone m6, its successor m4, and Ratfor, so I suspect the difficulty in doing things like this is intentional. I guess I should ask McIlroy, who is responsible for pushing m4 to its absolute limits and was present when the C preprocessor was being designed: https://www.cs.dartmouth.edu/~doug/barem4.m4 https://www.cs.dartmouth.edu/~doug/barem4.m4 _ Pure macros as a programming language _ _ m4 is Turing complete even when stripped to the bare minimum _ of one builtin: `define'. This is not news; Christopher _ Strachey demonstrated it in his ancestral GPM, described in _ "A general- purpose macrogenerator", The Computer Journal 8 _ (1965) 225-241. _ _ This m4 program more fully illustrates universality by _ building familiar programming capabilities: unlimited _ precision integer arithmetic, boolean algebra, conditional _ execution, case-switching, and some higher-level operators _ from functional programming. In support of these normal _ facilities, however, the program exploits some unusual _ programming idioms: _ _ 1. Case-switching via macro names constructed on the fly. _ 2. Equality testing by redefining macros. _ 3. Representing data structures by nested parenthesized lists. _ 4. Using macros as associative memory. _ 5. Inserting nested parameter symbols on the fly. _ _ Idioms 2 and 5 are "reflective": the program writes code _ for itself. It's very easy to get into enormous amounts of trouble in m4, m6, or GPM. The C preprocessor is not without its problems, but it is rare that I have difficulty in understanding why a given gcc -E invocation produces the output it does.
- 0x69420 1y agogenuinely remarkable, the altogether perhaps even productive mischief you can get up to, especially with `__VA_OPT__` becoming a proper standard in both C and C++ so you don't have to feel dirty about using it. i recently made use of plenty of ugly tricks in this vein to take a single authoritative table of macro invocations that defined a bunch of pixel formats, and make them graduate from defining bitfield structs to classes with accessors that performed good old fashioned shifts and masks, all without ever specifying the individual bit offsets of channels, just their individual widths, and macro magic did the rest. no templates, no actual c++, could just as feasibly produce pure c bindings down the line by just changing a few names. getting really into this stuff makes you stop thinking of c function-like macros as functions of their arguments as such, but rather unary functions of argument lists, where arity roughly becomes the one notion vaguely akin to typing in the whole enterprise, or at least the one place where the compiler exhibits behaviour resembling that of a type checker. this was especially true considering the entries in the table i wound up with were variadic, terminating in variably many (name, width) parenthesised tuples. and i just... had the means to "uncons" them so to speak. fun stuff. this is worth it, imo, in precisely one context, which is: you want a single source of truth that defines fiddly but formulaic implementations spread across multiple files that must remain coordinated, and this is something you do infrequently enough that you don't consider it worthwhile introducing "real" "big boy" code gen into your build process. mind, you usually do end up having to commit to a little utility header that defines convenient macros (_Ex and such in the article), but hey. c'est la vie. basically x macros (https://en.wikipedia.org/wiki/X_macro https://en.wikipedia.org/wiki/X_macro) on heart attack quantities of steroids.
- Joker_vD 1y ago#define _H4X0R_CONVERT_ONE(arg) \ ((union { unsigned long long u; void *v; }){ \ .u = (unsigned long long)arg, \ }).v Couldn't this be just #define _H4X0R_CONVERT_ONE(arg) (void*)(uintptr_t)(arg) ? Also, thanks, now I can finally use void my_printf(const char *fmt, void* args[], size_t argc); ergonomically: #define my_printf(fmt, ...) (my_printf)((fmt), \ (void*[]){ H4X0R_VA_VOID_STAR_CONVERT(__VA_ARGS__) }, \ H4X0R_VA_COUNT(__VA_ARGS__)) int main(int argc, char **argv) { my_printf("int: %d, ptr: %p, str: %s, missing: %d\n", 42, argv, "Hello world!"); } $ gcc test.c && ./a.out int: 42, ptr: 0x7FFF46AA3E78, str: Hello world!, missing: %!d(MISSING) Funnily enough, the difference between passing ... and locally-allocated void*[] is basically who has to spill the data to the stack, the caller or the called function.
- viega 1y agoWell, I've done it that way if I'm willing to limit myself to pointers or ints up to a pointer size, but that doesn't work with floats or doubles, for instance. Ergonomically, I have tended to start using _Generic for static type checking where possible, and that pushes me more toward to avoiding arrays in types for this kind of thing.
- SAI_Peregrinus 1y agoOne can also (ab)use the build system to run arbitrary preprocessing steps with any language over the "C" input. You can have recursive macros by using M4 or Perl or Python or some other language to expand them, converting your "foo.c.in" into a "foo.c" to hand off to the C preprocessor & compiler. It still feels dirty, but it's often much easier to understand & debug.
- viega 1y agoYes, 100%. And since CPP doesn't actually understand C, it's not too hard to do some lightweight preprocessing that requires some real additional parsing. But while CPP is pretty finicky and not very modern, getting such things working seamlessly with C build systems can be vastly worse (though better than the days where the GNU tools were ubiquitous). I tend to find meson easy to use compared to all the others, and do this kind of thing, but it's still difficult and brittle.
- tester756 1y agoMacro is one of the ugliest features available in langs like C/CPP
- bluGill 1y agoI've ready the article 4 times already today and I'm still crying. This looks like the solution to a problem I'm having (C++, but I'm doing things that templates and constexpr can't do), but trying to get it all to work is painful. Kudos to the author at making an attempt to explain it.
- viega 1y agoIf you can give me specifics on how it's not clear, I'd very much want to improve it. Please DM me about it.
- bluGill 1y agoNot so much that you are not clear - this is an area that is weird enough that I don't think it is possible to be clear, but you gave me some useful hints. I got some of what I want working. Edit, I started writing and then realized that I need this to work on old gcc that doesn't support these tricks so I have to give up. (I can't wait until this old embedded hardware dies) What I'm trying to do is #define(A, B) static int A ## _ ## B = register(MKSTRING(A ## _ ## B) take any number of arguments. I can get the register working, but my attempts are creating the variable name fail.
- viega 1y agoThanks. Also, per another person in the thread, here are the two annotations he or she was asking for, heavily annotated. They are both only a couple lines, but it deals with things like the fact that you've got one more argument than you should have commas, and the use of the # and ## operators in these things. https://c.godbolt.org/z/6zqx1dsn3 https://c.godbolt.org/z/6zqx1dsn3
- bluGill 1y agoThanks. I didn't have enough levels of indirection. Once I add enough levels of indirection it works. Well works on my newer systems, I have to build for an embedded system where I'm stuck on an old compiler that doesn't support these tricks.
- jhallenworld 1y agoThe lack of (easy) recursion in CPP is so frustrating because it was always available in assembly languages with even very old and very simple macro assemblers- with the caveat that the recursion depth was often very limited, and no tail call elimination. For example, if you need to fill memory: ; Fill memory with backward sequence macro fill n word n if n != 0 fill n - 1 endif endm So "fill 3" expands to: word 3 word 2 word 1 word 0 There is no way this was not known about when C was created. They must have been burned by recursive macro abuse and banned it (perhaps from m4 experience as others have said). The other assembly language feature that I missed is the ability to switch sections. This is useful for building tables in a distributed fashion. Luckily you can do it with gcc.
- imglorp 1y ago> C has many advantages that have led to its longevity (60 years as perhaps the most important language). 53 years by my count. Did something relevant happen in 1960? Maybe author is alluding to B?
- viega 1y agoNo, should have said '50'. Have fixed that, thanks.
- vladsotirov 1y agoA simpler explanation of the technique in the article that I believe matches the C standard more closely is the following. Ideally the result of processing a macro ("macro-replacement") would contain no further macros. But in order to disallow infinite loops during processing of a macro, the C standard specifies that names of macros that would be processed recursively are instead painted blue ("marked") so as to never be processed. The technique in the article hinges on the fact that macro-replacement, as specified in the C standard, allows not only for the result of macro-replacement to contain these marked do-not-process macros, but also unmarked unprocessed macros. Specifically, such an unmarked unprocessed macro is a functional macro arising after processing a macro with an empty definition that separates the functional macro's name from its arguments. The technique for achieving recrusive macros is to introduce a functional macro whose definition is the would-be recursive macro, and to replace each occurence of the name of the would-be recursive macro in its own defintion with the functional macro interrupted by an macro with an empty definition. The result is an unmarked unprocessed recursive macro. Processing such a recursive macro, e.g. by including it as an argument to a macro, corresponds to taking one step of the recursion. Thus any pre-determined finite number of steps of a recurisvely defined macro can be performed. For example, the would-be recursive macro of the article #define _COUNT_ONE(x, ...) + 1 _COUNT_TOP(__VA_ARGS__) #define _COUNT_TOP(...) __VA_OPT__(_COUNT_ONE(__VA_ARGS__)) #define COUNT(...) (_COUNT_TOP(__VA_ARGS__) + 0) becomes #define EMPTY #define _COUNT_INDIRECT() _COUNT_ONE #define _COUNT_ONE(x, ...) + 1 _COUNT_TOP(__VA_ARGS__) #define _COUNT_TOP(...) __VA_OPT__(_COUNT_INDIRECT EMPTY()(__VA_ARGS__)) #define COUNT(...) (_COUNT_TOP(__VA_ARGS__) + 0) To process the COUNT(...) macro 5 times, allowing it to count up to 5 variable arguments, nest it 5 levels deep as an argument to a macro. #define EVAL1(...) __VA_ARGS__ #define EVAL5(...) EVAL1(EVAL1(EVAL1(EVAL1(EVAL1)))) EVAL5(COUNT(1,2,3))