3 ms·

I understand the use case of this, but when I see it I always wonder if, and think I would prefer, some external code generation step instead rather than fallin

I understand the use case of this, but when I see it I always wonder if, and think I would prefer, some external code generation step instead rather than falling back on macros in the preprocessor. Like an external script or something.

dataflow 2y ago
External codegen introduces a lot of friction in random places. Like how your editor can no longer understand the file before you start building. Or how it can go out of date with respect to the rest of your code until you build. If you can do it with a macro it tends to work better than codegen in some ways.
writebetterc 2y ago
IDEs understand preprocessor macros, so IDE features (jump2def, etc) work with this. IDEs also can expand the macro invocations. So, I prefer the macros when possible :-).
- deleted 2y ago
  [deleted]
- Someone 2y ago
  > IDEs understand preprocessor macros, so IDE features (jump2def, etc) work with this. Do they? X macros often are used with token pasting (https://gcc.gnu.org/onlinedocs/cpp/Concatenation.html https://gcc.gnu.org/onlinedocs/cpp/Concatenation.html), as for example in (FTA) #define AST_BEGIN_SUBCLASSES(NAME) START_##NAME , Are modern IDEs/compiler toolchains smart enough to tell you that START_foo was created by an expansion of that macro?
  - jcelerier 2y ago
    any non-toy IDE can do that. Most IDEs use clang directly for parsing nowadays.
  - tom_ 2y ago
    Yes. I use this with VS2019 for generating enum names, and they interact fine with auto complete and go to definition.
wat10000 2y ago
Now you have a additional stage in your build, a bunch of new code to maintain, and either a bespoke language embedded in your standard C++ or a bunch of code emitting C++ separately from the code it logically belongs with. Compare with a solution that's 100% standard C++, integrates into your build with zero work, can be immediately understood by anyone reasonably skilled in the language, and puts the "generated" code right where it belongs.
- MITSardine 2y ago
  CMake makes this pretty painless. My codegen targets have only two additional instructions to handle the generation itself and dependencies: add_custom_command to call the codegen exec, and then add_custom_target to wrap my outputs in a "virtual" target I can then make the rest of my program depend on, but this is just for tidying up. And I'll dispute the fact that any complex C prepro task "can be immediately understood by anyone reasonably skilled in the language". Besides, code should ideally be understood by "anyone reasonably likely to look at this code to work in it", not "reasonably skilled".
  - wat10000 2y ago
    This isn't complex. It's a bit unusual, but not hard to understand if you understand the basics of how #include and #define work. If you're working on the sort of C++ codebase that would benefit from this sort of code generation, and you're not reasonably skilled in C++, then god help you.
    - MITSardine 2y ago
      Are you talking about the X macro itself, or more generally? I may be the obtuse one here, but for a more complex example, it took me a few hours to manage to make nested loops using Boost PP (for explicit instantiations). Even so, I avoid having to write a new one that's not a quick copy-paste because it's quite different from usual C++ programming, so my painfully acquired understanding quickly evaporated... as I suspect is the case of anyone who doesn't particularly focus on the C prepro. In the end, it's just simpler to get some Python script or C++ program to write a string and dump that to a file than to write something illegible with the C preprocessor, if doing something at all complicated (in my opinion).
jasonthorsness 2y ago
The C# "source generator" approach is a good compromise; it runs within the build chain so has the ease-of-use of macros in that respect, but they don't need to be written in a weird macro language (they are C# or can call external tool) and when you debug your program, you debug through the generated source and can see it, more accessible than macros. Not sure if there is something similar in C/C++ integrated with the common toolchains. But when working outside C/C++ I've found myself missing the flexibility of macros more times than I can count.
- TeMPOraL 2y ago
  > But when working outside C/C++ I've found myself missing the flexibility of macros more times than I can count. Me to, and that's even in Lisp! Preprocessor macros are hard and bugprone because they share the failings of Unix philosophy of "text as universal interface" - you're playing with unstructured (or semi-structured) pieces of text, devoid of all semantics. And this is also what makes them occasionally useful - some code transformations are much, much easier to do when you can manipulate the text form directly, ignoring syntax and grammar and everything. Only the final value must be correct code - starting point and intermediary values can be anything, and you don't need to make sure you can get from here to there through valid data transformations. This is a really powerful capability to have. (I also explicitly compared preprocessor macros to "plaintext everything" experience that's seen as divine wisdom, to say: y'all are slinging unstructured text down the pipes way too much, and using preprocessor way too little.)
mauvehaus 2y ago
I've done this in C with the C preprocessor and Java with m4[0]. The upside of doing it natively is that it keeps the build simpler. And everybody at least knows about the existence of the C preprocessor, even if they don't know it well. And it's fairly limited, which prevents you from getting too clever. The big downside of doing it with the C preprocessor is that the resulting code looks like vomit if it's more than a line or two because of the lack of line breaks in the generated code. Debugging it is unenjoyable. I'd recommend against doing anything super clever. The upside of doing it out of band is that your generated source files look decent. m4 tends to introduce a little extra whitespace, but it's nothing objectionable. Plus you get more power if you really need it. The downside is that almost nobody knows m4[1]. If you choose something else, it becomes a question of what, does anyone else know it, and is it available everywhere you need to build. Honestly, integrating m4 into the build in ant really wasn't too bad. We were building on one OS on two different architectures. For anything truly cross-platform, you'll likely run into all the usual issues. ETA: Getting an IDE to understand the out of band generation might be a hassle, as other folks have mentioned. I'm a vim kinda guy for most coding, and doing it either way was pretty frictionless. The generated java code was read-only and trivial, so there wasn't a lot of reason to ever look at it. By the time you get to debugging, it would entirely transparent because you're just looking at another set of java files. [0] This was so long ago, I no longer remember why it seemed like a good idea. I think there was an interface, a trivial implementation, and some other thing? Maybe something JNI-related? At least at first, things were changing often enough that I didn't want to have to keep three things in sync by hand. [1] Including me. I re-learn just enough to get done with the job at hand every time I need it.
drwu 2y ago
Another issue is cross-compiling. External code generation requires (cross) execution of the (cross) compiled binary program.
- rcxdude 2y ago
  Or to build the generating code for the host instead. Most build systems that support cross-compilation can do this, except CMake.
MITSardine 2y ago
After trying to wrangle Boost PP and other advertised compile-time libraries such as Boost Hana (which still has some runtime overhead compared to the same logic with hardcoded values), I've finally converged to simply writing C++ files that write other C++ files. Could be Python, but I rather keep the build simple in my C++ project. Code generation is painless with CMake, no idea with other build configuration utilities.
- cbuq 2y ago
  This sounds pragmatic, but are you writing C++ executables that when run create the generated code? Are there templating libraries involved?
  - MITSardine 2y ago
    Yeah, it's all done automatically when you build, and dependencies are properly taken into account: if you modify one of the code generating sources, its outputs are regenerated, and everything that depends on them is correctly recompiled. This doesn't take much CMake logic at all to make work. In my case, no, it's dumb old code writing strings and dumping that to files. You could do whatever you want in there, it's just a program that writes source files. I do use some template metaprogramming where it's practical versus code generation, and Boost Hana provides some algorithmic facilities at compile time but those incur some runtime cost. For instance, you can write a while loop with bounds evaluated at compile time, that lets you use its index as a template parameter or evaluate constexpr functions on. But sometimes the best solution has been (for me, performance/complexity wise) to just write dumb files that hardcode things for different cases.
- Arech 2y ago
  > Boost Hana (which still has some runtime overhead compared to the same logic with hardcoded values) Can you elaborate on that? What was your use-case for which this was true?
  - MITSardine 2y ago
    One case I benchmarked was Bernstein/Bézier and Lagrange element evaluation. This is: given a degree d triangle or tetrahedron, given some barycentric coordinates, get the physical coordinate and the Jacobian matrix of the mapping. Degree 2, Lagrange: - runtime: 3.6M/s - Hana: 16.2M/s - Hardcoded: 37.7M/s Degree 3, Lagrange: 2.6M/s, 6.4M/s, 13.4M/s (same order). "Runtime" here means everything is done using runtime loops, "Hana" using Boost Hana to make loops compile-time and use some constexpr ordering arrays, "hardcoded" is a very Fortran-looking function with all hardcoded indices and operations all unrolled. As you see, using Boost Hana does bring about some improvement, but there is still a factor 2x between that and hardcoded. This is all compiled with Release optimization flags. Technically, the Hana implementation is doing the same operations in the same order as the hardcoded version, all indices known at compile time, which is why I say there must be some runtime overhead to using hana::while. In the case of Bernstein elements, the best solution is to use de Casteljau's recursive algorithm using templates (10x to 15x speedup to runtime recursive depending on degree). But not everything recasts itself nicely as a recursive algorithm, or I didn't find the way for Lagrange anyways. I did enable flto as, from my understanding (looking at call stacks), hana::while creates lambda functions, so perhaps a simple function optimization becomes a cross-unit affair if it calls hana::while. (speculating) Similar results to compute Bernstein coefficients of the Jacobian matrix determinant of a Q2 tetrahedron, factor 5x from "runtime" to "hana" (only difference is for loops become hana::whiles), factor 3x from "hana" to "hardcoded" (the loops are unrolled). So a factor 15x between naive C++ and code generated files. In the case of this function in particular, we have 4 nested loops, it's branching hell where continues are hit very often.
paulddraper 2y ago
Spoken like a Go programmer :D That introduces some other tool (with its own syntax), an extra build step, possible non-composability with other features. A preprocessor (for good or bad) IS a code generation step.
rogual 2y ago
This is what I do, these days. Whenever I would previously have reached for X-macros or some other macro hack, I tend to use Cog [1] now instead. It's quite a clever design; you write Python to generate your C++ code and put it inside a comment. Then when you run the Cog tool on your source file, it writes the generated code directly into your C++ file right after your comment (and before a matching "end" comment). This is great because you don't need Cog itself to build your project, and your IDE still understands your C++ code. I've also got used to being able to see the results of my code generation, and going back to normal macros feels a bit like fiddling around in the dark now. [1] https://nedbatchelder.com/code/cog/ https://nedbatchelder.com/code/cog/
surajrmal 2y ago
I wonder if c++26 reflection can manage to replace the need for this pattern with something more robust.
yason 2y ago
Using the C preprocessor is standard, available, compatible and the major usage patterns are "known". For a lot of cases, they're way easier to reason about rather than learning how an external generation tool is used to generate the code. In order to understand these macros all I need is to read the source code where they're used. Nothing C++ related in the pattern though. This C preprocessor trickery is practically so classic you couldn't necessarily even call it a "trick".

dataflow 2y ago

External codegen introduces a lot of friction in random places. Like how your editor can no longer understand the file before you start building. Or how it can go out of date with respect to the rest of your code until you build. If you can do it with a macro it tends to work better than codegen in some ways.

writebetterc 2y ago

IDEs understand preprocessor macros, so IDE features (jump2def, etc) work with this. IDEs also can expand the macro invocations. So, I prefer the macros when possible :-).

deleted 2y ago
[deleted]
Someone 2y ago
> IDEs understand preprocessor macros, so IDE features (jump2def, etc) work with this. Do they? X macros often are used with token pasting (https://gcc.gnu.org/onlinedocs/cpp/Concatenation.html https://gcc.gnu.org/onlinedocs/cpp/Concatenation.html), as for example in (FTA) #define AST_BEGIN_SUBCLASSES(NAME) START_##NAME , Are modern IDEs/compiler toolchains smart enough to tell you that START_foo was created by an expansion of that macro?
- jcelerier 2y ago
  any non-toy IDE can do that. Most IDEs use clang directly for parsing nowadays.
- tom_ 2y ago
  Yes. I use this with VS2019 for generating enum names, and they interact fine with auto complete and go to definition.

wat10000 2y ago

Now you have a additional stage in your build, a bunch of new code to maintain, and either a bespoke language embedded in your standard C++ or a bunch of code emitting C++ separately from the code it logically belongs with. Compare with a solution that's 100% standard C++, integrates into your build with zero work, can be immediately understood by anyone reasonably skilled in the language, and puts the "generated" code right where it belongs.

MITSardine 2y ago
CMake makes this pretty painless. My codegen targets have only two additional instructions to handle the generation itself and dependencies: add_custom_command to call the codegen exec, and then add_custom_target to wrap my outputs in a "virtual" target I can then make the rest of my program depend on, but this is just for tidying up. And I'll dispute the fact that any complex C prepro task "can be immediately understood by anyone reasonably skilled in the language". Besides, code should ideally be understood by "anyone reasonably likely to look at this code to work in it", not "reasonably skilled".
- wat10000 2y ago
  This isn't complex. It's a bit unusual, but not hard to understand if you understand the basics of how #include and #define work. If you're working on the sort of C++ codebase that would benefit from this sort of code generation, and you're not reasonably skilled in C++, then god help you.
  - MITSardine 2y ago
    Are you talking about the X macro itself, or more generally? I may be the obtuse one here, but for a more complex example, it took me a few hours to manage to make nested loops using Boost PP (for explicit instantiations). Even so, I avoid having to write a new one that's not a quick copy-paste because it's quite different from usual C++ programming, so my painfully acquired understanding quickly evaporated... as I suspect is the case of anyone who doesn't particularly focus on the C prepro. In the end, it's just simpler to get some Python script or C++ program to write a string and dump that to a file than to write something illegible with the C preprocessor, if doing something at all complicated (in my opinion).

jasonthorsness 2y ago

The C# "source generator" approach is a good compromise; it runs within the build chain so has the ease-of-use of macros in that respect, but they don't need to be written in a weird macro language (they are C# or can call external tool) and when you debug your program, you debug through the generated source and can see it, more accessible than macros. Not sure if there is something similar in C/C++ integrated with the common toolchains. But when working outside C/C++ I've found myself missing the flexibility of macros more times than I can count.

TeMPOraL 2y ago
> But when working outside C/C++ I've found myself missing the flexibility of macros more times than I can count. Me to, and that's even in Lisp! Preprocessor macros are hard and bugprone because they share the failings of Unix philosophy of "text as universal interface" - you're playing with unstructured (or semi-structured) pieces of text, devoid of all semantics. And this is also what makes them occasionally useful - some code transformations are much, much easier to do when you can manipulate the text form directly, ignoring syntax and grammar and everything. Only the final value must be correct code - starting point and intermediary values can be anything, and you don't need to make sure you can get from here to there through valid data transformations. This is a really powerful capability to have. (I also explicitly compared preprocessor macros to "plaintext everything" experience that's seen as divine wisdom, to say: y'all are slinging unstructured text down the pipes way too much, and using preprocessor way too little.)

mauvehaus 2y ago

I've done this in C with the C preprocessor and Java with m4[0]. The upside of doing it natively is that it keeps the build simpler. And everybody at least knows about the existence of the C preprocessor, even if they don't know it well. And it's fairly limited, which prevents you from getting too clever. The big downside of doing it with the C preprocessor is that the resulting code looks like vomit if it's more than a line or two because of the lack of line breaks in the generated code. Debugging it is unenjoyable. I'd recommend against doing anything super clever. The upside of doing it out of band is that your generated source files look decent. m4 tends to introduce a little extra whitespace, but it's nothing objectionable. Plus you get more power if you really need it. The downside is that almost nobody knows m4[1]. If you choose something else, it becomes a question of what, does anyone else know it, and is it available everywhere you need to build. Honestly, integrating m4 into the build in ant really wasn't too bad. We were building on one OS on two different architectures. For anything truly cross-platform, you'll likely run into all the usual issues. ETA: Getting an IDE to understand the out of band generation might be a hassle, as other folks have mentioned. I'm a vim kinda guy for most coding, and doing it either way was pretty frictionless. The generated java code was read-only and trivial, so there wasn't a lot of reason to ever look at it. By the time you get to debugging, it would entirely transparent because you're just looking at another set of java files. [0] This was so long ago, I no longer remember why it seemed like a good idea. I think there was an interface, a trivial implementation, and some other thing? Maybe something JNI-related? At least at first, things were changing often enough that I didn't want to have to keep three things in sync by hand. [1] Including me. I re-learn just enough to get done with the job at hand every time I need it.

drwu 2y ago

Another issue is cross-compiling. External code generation requires (cross) execution of the (cross) compiled binary program.

rcxdude 2y ago
Or to build the generating code for the host instead. Most build systems that support cross-compilation can do this, except CMake.

MITSardine 2y ago

After trying to wrangle Boost PP and other advertised compile-time libraries such as Boost Hana (which still has some runtime overhead compared to the same logic with hardcoded values), I've finally converged to simply writing C++ files that write other C++ files. Could be Python, but I rather keep the build simple in my C++ project. Code generation is painless with CMake, no idea with other build configuration utilities.

cbuq 2y ago
This sounds pragmatic, but are you writing C++ executables that when run create the generated code? Are there templating libraries involved?
- MITSardine 2y ago
  Yeah, it's all done automatically when you build, and dependencies are properly taken into account: if you modify one of the code generating sources, its outputs are regenerated, and everything that depends on them is correctly recompiled. This doesn't take much CMake logic at all to make work. In my case, no, it's dumb old code writing strings and dumping that to files. You could do whatever you want in there, it's just a program that writes source files. I do use some template metaprogramming where it's practical versus code generation, and Boost Hana provides some algorithmic facilities at compile time but those incur some runtime cost. For instance, you can write a while loop with bounds evaluated at compile time, that lets you use its index as a template parameter or evaluate constexpr functions on. But sometimes the best solution has been (for me, performance/complexity wise) to just write dumb files that hardcode things for different cases.
Arech 2y ago
> Boost Hana (which still has some runtime overhead compared to the same logic with hardcoded values) Can you elaborate on that? What was your use-case for which this was true?
- MITSardine 2y ago
  One case I benchmarked was Bernstein/Bézier and Lagrange element evaluation. This is: given a degree d triangle or tetrahedron, given some barycentric coordinates, get the physical coordinate and the Jacobian matrix of the mapping. Degree 2, Lagrange: - runtime: 3.6M/s - Hana: 16.2M/s - Hardcoded: 37.7M/s Degree 3, Lagrange: 2.6M/s, 6.4M/s, 13.4M/s (same order). "Runtime" here means everything is done using runtime loops, "Hana" using Boost Hana to make loops compile-time and use some constexpr ordering arrays, "hardcoded" is a very Fortran-looking function with all hardcoded indices and operations all unrolled. As you see, using Boost Hana does bring about some improvement, but there is still a factor 2x between that and hardcoded. This is all compiled with Release optimization flags. Technically, the Hana implementation is doing the same operations in the same order as the hardcoded version, all indices known at compile time, which is why I say there must be some runtime overhead to using hana::while. In the case of Bernstein elements, the best solution is to use de Casteljau's recursive algorithm using templates (10x to 15x speedup to runtime recursive depending on degree). But not everything recasts itself nicely as a recursive algorithm, or I didn't find the way for Lagrange anyways. I did enable flto as, from my understanding (looking at call stacks), hana::while creates lambda functions, so perhaps a simple function optimization becomes a cross-unit affair if it calls hana::while. (speculating) Similar results to compute Bernstein coefficients of the Jacobian matrix determinant of a Q2 tetrahedron, factor 5x from "runtime" to "hana" (only difference is for loops become hana::whiles), factor 3x from "hana" to "hardcoded" (the loops are unrolled). So a factor 15x between naive C++ and code generated files. In the case of this function in particular, we have 4 nested loops, it's branching hell where continues are hit very often.

paulddraper 2y ago

Spoken like a Go programmer :D That introduces some other tool (with its own syntax), an extra build step, possible non-composability with other features. A preprocessor (for good or bad) IS a code generation step.

rogual 2y ago

This is what I do, these days. Whenever I would previously have reached for X-macros or some other macro hack, I tend to use Cog [1] now instead. It's quite a clever design; you write Python to generate your C++ code and put it inside a comment. Then when you run the Cog tool on your source file, it writes the generated code directly into your C++ file right after your comment (and before a matching "end" comment). This is great because you don't need Cog itself to build your project, and your IDE still understands your C++ code. I've also got used to being able to see the results of my code generation, and going back to normal macros feels a bit like fiddling around in the dark now. [1] https://nedbatchelder.com/code/cog/ https://nedbatchelder.com/code/cog/

surajrmal 2y ago

I wonder if c++26 reflection can manage to replace the need for this pattern with something more robust.

yason 2y ago

Using the C preprocessor is standard, available, compatible and the major usage patterns are "known". For a lot of cases, they're way easier to reason about rather than learning how an external generation tool is used to generate the code. In order to understand these macros all I need is to read the source code where they're used. Nothing C++ related in the pattern though. This C preprocessor trickery is practically so classic you couldn't necessarily even call it a "trick".