4 ms·

Compiling C++ with the Clang API

103 points by ingve 2y ago

fooblaster 2y ago
This is pretty cool. What are the use cases?
- wdpk 2y ago
  compile c++ code at runtime for instance. Lots of use cases, most obvious one specializing/instantiating dense computational kernels on values only known at runtime... but so many more things would be possible if the compiler was just a reusable library.
  - ska 2y ago
    This is something Common Lisp got really right.
    - wdpk 2y ago
      well being homoiconic and dynamic helps quite a bit... This being said, if you squint a bit and get used to the syntax, c++ variadic templates are just a compile-time lisp (really templates are just generalized functions over types) and the template mechanism is 100% pure, with a runtime capability of evaluating those pure monadic computational effects defined at compile-time to runtime, there is no more boundary (not saying it's a thing that should be done all the time). The main advantage then over functional languages is the fact that c++ optimizing compilers are already pretty good at optimization so assuming that you can afford to re-compile at runtime the tight inner loops or critical paths (say at "configuration time" when adding some latency might not be a big deal), a lot of otherwise impossible optimizations could probably be done better (thinking of loop invariants, polyhedral, unrolling, constant propagation, aliasing, row major to column major etc etc) probably the result would also be better than what a JIT compiler and profiler would be able to achieve too.
      - gpderetta 2y ago
        > template mechanism is 100% pure Let's say 99.99%. You underestimate the resourcefulness of C++ (meta-)programmers and their ability to leverage implementations or specification bugs for "useful" purposes: https://b.atch.se/posts/non-constant-constant-expressions/ https://b.atch.se/posts/non-constant-constant-expressions/
- yuserx 2y ago
  ~22MB download for a full IDE https://www.kvraudio.com/forum/viewtopic.php?t=560622 https://www.kvraudio.com/forum/viewtopic.php?t=560622
- menaerus 2y ago
  REPL and JIT come to mind.
- secondcoming 2y ago
  I’ve wanted something like this to compile XML business rules to native code on the fly. I would have a script that converted the XML to C and then turn that into machine code that I’d load directly from the .text section of the binary (shellcode style). Turns out this is error prone because compilers can emit things like jump tables into the .rodata section, so you need that too. It’s easier to just create a shared library.
  - high_na_euv 2y ago
    XML business rules? Is it xml-oriented-programming again? They were doing it in Java 25 years ago
- mshockwave 2y ago
  I think this is exactly how Zig compiler does under the hood for C/C++ sources. So I guess you can do a similar thing for your own programming languages to support interop.
deleted 2y ago
[deleted]
t43562 2y ago
What I always wanted from Clang and was not smart enough to understand was how to use it to get an AST or similar so that I could rewrite code. I was working on a big project with old code that needed a radical overhaul to make it ready for new boost and new C++ versions. It seemed a huge task to do this accurately by hand. I also thought about combining this with some static analysis of a trivial kind - looking for behaviours that are not wrong in C++ overall but were wrong in the context of that project. I know there are several tools for clang that do rewrites and the unfortunate problem was that clang at the time wasn't able to build that code for some reasons that I cannot remember now. The rewriting tools that were available were not very sophisticated from my memory and made it difficult to do the kinds of changes that I thought would be needed. So I'm not so interested in being able to compile but being able to parse and then walk the parse tree in some very easy language like python. I think the answer now might be "libclang": https://pypi.org/project/libclang/ https://pypi.org/project/libclang/
- menaerus 2y ago
  libclang has been with us for ~15 years already. > I'm not so interested in being able to compile but being able to parse Chances are that you're not going to be able to walk the AST without getting errors unless the (lib)clang can cleanly parse the code. There are at least two prerequisites for that: (1) marrying your build system with the (lib)clang so that you pass the exact include directories, include paths, build flags, etc. for any given translation unit, and (2) ability to build the code with clang. If you use CMake then it's going to be a little bit easier. If not, I recommend first switching to it. You will also have to be somewhat creative about making it possible to correctly parse the code in the header files - a header file fundamentally isn't a translation-unit. Back in the days, I wrote some of the utilities built on top of the libclang and the process was very hairy. Although I pulled it off, for the reasons above it required a lot of unsexy work exactly around the integration of (CMake) build system, patching the codebase itself, and working around the design limitations imposed by the language itself. Occasionally, libclang wouldn't offer me enough control over AST so I also had to work around those limitations by patching the libclang code itself. libtooling seemingly gives more control but I have never tried it. If you still want to try it, I think your best bet is to find something similar that already exists and build upon it.
izaro 2y ago
[flagged]
esbranson 2y ago
The Roslyn APIs are the .NET equivalent: https://learn.microsoft.com/en-us/dotnet/csharp/roslyn-sdk/compiler-api-model https://learn.microsoft.com/en-us/dotnet/csharp/roslyn-sdk/c...
- HdS84 2y ago
  MS best idea was to create Source Generators. Absurdly powerful for some problems.
  - pjmlp 2y ago
    They still miss something like good old T4 templates though.
    - HdS84 2y ago
      Yes, that is missing. I thought about creating something like that, but it's a lot of work. Also, the huge number of source generators have a problem: a SG cannot see code generated by a another sg. There is no ability to stack them. That will be a problem as more and more sg's generate source. E.g. I have a SG which creates interfaces. No other sg can use these interfaces.
tway223 2y ago
I have been wondering if someone could improve golang's cgo infra using clang like what zig is doing..

103 points by ingve 2y ago

fooblaster 2y ago

This is pretty cool. What are the use cases?

wdpk 2y ago
compile c++ code at runtime for instance. Lots of use cases, most obvious one specializing/instantiating dense computational kernels on values only known at runtime... but so many more things would be possible if the compiler was just a reusable library.
- ska 2y ago
  This is something Common Lisp got really right.
  - wdpk 2y ago
    well being homoiconic and dynamic helps quite a bit... This being said, if you squint a bit and get used to the syntax, c++ variadic templates are just a compile-time lisp (really templates are just generalized functions over types) and the template mechanism is 100% pure, with a runtime capability of evaluating those pure monadic computational effects defined at compile-time to runtime, there is no more boundary (not saying it's a thing that should be done all the time). The main advantage then over functional languages is the fact that c++ optimizing compilers are already pretty good at optimization so assuming that you can afford to re-compile at runtime the tight inner loops or critical paths (say at "configuration time" when adding some latency might not be a big deal), a lot of otherwise impossible optimizations could probably be done better (thinking of loop invariants, polyhedral, unrolling, constant propagation, aliasing, row major to column major etc etc) probably the result would also be better than what a JIT compiler and profiler would be able to achieve too.
    - gpderetta 2y ago
      > template mechanism is 100% pure Let's say 99.99%. You underestimate the resourcefulness of C++ (meta-)programmers and their ability to leverage implementations or specification bugs for "useful" purposes: https://b.atch.se/posts/non-constant-constant-expressions/ https://b.atch.se/posts/non-constant-constant-expressions/
yuserx 2y ago
~22MB download for a full IDE https://www.kvraudio.com/forum/viewtopic.php?t=560622 https://www.kvraudio.com/forum/viewtopic.php?t=560622
menaerus 2y ago
REPL and JIT come to mind.
secondcoming 2y ago
I’ve wanted something like this to compile XML business rules to native code on the fly. I would have a script that converted the XML to C and then turn that into machine code that I’d load directly from the .text section of the binary (shellcode style). Turns out this is error prone because compilers can emit things like jump tables into the .rodata section, so you need that too. It’s easier to just create a shared library.
- high_na_euv 2y ago
  XML business rules? Is it xml-oriented-programming again? They were doing it in Java 25 years ago
mshockwave 2y ago
I think this is exactly how Zig compiler does under the hood for C/C++ sources. So I guess you can do a similar thing for your own programming languages to support interop.

deleted 2y ago

[deleted]

t43562 2y ago

What I always wanted from Clang and was not smart enough to understand was how to use it to get an AST or similar so that I could rewrite code. I was working on a big project with old code that needed a radical overhaul to make it ready for new boost and new C++ versions. It seemed a huge task to do this accurately by hand. I also thought about combining this with some static analysis of a trivial kind - looking for behaviours that are not wrong in C++ overall but were wrong in the context of that project. I know there are several tools for clang that do rewrites and the unfortunate problem was that clang at the time wasn't able to build that code for some reasons that I cannot remember now. The rewriting tools that were available were not very sophisticated from my memory and made it difficult to do the kinds of changes that I thought would be needed. So I'm not so interested in being able to compile but being able to parse and then walk the parse tree in some very easy language like python. I think the answer now might be "libclang": https://pypi.org/project/libclang/ https://pypi.org/project/libclang/

menaerus 2y ago
libclang has been with us for ~15 years already. > I'm not so interested in being able to compile but being able to parse Chances are that you're not going to be able to walk the AST without getting errors unless the (lib)clang can cleanly parse the code. There are at least two prerequisites for that: (1) marrying your build system with the (lib)clang so that you pass the exact include directories, include paths, build flags, etc. for any given translation unit, and (2) ability to build the code with clang. If you use CMake then it's going to be a little bit easier. If not, I recommend first switching to it. You will also have to be somewhat creative about making it possible to correctly parse the code in the header files - a header file fundamentally isn't a translation-unit. Back in the days, I wrote some of the utilities built on top of the libclang and the process was very hairy. Although I pulled it off, for the reasons above it required a lot of unsexy work exactly around the integration of (CMake) build system, patching the codebase itself, and working around the design limitations imposed by the language itself. Occasionally, libclang wouldn't offer me enough control over AST so I also had to work around those limitations by patching the libclang code itself. libtooling seemingly gives more control but I have never tried it. If you still want to try it, I think your best bet is to find something similar that already exists and build upon it.

izaro 2y ago

[flagged]

esbranson 2y ago

The Roslyn APIs are the .NET equivalent: https://learn.microsoft.com/en-us/dotnet/csharp/roslyn-sdk/compiler-api-model https://learn.microsoft.com/en-us/dotnet/csharp/roslyn-sdk/c...

HdS84 2y ago
MS best idea was to create Source Generators. Absurdly powerful for some problems.
- pjmlp 2y ago
  They still miss something like good old T4 templates though.
  - HdS84 2y ago
    Yes, that is missing. I thought about creating something like that, but it's a lot of work. Also, the huge number of source generators have a problem: a SG cannot see code generated by a another sg. There is no ability to stack them. That will be a problem as more and more sg's generate source. E.g. I have a SG which creates interfaces. No other sg can use these interfaces.

tway223 2y ago

I have been wondering if someone could improve golang's cgo infra using clang like what zig is doing..