13 ms·
Circle – A C++ compiler with compile-time imperative metaprogramming
- umvi 6y agoI've just started storing metadata in JSON format. Then, I have a cmake target that invokes a python program that reads in the JSON and generates .h/.cpp files which can then be consumed by other targets. The advantage of this is that I can then use the same metadata JSON to generate documentation, or whatever I want.
- reikonomusha 6y agoHave you learned about Lisp macros yet? If you’re intrigued by this idea, Lisp macros are a few steps above in terms of functionality, integration, etc.
- TylerE 6y agoOr nim, which is even more powerful
- discardable_dan 6y agoFirst, can you talk a little about how Nim's macro system is more-powerful than Lisp's? That seems like a fairly broad claim, and unless it allows you to influence the compiler at compile time (and maybe even then), I am curious how that works out. Second, one of the biggest issues with lisp is that macros end up sort of slow, especially when you're using them + lists to replicate ADTs with pattern matching (versus, say, ML, which can do very fast dispatch based on types). Doesn't Nim fall into that same trap?
- TylerE 6y agoIt can do absolutely everything lisp macros can do, with more flexible input syntax, and zero runtime overhead, while being both strongly typed and giving the macro body access to the types (PS: Lisp fanboys downvoting this, how about attempting to make a counterargument instead of just hitting the downvote button?)
- emidln 6y agoLisp macros have runtime overhead? I don't think reader macros or compile macros have any runtime overhead unless you add it yourself. More flexible input syntax than a programmable reader? Data like sexps and edn is pretty flexible to begin with, but a lot of lisps have fully programmable readers for arbitrary syntax. Some lisps even support grammar dialects to switch out of sexps entirely (such as racket).
- TylerE 6y agoNim macros operate directly on the ast while providing flexible user friendly (read: not sexp) calling syntax. Everything is fully typed, and the types are available to the macro expansion. Absurdly powerful things have been done, for instance https://github.com/krux02/opengl-sandbox https://github.com/krux02/opengl-sandbox which crosscompiles nim to GPU shaders.
- ori_b 6y agoIt's like you didn't read the previous poster: They're claiming lisp macros can work BEFORE the compiler generates an AST.
- TylerE 6y agoYes, it's the same in nim. For instance, see this example which defines a compact definition syntax that is certainly not valid vanilla nim: https://peterme.net/metaprogramming-and-read-and-maintainability-in-nim.html https://peterme.net/metaprogramming-and-read-and-maintainabi... Refer to lines 63-88 of the large final example.
- ori_b 6y agoAnd if I wanted to make 'for' into a valid variable name, or change the tokens for specifying numbers, for example, how would I go about doing that?
- umvi 6y agoAre you talking about using Lisp macros to generate C/C++/documentation? I'm not sure how Lisp macros would be better than static metadata files being operated on by some high level scripting language.
- reikonomusha 6y agoThe author created a DSL which generates code. Macros are another way to do that.
- acchow 6y agoSo your template metaprogramming is itself generated?
- umvi 6y agoI'm guessing this is a joke, but in case it's not - sure, technically you could generate template metaprogramming with an additional layer of metaprogramming as I've described. But since you already have metaprogramming at the higher level, there is not as much need for template metaprogramming IMO
- kbenson 6y agoIt makes my eyes bleed, but so does C++ by itself to a slight degree. I imagine those that like C++ (or are forced to use it) might find it very interesting. I wonder how this compares to hygienic macros that some languages support?
- FpUser 6y agoI use C++ by choice for some projects. Still I think C++ is enough beast on its own. No need to add another preprocessor
- spekcular 6y ago"Your scientists were so preoccupied with whether or not they could, they didn’t stop to think if they should."
- gonzo41 6y agoOh good, C++ is more complex, how will i ever live without this extra complexity.
- banachtarski 6y agoCircle is honestly a breath of fresh air and much more palatable in terms of usability/speed than the existing constexpr facilities or proposed reflection IMO.
- dexter0 6y agoThis looks amazing, but the fact that it is implemented in an entirely new compiler (atop LLVM, sure) is a major turn-off. Would be more useful if it were implemented as a meta compiler instead.
- orbifold 6y agoThere is basically no way to modify clang or GCC to support executing arbitrary C++ code at compile time. Something like that would be a fork that wouldn’t get accepted upstream. According to the author the compiler front end is ~100kLoc which is an order of magnitude less than what clang is.
- pizlonator 6y agoThis is so wonderfully excellent. I can't wait to use it.
- halayli 6y agoI am not sure if it's possible to implement this as pluggable pass to llvm, but it would widen adoption.
- jhj 6y agoThe backend is LLVM. Sean wrote his own C++17 compatible frontend to provide this functionality (thus, Circle is effectively a replacement for clang).
- amiga_500 6y agoOne can only imagine the error messages from the c++ layer after a semi-botched circle compile!
- klyrs 6y agoI'm not sure about other compilers, but I use gcc. Mild errors in templates can produce reams of errors, the first of which might have a good hint, and if not, the actual error is tucked away in, perhaps, the third attempted template substitution. This is a very valid concern. Good error messages are hard to produce, and gcc doesn't even try to figure out which brackets are mismatched. Mix that in with a supercharged comptime constructions, and I'm quite terrified. I don't have these issues with macros anymore, thanks to templates, but I nearly prefer them.
- orbifold 6y agoThere is no separate C++ layer and the nice thing is that you can just use exceptions at compile time to signal errors. So the errors are actually much nicer than with template meta programming.
- lasagnaphil 6y agoActually the error messages are much nicer than you've thought. Here's an example: $ circle taco4.cxx -I /home/sean/projects/opensource/taco/include -M /home/sean/projects/opensource/taco/Debug/lib/libtaco.so -filetype=ll Creating compute_1 function macro expansion: taco4.cxx:428:3 error in expansion of macro auto call_taco(const char*, const options_t&) code injection: taco4.cxx:420:6 error thrown in @expression("c") error: @expression:1:1 undeclared identifier 'c' A well-defined meta language would probably have way better error messages than an accidentally-discovered language based on insane template hacks.
- nickysielicki 6y agoThis is where C++ is headed anyway. There's just a little bit more work that's needed in c++23, but it's planned, and that will make it possible to use a lot more of the STL in consteval/constexpr contexts. I think this is a killer feature. Between execution policies and constexpr-all-the-things, I really think C++ is set to pull away in a way that other languages will have difficulty competing with.
- closeparen 6y agoHow does this differ from code generation?
- humanrebar 6y agoIt's portable and doesn't add any new build dependencies. No need to model extra phases in your build system either.
- closeparen 6y agoIt would seem that most mainstream languages other than C/C++ have already bit the bullet on fancy build systems, due to dependency management, so codegen is not that expensive.
- bhawks 6y agoAny code you compile can now execute arbitrary syscalls, shell out to git, interact with the user's terminal, host a web service or whatever else as you build it and then write the results to your final executable code? While most ecosystems are moving to hermetic, reproducible builds this is going full steam on the other direction. Goodbye 'it works on my machine' hello 'it works on Jane's machine when Ryan builds it on Tuesdays. Oh Ryan WFH on Tuesday so it gets built on his home machine!' I wanted to like this but it seems to have gone way too far. It is a college project gone wild.
- Reelin 6y agoIf you do any codegen can't all of that already happen though? Many (most?) build systems can already execute more or less arbitrary logic. Reproducible builds are only guaranteed if you actually verify the signature of the output. There's also runtime reflection in most dynamic languages, and don't forget self modifying code! No technical restriction can prevent people from writing bad code at the end of the day because bad code is really a social issue.
- jcalabro 6y agoReminds me of the Jai language, in development now: https://www.youtube.com/watch?v=TH9VCN6UkyQ&list=PLmV5I2fxaiCKfxMBrNsU1kgKJXD3PkyxO https://www.youtube.com/watch?v=TH9VCN6UkyQ&list=PLmV5I2fxai...
- symstym 6y agoThis seems similar to Zig (https://ziglang.org/#Compile-time-reflection-and-compile-time-code-execution https://ziglang.org/#Compile-time-reflection-and-compile-tim...)
- pickdenis 6y agoZig seems to have arbitrary compile-time code evaluation, but not the kind of AST generation you see here[1]. Nim macros seem to be a closer analogue[2]. [1]: https://github.com/seanbaxter/circle/blob/master/gems/rpn.md#lower-the-rpn-ast https://github.com/seanbaxter/circle/blob/master/gems/rpn.md... [2]: https://nim-lang.org/docs/macros.html https://nim-lang.org/docs/macros.html
- logicchains 6y agoThere's ongoing work to add support for generating arbitary types at compile time: https://github.com/ziglang/zig/issues/383 https://github.com/ziglang/zig/issues/383.
- thechao 6y agoNow add 'operator ;', 'operator if(...)', 'operator .', et al and we've got ourselves a Lisp!
- Reelin 6y agoPersonally I'll just use ECL until CLASP gets its optimizations sorted out.
- mckinney 6y agoReminds me of Manifold[1] for Java. But Manifold integrates seamlessly with the Java compiler as a plugin as opposed to being a separate compiler, which makes Circle less approachable imo. Still, I like the F# type provider-ish aspect of this. [1]: https://github.com/manifold-systems/manifold https://github.com/manifold-systems/manifold
- tpoindex 6y agoThanks for pointing out Manifold, looks delicious. As a historical note, PL/I had a preprocessor that itself was a subset of PL/I. I don't recall any type safety features, but was quite useful for creating DSLs within a program.
- kotrunga 6y agoRegardless of how great the stuff in the repo is (I'll check it out)... this is amazing: "it's like c++ that went away to train with the league of shadows and came back 15 years later and became batman"
- smt88 6y agoI'm going to disagree here. It reminds me of the "Python Ninja"-type job titles that became popular ~2012. It's so preoccupied with standing out and breaking the formality of its medium that it actually says nothing. The repo description literally says two things: Circle is a C++ automation language, and Circle trained with the League of Shadows. The rest requires clicks. That's a terrible way to drive adoption for something that obviously was difficult to build. Sometimes being earnest and straightforward is the best way to break down skepticism, not by using meaningless hipster taglines.
- rogerclark 6y agothere's about a zillion pages of detailed documentation on top of that
- smt88 6y agoThe intro text is not detailed enough to tell me whether I want to click on any of those. How hard is it to write a one-sentence explanation of what a project is that doesn't involve ninjas?
- asdf_snar 6y agoFrom the abstract, "Circle is a compiler that extends C++17 for new introspection, reflection and compile-time execution. An integrated interpreter allows the developer to execute any function during source translation. Compile-time flavors of control flow statements assist in generating software programmatically. The configuration-oriented programming paradigm captures some of the conveniences of dynamic languages in a statically-typed environment, helping separate a programmer's high-level goals from the low-level code and allowing teams to more effectively reason about their software." While I appreciate your qualm, respectfully I think that no matter how it were introduced, someone would've quibbled. "You should be straightforward", "you should be more detailed", "you should have more ninjas". There's no win. The solution is to hope the readers you care about will be adventurous enough to make two clicks.
- fredsanford 6y agoIs it named Circle because you'll be running in a circle trying to figure out the error messages?
- inetknght 6y agoI thought error messages related to C++ templates were unfathomable. Then I started using Docker.
- jzelinskie 6y agoJokes like this are common now, but, as an outsider, I think it's crazy that the C++ community says stuff like this as if they weren't dealing with template errors and slowly making everything work under constexpr.
- stephc_int13 6y agoMetaprogramming is very alluring on the surface, we've all been frustrated by the limitations of our languages of choice at some point or another. But, I think this trend might lead to extremely hard to read code, and there is a good chance that this hard-to-read code will be treated as some black box/voodoo. It might not be a better idea than self-modifying machine code or some of the wildest C macros...
- psychoslave 6y agoYou can write non maintainable code at any level of abstraction. How hard it will be to debug is more dependent on the available tools to link the error to some source code. For example, there's a huge load of source to source compilers used in the web stack now. This is not such a big deal it seems, probably because debuggers make an adequate job to link the error to the original source. Actually I didn't directly touched Babylscript and so on, but I didn't saw much complaint about traceability of errors, so here I just guess: a more informed point of view would be valuable here. Self-modifying code might have been its purpose in highly resource constrained environment. But otherwise, in my opinion, generating a whole distinct source or tailoring a runnable AST has always been more understandable while offering the same level of flexibility.
- logicchains 6y ago>But, I think this trend might lead to extremely hard to read code, and there is a good chance that this hard-to-read code will be treated as some black box/voodoo. This already happens with metaprogramming in C++. See e.g. the source of the Boost Preprocessor library. Circle just makes this code a bit easier to read. Even code using the library: compare the compile-time Duff's device generation in https://www.boost.org/doc/libs/1_72_0/libs/preprocessor/doc/examples/duffs_device.c https://www.boost.org/doc/libs/1_72_0/libs/preprocessor/doc/... to that in https://github.com/seanbaxter/circle/blob/master/examples/README.md#duffs-device https://github.com/seanbaxter/circle/blob/master/examples/RE... .
- MauranKilom 6y agoI disagree with using Boost Preprocessor C code as an example for C++ metaprogramming.
- yrashk 6y agoLove the level and amount of documentation/tutorials provided by the author! Also, wondering when and if the source code of Circle itself will be available.
- yzh 6y agoSean is a very good engineer. Another of his work is moderngpu, a GPU primitive library. It has the same level of doc/tutorials. I really like one thing he wrote: Software is an asset, code a liability. On good days I'd add 200 or 300 lines to this repository. On great days I'd subtract 500.
- fuckface123 6y agoisn't C++ with compile-time imperative metaprogramming now called... C++?
- gerash 6y agoI'm all for doing things at compile time rather than run-time but both the syntax and debugability of CPP template metaprogramming suck. I wonder if C++ should pull a Python 3 and clean up the syntax. Given that the transition for Py3 took more than a decade maybe they should do it in small but frequent language upgrades instead.
- adrianN 6y agoC and C++ live on backwards compatibility. There are many decades old codebases that are still being worked on. You can't clean up the syntax if that breaks backwards compatibility, you'd just end up creating a new language instead.
- gerash 6y agoIMO too much backwards compatibility hinders progress. It's obviously not sustainable to be forever backwards compatible all the way to the 70s. I'd say old code bases can simply stick with the older compiler. As for whether it'd be considered a new language or not, it depends. I wouldn't consider Py3 a new language even though it's not backwards compatible.
- sesuximo 6y agoGeneral software tip: don't make a new version if you'll have to maintain the old and the new. It gets annoying real quick. much easier to tell your users to update and be reasonable with the changes I actually don't see a problem keeping the old syntax alive. I do see a problem with new programmers using obsolete constructs
- gpderetta 6y agothe C++ committee is willing to break backward compatibility in isolated areas if it is really worth paying the price, but it will (thankfully) never pull a Python3.
- rurban 6y agoThe syntax choices are for one much better than the C++ committee ideas. Much less eyebleed.
- sesuximo 6y agoReally? I think fold statements are really nice. So often you get what you want by just writing `...`
- clord 6y agoThe @include thing makes me wonder if a circle library could compete with modules.
- orbifold 6y agoI think the author mentioned that he plans to support C++-modules as is. He already supports concepts.
- jhallenworld 6y agoVerilog and VHDL have something akin to this form of imperative meta-programming. (See Verilog's "generate" and "genvar" keywords for example). But there is no iterating over or building members of an interface (the closest thing SystemVerilog has to a user defined type).
- beshrkayali 6y agoOne of the reasons to look into Nim. Nim has generics, templates and macors. Macros are procedural, and they make things like generating a dsl for your software or library a simple matter. All with a reasonably simple pythonic syntax.
- Zardoz84 6y agoI would keep using D ...
- slezyr 6y agotemplate<typename... types_t> struct ??? { @meta for(int i = 0; i < sizeof...(types_t); ++i) types_t...[i] @(i); }; Oh no.
- torb-xyz 6y agoThere's a pretty interesting podcast interview with the author here: https://cppcast.com/circle-language/ https://cppcast.com/circle-language/ I find the idea of using this as a rapid testing and exploration of potential new C++ features quite interesting.
- msoloviev 6y agoThis looks like a more mature/different approach to the same thing I tried to do in a side project a while ago (https://github.com/blackhole89/macros https://github.com/blackhole89/macros), down to similar aesthetic choices. However, I pitch my project as a macro system rather than compile-time execution. While here the metalanguage is itself a C++-lookalike, in my project I wound up with something that is perhaps better described as a weird TeX (which seems to be what happens naturally when your machine model is based on binding and substitution). On one hand, C++ is surely more powerful/expressive and there is elegance to having the language and metalanguage use the same idiom. On the other, I feel like this might actually making Circle a bit more confusing to use at times, as it becomes less clear exactly when what parts of the code are executed, and you could even imagine a typo accidentally lifting a part of the code from runtime to compile time, resulting in mystifying bugs. (I am myself puzzled by the "serialize structs generically" example: does it imply that the template is specialised before the @meta for is executed?)
- einpoklum 6y agoWhat I miss in C++ isn't more ability to _run_ things at compile-time, but rather more ability to _compile_ things at run-time! I want to finalize my optimizations with run-time-available information - and to make run-time-definable compositions of pieces of code. Something like this: https://blog.quarkslab.com/easyjit-just-in-time-compilation-for-c.html https://blog.quarkslab.com/easyjit-just-in-time-compilation-... but better still.