19 ms·
Warp, a fast preprocessor for C and C++
- jevinskie 13y agoAre precompiled headers and the performance increase from switching to clang[0] not enough? [0]: http://clang.llvm.org/features.html#performance http://clang.llvm.org/features.html#performance
- lbrandy 13y ago> Is X not enough? Nope, it's not, for all X.
- Danieru 13y agoError: Type mis-match near "all". Usage of symbol X as type 'Situation' in line 2 does not match usage in line 1 as 'Solution'.
- pjmlp 13y agoSurely not. I know of a project that takes around 3 hours to build with Clang when you do a "make all".
- deadfa11 13y agoSure, but is that using PCHs? And how much of that time is preprocessing? I don't think the parent question is whether clang can compile any project in less than 3 hours. I think the interesting question is if warp is more compelling than clang from a preprocessing perspective.
- pjmlp 13y agoYes with PCH and full build not just preprocessing.
- duaneb 13y agoI mean, I do too, but they take even longer with gcc. I can remember when compiling could take a full day. These metrics mean nothing without comparison. Clang is hands down the fastest c/++ compiler I've ever used.
- jevinskie 13y agoI got a chance to try this out. This is the file I used: https://gist.github.com/jevinskie/9843152 https://gist.github.com/jevinskie/9843152 Top-of-trunk clang from http://llvm.org/apt/ http://llvm.org/apt/ takes ~2.1 seconds on my machine to preprocess the file to /dev/null. warp (compiled with -O4 -frelease -fno-bounds-check) takes ~2.8 seconds. So this test case is faster with clang even without precompiled headers. It is hard to make a benchmark for clang's precompiled headers because the AST is lazy-loaded from the PCH. You would have to actually have code use values from the header. EDIT: I forgot another advantage of clang: the preprocessing and compilation and assembly are all done within the same process, eliminating process creation overhead.
- gillianseed 13y agoThe compile speed difference between a modern GCC and Clang/LLVM is nowhere near that dramatic, this comparison you linked is against GCC 4.21 and GCC 4.0 which are 7 and 9 years old respectively.
- jevinskie 13y agoI really only meant to refer to the -fsyntax-only portion which is all that matters when it comes to C preprocessor performance. I know GCCs codegen has steadily improved but I'm not aware of any work to speed up the parser.
- comex 13y agoPrecompiled headers are a pain to use. But clang modules are supposed to be precompiled headers done right: http://clang.llvm.org/docs/Modules.html http://clang.llvm.org/docs/Modules.html When they're usable for C++, there should be little reason to use any special preprocessor, since every header file (not just a static common set) need only be compiled once into a binary format rather than being included into N source files. It can't happen any sooner for me... but as of recently, they're still very broken, so I'm still waiting.
- pmr_ 13y agoClang modules for C++ are going nowhere as long as the concerned working group of WG21 isn't proposing anything. While it seems that everybody has almost the same idea how modules should look as part of the language almost no one can agree on how they should be specified or what part of a module should actually be specified at all.
- comex 13y agoI was under the impression that clang is not waiting for standardization to implement this stuff (indeed, C frameworks on OS X already ship module.map files). Am I wrong?
- pmr_ 13y agoAFAIK the main reason for implementing modules for C++ was to show that it can be done, which makes standardization of a major feature much more likely. As long as there is only one compiler and no guarantee that you will not have to change everything once again as standardization is complete no one is going to touch a large cross-platform codebase and add module support.
- comex 13y agoWell, if the performance gain is enough, I certainly would. (The codebase I'm thinking of is not huge, but large enough that I think modules would make a significant compile speed difference.) I think that with the current implementation, if your header files are sane (no depending on previously included files) you can autogenerate a module map file, one module per header, and have it just work. But I'm not sure if there are any wrinkles, because in the current state all I was able to achieve was clang crashing.
- aDevilInMe 13y agoDoes Andrei have a sticky D button on his keyboard? He seems to mention it in every other sentence, where as I thought this post was about C and C++. I would ask if there are any improvements between using wrap or just using clang.
- m0nastic 13y agoWarp is written in D, by the guy who created the language. It doesn't seem weird that Andrei would want to talk about that (especially considering the fact that I assume he's very much interested in seeing D in use in more places). I had the same question about how it compares to clang though. I suppose now that it's open source, someone can do some testing.
- aDevilInMe 13y ago" I assume he's very much interested in seeing D in use in more places" Yes, I think you are correct and this is what I was getting at. It seems like a propaganda exercise to promote D, you only have to look at the end of the article to see this "And join the D language community for the D Conference 2014 on May 21-23 in Menlo Park, CA." For the record I am very much aware of who Andrei is and the influences he has on certain languages.
- AsmMAn 13y agoWhat's wrong with that?
- fpelliccioni 13y agoFor me is propaganda, too!
- DannyBee 13y agoI would be shocked if there are improvements over clang (and if there were, clang would fix them) in the general case. clang has a pretty optimized preprocessor (It even has things like using SSE2 to do some neat things with character processing). GCC has a fairly modern one, but it's still beatable (and clang beats it handily).
- WalterBright 13y agoWalter here (author of Warp). AMA about Warp.
- haberman 13y agoCould you describe in a bit more detail how the ranges-and-algorithms style applies to Warp? What are the major algorithms that you are gluing together? What is the high-level design of Warp?
- WalterBright 13y ago> how the ranges-and-algorithms style applies to Warp? First off, a text preprocessor is a classic filter program, and ranges-and-algorithms is a classic filter program solution. Hence, if that didn't work out well for Warp's design, that would have been a massive failure. The classic preprocessor design, however, is to split the source text up into preprocessing tokens, process the tokens, and then reconstitute output text from the token stream. Warp doesn't work like that. It deals with everything as text, and desperately tries to avoid tokenizing anything. The ranges used are all ranges of text in various stages of being preprocessed. A major attempt is done to minimize any state kept around, and to avoid doing memory allocation as much as possible. Warp doesn't use much of any classic algorithms. They're all custom ones built by carefully examining the Standard description of how it should work.
- haberman 13y agoSo is the main data processing pipeline broken into several decoupled stages ("algorithms") that pass buffer pointer/length pairs between them ("ranges")? Sorry if I'm misunderstanding, I'm not very familiar with these terms as they are used in D. What are the main stages/algorithms that constitute the processing pipeline? I'm mainly trying to understand the high-level roadmap/design enough that I can use the source code itself to answer my more detailed questions. :)
- WalterBright 13y ago
- wehadfun 13y agoCould any of this be used to improve comiling speed in other languages?
- pmr_ 13y agoWhich other languages use the C/C++ preprocessor? I've seen it used for generating data or other source code when a preprocessor comes in handy, but never as a full-fledged component of another language. I also think that a modern language that uses includes instead of modules is just outright insane.
- plorkyeran 13y agoObjective-C(++). I don't think obj-c changes anything about how the preprocessor works, so presumably it'll work fine for that. Using the C preprocessor in a non-superset-of-C language would be a pretty odd choice.
- duaneb 13y agoObjective-c introduced `#import`, which guards from double including by default. I would suspect and performance gains are tempered with a "pure" objc project.
- aidenn0 13y agoMany assemblers have the option to use the c preprocessor.
- nemetroid 13y agoThe Glasgow Haskell Compiler has a language pragma for running CPP before compilation. I've seen it used a few times, for #ifdef compatibility between Windows and Linux for low-level stuff. I don't think anyone would ever want to use #include or macros, so the issue of speed is not very interesting in that context. example: http://hackage.haskell.org/package/system-filepath-0.4.10/docs/src/Filesystem-Path-CurrentOS.html http://hackage.haskell.org/package/system-filepath-0.4.10/do...
- 13y ago
- zenbowman 13y agoMust say facebook has quite the lineup of all-star programmers: - Carmack - Abrash - Alexandrescu - Kent Beck Cannot say I'm not a little jelly of those who get to spend time with these fine gents. http://www.southparkstudios.com/clips/babjj8/jelly-school http://www.southparkstudios.com/clips/babjj8/jelly-school
- deleted 13y ago[deleted]
- steveklabnik 13y agoI'm thinking trading cards... EDIT: Unsure why my parent comment was deleted, but I was suggesting that trading cards of famous programmers with their accomplishments on the back would be fun(ny).
- aristidb 13y agoI find it a bit weird that the top comment in this story fawns over several star programmers, but ignores the guy who actually wrote the tool that the story is about.
- boxy_brown 13y agoThe article suggests he is not a full-time employee of Facebook.
- WalterBright 13y agoI'm not an employee of Facebook.
- AsmMAn 13y agoThanks. Otherwise no D language could exists.
- deleted 13y ago[deleted]
- dllthomas 13y agoSo, to be clear, this is a reimplementation of cpp?
- WalterBright 13y agoI have never looked at the source code to cpp. But Warp is designed to be a drop-in replacement. It doesn't produce char-by-char exactly the same output as cpp, as the whitespace differs, and the decisions about when/where to produce linemarker records are different. But the output is functionally identical (any differences are bugs in either Warp or cpp).
- dllthomas 13y agoThat fits well within my use of the term "reimplementation." I appreciate the specificity of your response, though - always best to answer every possible question when you're not sure which is meant.
- haberman 13y agoInteresting, I did not realize the standard left room for preprocessors to make implementation-defined decisions!
- mfonda 13y agoGreat line from the interview: > WB: I can guarantee that you are wrong about where your code is spending most of its time if you haven't run a profiler on it. Definitely a good thing for all programmers to think about!
- joeyd 13y agoPardon my unorthodox approach, would greatly appreciate your feedback on news.ycombinator.com/item?+748916
- yetanotherphd 13y agoOne thing that really sped up my compiler was eating a ketogenic diet, and eliminating preprocessed foods.