6 ms·

> back of the envelope calculation for how long it should take: 25 min / 128 = 12 sec (or maybe 24 sec since hyper-threads aren't real cores). Yet it takes 170s

> back of the envelope calculation for how long it should take: 25 min / 128 = 12 sec (or maybe 24 sec since hyper-threads aren't real cores). Yet it takes 170s to compile everything. I'd aim for this linear speedup for compiling (sans overhead to compile a small crate), but the linking part won't be faster, maybe even slower. Maybe a slightly bigger envelope can tell you how much performance is there to extract and the cost of using "too many" crates (which I'm not even sure it's too many, maybe your original crate was too big to ease incremental compilation?)

lsuresh 2y ago
Towards the end, the article says it takes 7s for linking using mold.
- dietr1ch 2y ago
  Yeah, I read that part too late. In that case it seems that there's indeed a lot of overhead when building many crates from a cold-start, but it pays off in wall time and can probably save resources in incremental builds.
  - lsuresh 2y ago
    Yes. We found both cold and incremental builds sped up. The incremental builds were the main win -- small changes to the SQL can sometimes complete in seconds for what used to be a full recompilation.
    - rc00 2y ago
      Rewrite it in Zig? You might even be able to sidestep the LLVM bottleneck entirely. https://news.ycombinator.com/item?id=43016944 https://news.ycombinator.com/item?id=43016944
UncleEntity 2y ago
Yeah, I was working on this project to generate a python C-API module from the SVG schema and found when I generated one huge file (as opposed to one file per class) the compilation times were significantly faster. Or maybe it was generating the C++ SVG library, don't quite remember which one was super slow as I did both at around the same time since the code changes between the two were minimal. Looks like I settled on the slower 'one class per file' compilation method for whatever reason, probably because generating a 200k+ file didn't seem like such a good idea.

lsuresh 2y ago

Towards the end, the article says it takes 7s for linking using mold.

dietr1ch 2y ago
Yeah, I read that part too late. In that case it seems that there's indeed a lot of overhead when building many crates from a cold-start, but it pays off in wall time and can probably save resources in incremental builds.
- lsuresh 2y ago
  Yes. We found both cold and incremental builds sped up. The incremental builds were the main win -- small changes to the SQL can sometimes complete in seconds for what used to be a full recompilation.
  - rc00 2y ago
    Rewrite it in Zig? You might even be able to sidestep the LLVM bottleneck entirely. https://news.ycombinator.com/item?id=43016944 https://news.ycombinator.com/item?id=43016944

UncleEntity 2y ago

Yeah, I was working on this project to generate a python C-API module from the SVG schema and found when I generated one huge file (as opposed to one file per class) the compilation times were significantly faster. Or maybe it was generating the C++ SVG library, don't quite remember which one was super slow as I did both at around the same time since the code changes between the two were minimal. Looks like I settled on the slower 'one class per file' compilation method for whatever reason, probably because generating a 200k+ file didn't seem like such a good idea.