8 ms·

Looking at the TiKV compile time graph, if that's for 2 million lines of real-world (not synthetically generated) code, that actually seems kind of reasonable.

Looking at the TiKV compile time graph, if that's for 2 million lines of real-world (not synthetically generated) code, that actually seems kind of reasonable. ~12 min for a full release build? ~5 min for a full dev build? What looks like several seconds for an incremental build? I suppose it's good that a (former?) core contributor and large user of Rust holds the language to such high standards, but this doesn't seem especially bad. I'm kind of curious how much faster compiling a large Go codebase is. How fast, e.g. does the entirety of Kubernetes compile? I'd imagine it's probably under a minute, but is it several seconds?

nneonneo 7y ago
The only relevant info I could dig up was this issue from Kubernetes in 2016: https://github.com/kubernetes/kubernetes/issues/27444 https://github.com/kubernetes/kubernetes/issues/27444 It seems to be discussing the speed of building Kubernetes on their build-servers, targeting multiple architectures at once. Unfortunately all the build servers are internal to Google so I don't know exactly what's being included (or what's being run), but this comment (https://github.com/kubernetes/kubernetes/issues/27444#issuecomment-226438757 https://github.com/kubernetes/kubernetes/issues/27444#issuec...) details the breakdown of time spent: 30m of build 7m on cluster startup 12m on tests 7m on cluster teardown So, taken with an enormous hunk of rock salt, it seems like Kubernetes might take around 30m to build on a cluster, to build (possibly multiple) release artifacts. Seems in line with Rust.
- melling 7y ago
  What’s the state of parallel builds? We’re getting 16/32 core cpus. In a few years, we’re all going have Threadrippers. Of course everything can’t be done in parallel.
  - granzymes 7y ago
    Almost all parts of the compiler (at least for LLVM) are still strictly single-threaded. You can't even use threads yourself if you write your own LLVM module pass.
    - fnord123 7y ago
      Some of the parts of a compiler are inherently serial. e.g. parsing a file. But if you have to parse multiple files, then hey presto, parallelism. The linker has to work as a spooler for writing the resulting executable or shared objects, but afaik, there's no reason for symbol resolution to not be done in parallel. And if you want to optimize your GOT then that could also be sorted by a page rank style optimization which is the algorithm which was the foundation of the phrase 'embarrassingly parallel'.
  - pjmlp 7y ago
    Given that most sweet shop ITs provide i7 and i5 with 8GB and HDD, good luck getting that hardware adopted there.
- ptomato 7y ago
  That's compiling for 10 os/arch combos, so ~3m per.
  - topspin 7y ago
    That makes a lot more sense. Go was specifically designed to compile fast.
  - nneonneo 7y ago
    On the other hand, these are running on some big cluster with a whole bunch of CPUs, so I wonder if you’d still get 3m per build on something like a laptop?
    - ptomato 7y ago
      Any individual build they're referencing there runs on a single vm (the cluster referenced there is the k8s cluster being built up and torn down on that vm to run e2e tests on). It looks like at the time they were using GCP standard-8 instances, which are 4 core/8 thread w/30GB of ram. It's hard to say exactly what the current build time is, as it looks like they do some build caching now; on my desktop a fully clean build of current kubernetes master (3.2 million loc) for linux/amd64 takes just under 2 minutes.
nerdponx 7y ago
It's interesting seeing these kinds of compilation times being "long" from a machine learning perspective. The equivalent to compiling code in machine learning is training a model. Even on good hardware you can spend hours training a single model. Some of the really big pre-trained models like BERT can take days being trained on a farm of top-of-the-line purpose-built GPUs, which is why people almost never re-train them from scratch without similarly huge amounts computing power and very specific needs.
- jzoch 7y ago
  The equivalent to compiling in machine learning is compiling :). You just ALSO have to train your model.
dilap 7y ago
> I'm kind of curious how much faster compiling a large Go codebase is. How fast, e.g. does the entirety of Kubernetes compile? I was curious too, so I just tried it. On my 6-core previous-gen MacBook pro, I get 2m18s. That's for SLoc = 3296486 (assuming all deps are vendored and non-deps are not-vendored, which I think is true), so about 16K SLoc/s. Or to put it another way, if Rust compiled as quickly as Go, we'd expect to compile a release build of TiKV in 2m.
- amelius 7y ago
  Can you see how long incremental builds take (under a minor change)?
  - dilap 7y ago
    it was very slow, almost a minute! but it has a custom build script, that looks like it’s not really made for incremental builds? it looks like there’s also bazel support, which i would guess is much faster, but i couldn’t figure out how to get it to work...
- dnpp123 7y ago
  It would be interesting to have a `reverse` blog post "why Go compilation time is fast" e.g. which optimizations are overlooked/done quickly by the go compiler ? Could the binary quality be improved by increasing the compilation time ? AFAIK you don't have optimization flags in go (or the default one are optimum). And to be honnest, I have no idea what the compiled Go code looks like. As much as I have no idea what actual instructions are executed by the python interpreter.
  - gwd 7y ago
    Well one big reason is the biggest feature people are complaining about: A lack of generics. According to [1], there's a fundamental trade-off with respect to generics: "The generic dilemma is this: do you want slow programmers, slow compilers and bloated binaries, or slow execution times?" At the moment, Go has no generics, which translates to "slow programmers", but not "slow compilers". It's likely that the simplicity of the Go language as a whole -- which leads to many complaints -- is a major factor in the fast compile times. EDIT: Lest it seem like I'm bashing Go here, the quote above is from one of the core Golang developers (in 2009 no less), and is mirrored in this from the proposal [2] from another core Golang developer. It's meant to be a shorthand for, "Generics do actually save programmer time and effort, and Go programmers are working harder than necessary because the language doesn't have generics yet." The point of this post is to point out that generics have a cost (slow compilation time) and that fast compilation time has a cost (no generics -- at least, not without inventing a new way of doing generics). It's possible to love something while still seeing its deficiencies and wishing its improvement. In fact, I'd argue that's the only way to truly love anyone or anything. [1] https://research.swtch.com/generic https://research.swtch.com/generic [2] https://go.googlesource.com/proposal/+/master/design/15292-generics.md https://go.googlesource.com/proposal/+/master/design/15292-g...
garmaine 7y ago
Yeah, I have a C++ project a fraction of that size that takes longer to compile. I'm quite happy with Rust :) Glad a core contributor has such high standards though.
- unlinked_dll 7y ago
  Is it template heavy? One thing I've noticed code switching to Rust from C++ is that I can write generic heavy code without seeing massive delays in compile time. Conversely in C++ I'd have to create source files with explicit instantiation to get incremental builds of my template usage which is a pain.
zyang 7y ago
Compare to swift it does look pretty reasonable.
foolfoolz 7y ago
i love scala. it would not only be longer but you’d have to compile it 3 times for each major version!
- threeseed 7y ago
  So you're not cross compiling your code for Scala JS (0.6/1) and Scala Native as well meaning 6 full compile cycles ? Shame on you.
p1necone 7y ago
As far as I can tell most of the people complaining about Rust having painfully long compile times are just parroting second hand information and don't actually know. Kinda like the pervasive "but isn't Java really slow?" thing that still doesn't seem to have died either. Edit: yes I know this article was written by someone who knows what they're talking about (and as they're a steward of the project I understand why they're calling this out for improvement, and somewhat exaggerating how bad it is). I was talking about the occasional comments I see on hacker news about how people aren't even bothering to try rust because compile times are "unusably" bad. 5 mins for a dev build of 2 million lines of code is not anywhere near reason to not even bother trying the language. No personal project or learning scratchpad is ever going to come close to 2 million lines. And it's not worse than many other languages that are used for large projects.
- staticassertion 7y ago
  > As far as I can tell most of the people complaining about Rust having painfully long compile times are just parroting second hand information and don't actually know. My experience is the opposite. Most rust devs don't work on very large projects from what I can tell, so compile times tend to stay manageable. It's a small percentage that work on projects large enough to hit this problem, and when you hit it it hurts.
- saghm 7y ago
  I don't necessarily disagree, but this was written by one of the original authors of the Rust compiler, so I think he knows what he's talking about.
- steveklabnik 7y ago
  I have a few beefs with this article, but to suggest Brian doesn’t know much about Rust is just plain incorrect. He is literally the #2 all time (human) contributor to the project: https://thanks.rust-lang.org/rust/all-time/ https://thanks.rust-lang.org/rust/all-time/
- nwmcsween 7y ago
  Well if you read the article you would have known the author is one of the co-founders of Rust.
akiselev 7y ago
Yeah those build times are insanely good for a 2 million line project. I suspect, however, that the codebase makes judicious use of generics. A clean dev build for one of my more experimental projects that uses generic heterogenous lists takes five minutes to compile for fewer than two thousand lines of code and a single test case. In that project, however, I'm pushing Rust's type resolution algorithm to its limits (basically as a giant constraint solver looking for a unique solution).
jstimpfle 7y ago
There will always be applications for doing a fresh complete build. Like, searching for a bug in different code revisions. You're just not doing that if each compile takes 5 minutes. As a baseline, a non optimizing compiler for a simple language should be able to do 1 million lines of code per second. Of course, most languages are not simple. Incremental compilation is an essential operation. I probably do it more than 100 times a day. Just like editor responsiveness, it can almost not be fast enough, and if it takes too long it can bring me out of the flow. I would say that over 0.1 seconds any speed improvement is welcome. More than 3 seconds is definitely a nuisance. More than 15 seconds is extremely frustrating when dealing with certain kinds of code.
- greglindahl 7y ago
  > You're just not doing that if each compile takes 5 minutes. I've bisected large codebases for bugs several times where compiles take hours.
  - IshKebab 7y ago
    Yeah but I bet you wished it took less time.
- dwohnitmok 7y ago
  I'm curious, are there any 1 million LoC projects out there that compile in under a second?
  - keldaris 7y ago
    Sure. TCC (the tiny C compiler) runs at over 3 million LoC/s on my machine (on a single core!) and GCC debug builds aren't that far behind, so for C (or very orthodox C++) projects it's doable. I keep my own C++ codebases under an arbitrary target of 10s for fully optimized release builds and over half of that is wasted by the NVCC.
    - dwohnitmok 7y ago
forrestthewoods 7y ago
Shhhhhh! If we keep complaining about Rust build times being "slow" they'll keep making it better!
gpderetta 7y ago
Our ~1M line of C++ codebase take around ~15 mins on a "full" [1] rebuild, and that's with the support of a custom distributed build system. That's bearable considering that running the unit tests takes again as much. 12 mins for a full build of 2M lines of code on a single box is quite reasonable, and it is great if they are pushing for better performance. [1] because of aggressive caching if is hard to really do a clean build, the closest you can get is touching one of the 'god' headers included everywhere.
- vvanders 7y ago
  Yeah, I'm thinking back to an unnamed game engine that touching any of the core headers incurred a ~1 hour rebuild, ~30 minutes with incredibuild. Heck, if you've ever had a complex codebase with LTCG, I've seen cases where linking alone took ~25 minutes.
  - davedx 7y ago
    Does it start with Un and end with real Engine?
nwmcsween 7y ago
Linux kernel takes ~60 seconds on TR hardware according to phoronix, SLoC is what ~15-18 million
- cyphar 7y ago
  Most of those speed compile benchmarks are done with "make tinyconfig" -- where a small fraction of the kernel is being built (a few million SLoC at most). A kernel build using "make allyesconfig" takes significantly longer than a minute.
- nneonneo 7y ago
  A clean build of the Linux kernel can be very fast, but the config is very relevant. A custom build specific to your hardware will touch only a small fraction of the overall kernel tree, whereas a generic build including a large swath of hardware will take much longer. (AFAICT there's no easy way to measure the LoC actually compiled - but one rough way to estimate it would be to take all the .o files listed in the build process and count the LoC in the corresponding .c files and the .h files they depend on).
  - Sean1708 7y ago
    > AFAICT there's no easy way to measure the LoC actually compiled I take it you can't just run it through the preprocessor then count the lines of code that are output from that?
    - nneonneo 7y ago
      The kernel is composed of a bunch of independent source files which share headers; thus, running them through a preprocessor would massively overcount the LoC. (In theory, this would get you the “actual compiled LoC” but that’s a useless metric because most header files are just declarations that are easy to parse and “compile”).
  - ungamed 7y ago
    Easy way is to grab the debuginfo files, run it through addr2line and count.

nneonneo 7y ago

The only relevant info I could dig up was this issue from Kubernetes in 2016: https://github.com/kubernetes/kubernetes/issues/27444 https://github.com/kubernetes/kubernetes/issues/27444 It seems to be discussing the speed of building Kubernetes on their build-servers, targeting multiple architectures at once. Unfortunately all the build servers are internal to Google so I don't know exactly what's being included (or what's being run), but this comment (https://github.com/kubernetes/kubernetes/issues/27444#issuecomment-226438757 https://github.com/kubernetes/kubernetes/issues/27444#issuec...) details the breakdown of time spent: 30m of build 7m on cluster startup 12m on tests 7m on cluster teardown So, taken with an enormous hunk of rock salt, it seems like Kubernetes might take around 30m to build on a cluster, to build (possibly multiple) release artifacts. Seems in line with Rust.

melling 7y ago
What’s the state of parallel builds? We’re getting 16/32 core cpus. In a few years, we’re all going have Threadrippers. Of course everything can’t be done in parallel.
- granzymes 7y ago
  Almost all parts of the compiler (at least for LLVM) are still strictly single-threaded. You can't even use threads yourself if you write your own LLVM module pass.
  - fnord123 7y ago
    Some of the parts of a compiler are inherently serial. e.g. parsing a file. But if you have to parse multiple files, then hey presto, parallelism. The linker has to work as a spooler for writing the resulting executable or shared objects, but afaik, there's no reason for symbol resolution to not be done in parallel. And if you want to optimize your GOT then that could also be sorted by a page rank style optimization which is the algorithm which was the foundation of the phrase 'embarrassingly parallel'.
- pjmlp 7y ago
  Given that most sweet shop ITs provide i7 and i5 with 8GB and HDD, good luck getting that hardware adopted there.
ptomato 7y ago
That's compiling for 10 os/arch combos, so ~3m per.
- topspin 7y ago
  That makes a lot more sense. Go was specifically designed to compile fast.
- nneonneo 7y ago
  On the other hand, these are running on some big cluster with a whole bunch of CPUs, so I wonder if you’d still get 3m per build on something like a laptop?
  - ptomato 7y ago
    Any individual build they're referencing there runs on a single vm (the cluster referenced there is the k8s cluster being built up and torn down on that vm to run e2e tests on). It looks like at the time they were using GCP standard-8 instances, which are 4 core/8 thread w/30GB of ram. It's hard to say exactly what the current build time is, as it looks like they do some build caching now; on my desktop a fully clean build of current kubernetes master (3.2 million loc) for linux/amd64 takes just under 2 minutes.

nerdponx 7y ago

It's interesting seeing these kinds of compilation times being "long" from a machine learning perspective. The equivalent to compiling code in machine learning is training a model. Even on good hardware you can spend hours training a single model. Some of the really big pre-trained models like BERT can take days being trained on a farm of top-of-the-line purpose-built GPUs, which is why people almost never re-train them from scratch without similarly huge amounts computing power and very specific needs.

jzoch 7y ago
The equivalent to compiling in machine learning is compiling :). You just ALSO have to train your model.

dilap 7y ago

> I'm kind of curious how much faster compiling a large Go codebase is. How fast, e.g. does the entirety of Kubernetes compile? I was curious too, so I just tried it. On my 6-core previous-gen MacBook pro, I get 2m18s. That's for SLoc = 3296486 (assuming all deps are vendored and non-deps are not-vendored, which I think is true), so about 16K SLoc/s. Or to put it another way, if Rust compiled as quickly as Go, we'd expect to compile a release build of TiKV in 2m.

amelius 7y ago
Can you see how long incremental builds take (under a minor change)?
- dilap 7y ago
  it was very slow, almost a minute! but it has a custom build script, that looks like it’s not really made for incremental builds? it looks like there’s also bazel support, which i would guess is much faster, but i couldn’t figure out how to get it to work...
dnpp123 7y ago
It would be interesting to have a `reverse` blog post "why Go compilation time is fast" e.g. which optimizations are overlooked/done quickly by the go compiler ? Could the binary quality be improved by increasing the compilation time ? AFAIK you don't have optimization flags in go (or the default one are optimum). And to be honnest, I have no idea what the compiled Go code looks like. As much as I have no idea what actual instructions are executed by the python interpreter.
- gwd 7y ago
  Well one big reason is the biggest feature people are complaining about: A lack of generics. According to [1], there's a fundamental trade-off with respect to generics: "The generic dilemma is this: do you want slow programmers, slow compilers and bloated binaries, or slow execution times?" At the moment, Go has no generics, which translates to "slow programmers", but not "slow compilers". It's likely that the simplicity of the Go language as a whole -- which leads to many complaints -- is a major factor in the fast compile times. EDIT: Lest it seem like I'm bashing Go here, the quote above is from one of the core Golang developers (in 2009 no less), and is mirrored in this from the proposal [2] from another core Golang developer. It's meant to be a shorthand for, "Generics do actually save programmer time and effort, and Go programmers are working harder than necessary because the language doesn't have generics yet." The point of this post is to point out that generics have a cost (slow compilation time) and that fast compilation time has a cost (no generics -- at least, not without inventing a new way of doing generics). It's possible to love something while still seeing its deficiencies and wishing its improvement. In fact, I'd argue that's the only way to truly love anyone or anything. [1] https://research.swtch.com/generic https://research.swtch.com/generic [2] https://go.googlesource.com/proposal/+/master/design/15292-generics.md https://go.googlesource.com/proposal/+/master/design/15292-g...

garmaine 7y ago

Yeah, I have a C++ project a fraction of that size that takes longer to compile. I'm quite happy with Rust :) Glad a core contributor has such high standards though.

unlinked_dll 7y ago
Is it template heavy? One thing I've noticed code switching to Rust from C++ is that I can write generic heavy code without seeing massive delays in compile time. Conversely in C++ I'd have to create source files with explicit instantiation to get incremental builds of my template usage which is a pain.

zyang 7y ago

Compare to swift it does look pretty reasonable.

foolfoolz 7y ago

i love scala. it would not only be longer but you’d have to compile it 3 times for each major version!

threeseed 7y ago
So you're not cross compiling your code for Scala JS (0.6/1) and Scala Native as well meaning 6 full compile cycles ? Shame on you.

p1necone 7y ago

As far as I can tell most of the people complaining about Rust having painfully long compile times are just parroting second hand information and don't actually know. Kinda like the pervasive "but isn't Java really slow?" thing that still doesn't seem to have died either. Edit: yes I know this article was written by someone who knows what they're talking about (and as they're a steward of the project I understand why they're calling this out for improvement, and somewhat exaggerating how bad it is). I was talking about the occasional comments I see on hacker news about how people aren't even bothering to try rust because compile times are "unusably" bad. 5 mins for a dev build of 2 million lines of code is not anywhere near reason to not even bother trying the language. No personal project or learning scratchpad is ever going to come close to 2 million lines. And it's not worse than many other languages that are used for large projects.

staticassertion 7y ago
> As far as I can tell most of the people complaining about Rust having painfully long compile times are just parroting second hand information and don't actually know. My experience is the opposite. Most rust devs don't work on very large projects from what I can tell, so compile times tend to stay manageable. It's a small percentage that work on projects large enough to hit this problem, and when you hit it it hurts.
saghm 7y ago
I don't necessarily disagree, but this was written by one of the original authors of the Rust compiler, so I think he knows what he's talking about.
steveklabnik 7y ago
I have a few beefs with this article, but to suggest Brian doesn’t know much about Rust is just plain incorrect. He is literally the #2 all time (human) contributor to the project: https://thanks.rust-lang.org/rust/all-time/ https://thanks.rust-lang.org/rust/all-time/
nwmcsween 7y ago
Well if you read the article you would have known the author is one of the co-founders of Rust.

akiselev 7y ago

Yeah those build times are insanely good for a 2 million line project. I suspect, however, that the codebase makes judicious use of generics. A clean dev build for one of my more experimental projects that uses generic heterogenous lists takes five minutes to compile for fewer than two thousand lines of code and a single test case. In that project, however, I'm pushing Rust's type resolution algorithm to its limits (basically as a giant constraint solver looking for a unique solution).

jstimpfle 7y ago

There will always be applications for doing a fresh complete build. Like, searching for a bug in different code revisions. You're just not doing that if each compile takes 5 minutes. As a baseline, a non optimizing compiler for a simple language should be able to do 1 million lines of code per second. Of course, most languages are not simple. Incremental compilation is an essential operation. I probably do it more than 100 times a day. Just like editor responsiveness, it can almost not be fast enough, and if it takes too long it can bring me out of the flow. I would say that over 0.1 seconds any speed improvement is welcome. More than 3 seconds is definitely a nuisance. More than 15 seconds is extremely frustrating when dealing with certain kinds of code.

greglindahl 7y ago
> You're just not doing that if each compile takes 5 minutes. I've bisected large codebases for bugs several times where compiles take hours.
- IshKebab 7y ago
  Yeah but I bet you wished it took less time.
dwohnitmok 7y ago
I'm curious, are there any 1 million LoC projects out there that compile in under a second?
- keldaris 7y ago
  Sure. TCC (the tiny C compiler) runs at over 3 million LoC/s on my machine (on a single core!) and GCC debug builds aren't that far behind, so for C (or very orthodox C++) projects it's doable. I keep my own C++ codebases under an arbitrary target of 10s for fully optimized release builds and over half of that is wasted by the NVCC.
  - dwohnitmok 7y ago

forrestthewoods 7y ago

Shhhhhh! If we keep complaining about Rust build times being "slow" they'll keep making it better!

gpderetta 7y ago

Our ~1M line of C++ codebase take around ~15 mins on a "full" [1] rebuild, and that's with the support of a custom distributed build system. That's bearable considering that running the unit tests takes again as much. 12 mins for a full build of 2M lines of code on a single box is quite reasonable, and it is great if they are pushing for better performance. [1] because of aggressive caching if is hard to really do a clean build, the closest you can get is touching one of the 'god' headers included everywhere.

vvanders 7y ago
Yeah, I'm thinking back to an unnamed game engine that touching any of the core headers incurred a ~1 hour rebuild, ~30 minutes with incredibuild. Heck, if you've ever had a complex codebase with LTCG, I've seen cases where linking alone took ~25 minutes.
- davedx 7y ago
  Does it start with Un and end with real Engine?

nwmcsween 7y ago

Linux kernel takes ~60 seconds on TR hardware according to phoronix, SLoC is what ~15-18 million

cyphar 7y ago
Most of those speed compile benchmarks are done with "make tinyconfig" -- where a small fraction of the kernel is being built (a few million SLoC at most). A kernel build using "make allyesconfig" takes significantly longer than a minute.
nneonneo 7y ago
A clean build of the Linux kernel can be very fast, but the config is very relevant. A custom build specific to your hardware will touch only a small fraction of the overall kernel tree, whereas a generic build including a large swath of hardware will take much longer. (AFAICT there's no easy way to measure the LoC actually compiled - but one rough way to estimate it would be to take all the .o files listed in the build process and count the LoC in the corresponding .c files and the .h files they depend on).
- Sean1708 7y ago
  > AFAICT there's no easy way to measure the LoC actually compiled I take it you can't just run it through the preprocessor then count the lines of code that are output from that?
  - nneonneo 7y ago
    The kernel is composed of a bunch of independent source files which share headers; thus, running them through a preprocessor would massively overcount the LoC. (In theory, this would get you the “actual compiled LoC” but that’s a useless metric because most header files are just declarations that are easy to parse and “compile”).
- ungamed 7y ago
  Easy way is to grab the debuginfo files, run it through addr2line and count.