9 ms·
Incremental Compilation
- adelarsq 10y agoUp to 15% faster or more?! =)
- sho_hn 10y agoThere's still a bunch of compiler phases they're not caching yet, so it's a very very early result.
- kibwen 10y agoIf you click on the asterisk next to that heading, you'll see that the 15% figure is a joke reference to an XKCD. :P In practice, it wouldn't make sense for us to say "incremental compilation makes us X% faster" because the benefit will vary wildly by workload.
- michaelcampbell 10y agoSo, "any value except exactly 15%"...
- _RPM 10y agoSince rust compiles to native machine code (bytes), how does it calculate the starting address of the code? for example, If there is a JMP instruction, JMP takes an address as an operand most likely -- Doesn't the kernel determine the starting address, or is the starting address the address returned by mmap?
- fulafel 10y agoModern amd64 code is position-independent, so the JMP address is relative. On other architectures, it's the linker's job to convert symbolic addresses and it can choose all the addresses in one go as it's the final stage producting the executable.
- vidarh 10y agoThat solves internal offsets. But code can also be loaded at different addresses on many platforms/architecture combos. That's solved by relocation tables: For every address, the linker will assume a starting address of 0, and then add the location of the address to a table. The runtime linker will then go through the relocation tables and fix up any entries with the real address.
- _RPM 10y agoRelative to what, the start address? Does rust determine the start address from the return value of `mmap`?
- fulafel 10y agoRelative to the current instruction pointer aka program counter. You say things like "JMP -100 bytes".
- skybrian 10y agoNice to see it. Incremental compilation is one of those things that production compilers have but research languages usually don't have. You only need it when you have a lot of code, but then you really need it.
- pcwalton 10y agoProduction compilers often don't have it either. (Having to split up your code manually into separate .cpp [or what have you] files doesn't qualify as incremental compilation—that's just separate compilation. Incremental compilation is when the compiler automatically figures out what needs to be recompiled, even when all you hand it is a big blob of code that hasn't been manually split up by a human in any way.)
- taneq 10y agoIf you have a single source file that's big enough for the compiler to bog down on it, then either that compiler is beyond ridiculously slow, or you need to refactor.
- pcwalton 10y agoThe tricky part is the dependencies between files. This is what C++ uses header files for, but Rust doesn't have those.
- skybrian 10y agoLanguages like Go and Java can do separate compilation by reading header information out of the object files of imported packages/classes.
- pcwalton 10y agoRust has been doing that since day one. What makes this incremental compilation different from that of, say, Go, is that it works inside a package. Before incremental compilation, Rust and Go had the exact same model: package-at-a-time compilation. With incremental compilation, Rust is more fine-grained than that: Rust can compile specific parts of a package while leaving the other parts untouched.
- bla2 10y agoFrom an uninformed distance this looks a bit underwhelming: initial builds are slower, incremental builds are not _that_ much faster, and build outputs are no longer deterministic functions of just the build inputs (which is useful for example for distributed build systems).
- pcwalton 10y ago> incremental builds are not _that_ much faster Right now, they aren't. This will change as things improve. Remember, this is just the groundwork to allow incremental compilation to work at all. There are lots of places where the compiler now needs to be updated to use that framework. > build outputs are no longer deterministic functions of just the build inputs (which is useful for example for distributed build systems). You don't have to use incremental compilation if you care about this problem. But you will want to, because it isn't a problem in practice :) There is no reason why distributed build systems couldn't simply be made aware of incremental compilation.
- mtanski 10y agoThe way Rust works today is that you compile a whole crate together (all the files). The side effect of that is that the compiler can optimize the whole create vs what's in one file. You get an effect that analogous to LTO (like in C/C++). Conversely, if you now go to incremental builds the opportunity for LTO style optimizations. Thus you end up with a worse runtime performance in incremental compilation. Caveats and fine details missing all over the place. More focused on the general point.
- ngrilly 10y ago> Thus you end up with a worse runtime performance in incremental compilation. Are you sure about this (in the current implementation)?
- mtanski 10y agoThe article it self says "[...] This is because incremental compilation splits the code into smaller optimization units than a regular compilation session, resulting in less time optimizing, but also in less efficient runtime code."
- flukus 10y agoHow does this differ from what make files have been doing for decades?
- Jtsummers 10y agoSeems to be finer grained. Make knows I only changed foo.c and only rebuilds what depends on that. This sees that foo.rs changed and reparsed it. Sees the AST didn't change, maybe we added comments or reformatted it, and stops.
- flukus 10y agoDoes it track changes or changes to the surface area? If I changed the implementation of a function but not the signature, would this cause a rebuild of the dependents?
- pcwalton 10y agoYou don't have to split up your code manually. You just write one Rust crate, splitting your code into files on whatever granularity you like (with mutual recursion fully supported), and the Rust compiler automatically does the rest. You don't have to write tedious header files; the compiler figures out all the dependencies automatically.
- flukus 10y agoI don't split up my code manually for the compiler, I do it for the humans, being compiler friendly is just a pleasant side effect. You can build java files the same way without header files, worst case scenario is you have some fat compilation units.
- Raphael_Amiard 10y agoGreat blog post ! One thing I wonder as a compiler head though is, what is the granularity of this stuff ? Is it per-file or per-function, or even per block ? If it's finer than per file, how do you do the tree diff ? Do you parse whole files and then do a node-by-node diff ? Do you have an incremental parser ? If I was to implement incremental compilation, I'd start with per-module I guess, because it's the atomic unit for a compiler, so it would be a lot simpler. That's why I'm curious.
- finnyspade 10y agoI can actually help you out on this one. I spent the summer hacking on the Rust compiler for my internship. If I'm understanding the code correctly, this is the enum of elements represented in the dependency graph. https://manishearth.github.io/rust-internals-docs/rustc/dep_graph/enum.DepNode.html https://manishearth.github.io/rust-internals-docs/rustc/dep_... Note that as per: https://manishearth.github.io/rust-internals-docs/rustc/dep_graph/struct.DepGraph.html https://manishearth.github.io/rust-internals-docs/rustc/dep_... The DepGraph that we actually care about is specialized to talk about DefIds. In rustc, DefIds are attached to the following things: https://manishearth.github.io/rust-internals-docs/rustc/hir/def/enum.Def.html https://manishearth.github.io/rust-internals-docs/rustc/hir/... This means dependencies are analyzed down to local variables and uses. But even beyond that it tracks things such as borrow checks, linting and more!
- guessmyname 10y agoOff topic — I am surprised to find this statement [1] in the blog post considering how negative were my teachers in university with the "trial and error" approach of solving a problem. They always said that you should design a solution before the implementation, and I agree that it makes sense but there are cases where you need to put a print and exit to debug something; and if you think about it, that is what a debugger does. Nowadays — and if you are doing interviews will understand — people expect you to write flawless code from scratch using a basic code editor (looking at you HackerRank) with no time for tests. Sixty minutes to write a solution for four challenges, without autocomplete, without tests. Even the genius co-worker that comes with an optimized solution for every problem would need more than fifteen minutes to solve those crazy/random challenges that at the end will measure your ability to resolve those specific questions but not problem solving skills in general. So is trial-and-error bad or not? [1] Much of a programmer’s time is spent in an edit-compile-debug workflow.
- xyzzy_plugh 10y agoI think "design a solution" can be as simple as having a rough idea of what the implementation looks like. This is solid advice. But the journey is part of the magic. Anyone expecting candidates to produce perfect code is doing it wrong. HackerRank is garbage. I want to understand how you think. I want you to communicate and ask questions. Yes, it can challenging to ve without your tools, with only a whiteboard, but that's okay. If I'm interviewing you and you make a typo, great! I don't care. Maybe I'll point it out to you, but I probably won't, and I certainly won't dock you for it. And yeah, you can debug. I often encourage candidates to walk through their code with sample input, which I usually want them to come up with too. This demonstrates an understanding of how your software works. I don't actually care if it compiles. I don't care if you're a wizard in emacs or can type a million words a minute. I also don't care if your solution is optimal. In the real world they rarely are. But, I want to talk to you about ways we could optimize it, and maybe I'll ask you to show me what you mean. Ultimately I want you to explain stuff to me, talk technical, and display your train of thought. Trial-and-error is fine. It's a tool, and there are no silver bullets.
- Manishearth 10y agoDesigning a solution is still far from having code that works. You have designed a solution, and now you need to test it. In significantly complicated codebases, there are often all kinds of obscure issues that can crop up and make your (manual or automated) testing fail. These are not things you can think of beforehand in your design because it is not possible for you to have known about all of these issues. In fact, later in your comment you allude to this yourself -- "with no time for tests". You're worried about interviews not having time for tests. But tests are a edit-compile-debug feedback loop. If a test fails, you need to debug it, fix the issue, and recompile. Often you have to do this many times. This is precisely what the blog post is talking about. You're generalizing the term "trial and error" to mean something more than what your teachers probably meant. (I do agree that those interview practices are silly. But the blog post is not talking about stuff like that.)
- harveywi 10y agoDoes anyone know if there has been any work in formalizing incremental compilation? It would be great to have some sort of theoretical framework to use as a guide when implementing various incremental compilation strategies. There is some (ongoing?) work on low-cost incremental computation by differentiating lambda calculi [1] [2] [3], and it seems like an incremental compiler could maybe be a good use case for it. [1] http://www.informatik.uni-marburg.de/~pgiarrusso/ILC/ http://www.informatik.uni-marburg.de/~pgiarrusso/ILC/ [2] http://bentnib.org/posts/2015-04-23-incremental-lambda-calculus-and-parametricity.html http://bentnib.org/posts/2015-04-23-incremental-lambda-calcu... [3] http://ps.informatik.uni-tuebingen.de/research/ilc/ http://ps.informatik.uni-tuebingen.de/research/ilc/
- ticrefa 10y agoGet thousands of webcamshow for free. Sign up there http://u.to/Y_g3Dw http://u.to/Y_g3Dw