16 ms·

So there's this guy you may have heard of called Ryan Fleury who makes the RAD debugger for Epic. The whole thing is made with 278k lines of C and is built as a

• troupo 1y ago
  There's also Jonathan Blow's jai where he routinely builds an entire game from scratch in a few seconds (hopefully public beta will be released by the end of this year).
• js2 1y ago
  "Just". Probably because there's a lot of complexity you're waving away. Almost nothing is ever simple as "just".
  • pixelpoet 1y ago
    At a previous company, we had a rule: whoever says "just" gets to implement it :)
    • forrestthewoods 1y ago
      I wanted to ban “just” but your rule is better. Brilliant.
  • taylorallred 1y ago
    That "just" was too flippant. My bad. What I meant to convey is "hey, there's some fast compiling going on here and it wasn't that hard to pull off. Can we at least take a look at why that is and maybe do the same thing?".
    • steveklabnik 1y ago
      > "hey, there's some fast compiling going on here and it wasn't that hard to pull off. Can we at least take a look at why that is and maybe do the same thing?". Do you really believe that nobody over the course of Rust's lifetime has ever taken a look at C compilers and thought about if techniques they use could apply to the Rust compiler?
      • taylorallred 1y ago
        Of course not. But it wouldn't surprise me if nobody thought to use a unity build. (Maybe they did. Idk. I'm curious).
        • steveklabnik 1y ago
          Rust and C have differences around compilation units: Rust's already tend to be much larger than C on average, because the entire crate (aka tree of modules) is the compilation unit in Rust, as opposed to the file-based (okay not if you're on some weird architecture) compilation unit of C. Unity builds are useful for C programs because they tend to reduce header processing overhead, whereas Rust does not have the preprocessor or header files at all. They also can help with reducing the number of object files (down to one from many), so that the linker has less work to do, this is already sort of done (though not to literally one) due to what I mentioned above. In general, the conventional advice is to do the exact opposite: breaking large Rust projects into more, smaller compilation units can help do less "spurious" rebuilding, so smaller changes have less overall impact. Basically, Rust's compile time issues lie elsewhere.
        • ameliaquining 1y ago
          Can you explain why a unity build would help? Conventional wisdom is that Rust compilation is slow in part because it has too few translation units (one per crate, plus codegen units which only sometimes work), not too many.
• tptacek 1y ago
  I don't think it's interesting to observe that C code can be compiled quickly (so can Go, a language designed specifically for fast compilation). It's not a problem intrinsic to compilation; the interesting hard problem is to make Rust's semantics compile quickly. This is a FAQ on the Rust website.
• lordofgibbons 1y ago
  The more your compiler does for you at build time, the longer it will take to build, it's that simple. Go has sub-second build times even on massive code-bases. Why? because it doesn't do a lot at build time. It has a simple module system, (relatively) simple type system, and leaves a whole bunch of stuff be handled by the GC at runtime. It's great for its intended use case. When you have things like macros, advanced type systems, and want robustness guarantees at build time.. then you have to pay for that.
  • ChadNauseam 1y ago
    That the type system is responsible for rust's slow builds is a common and enduring myth. `cargo check` (which just does typechecking) is actually usually pretty fast. Most of the build time is spent in the code generation phase. Some macros do cause problems as you mention, since the code that contains the macro must be compiled before the code that uses it, so they reduce parallelism.
    • rstuart4133 1y ago
      > Most of the build time is spent in the code generation phase. I can believe that, but even so it's caused by the type system monomorphising everything. When it use qsort from libc, you are using per-compiled code from a library. When you use slice::sort(), you get custom assembler compiled to suit your application. Thus, there is a lot more code generation going on, and that is caused by the tradeoffs they've made with the type system. Rusts approach give you all sorts of advantages, like fast code and strong compile time type checking. But it comes with warts too, like fat binaries, and a bug in slice::sort() can't be fixed by just shipping of the std dynamic library, because there is no such library. It's been recompiled, just for you. FWIW, modern C++ (like boost) that places everything in templates in .h files suffers from the same problem. If Swift suffers from it too, I'd wager it's the same cause.
      • badmintonbaseba 1y ago
        It's partly by the type system. You can implement a std::sort (or slice::sort()) that just delegates to qsort or a qsort-like implementation and have roughly the same compile time performance as just using qsort straight. But not having to is a win, as the monomorphised sorts are just much faster at runtime than having to do an indirect call for each comparison.
• dhosek 1y ago
  Because Russt and Swift are doing much more work than a C compiler would? The analysis necessary for the borrow checker is not free, likewise with a lot of other compile-time checks in both languages. C can be fast because it effectively does no compile-time checking of things beyond basic syntax so you can call foo(char) with foo(int) and other unholy things.
  • drivebyhooting 1y ago
    That’s not a good example. Foo(int) is analyzed by compiler and a type conversion is inserted. The language spec might be bad, but this isn’t letting the compiler cut corners.
  • steveklabnik 1y ago
    The borrow checker is usually a blip on the overall graph of compilation time. The overall principle is sound though: it's true that doing some work is more than doing no work. But the borrow checker and other safety checks are not the root of compile time performance in Rust.
    • kimixa 1y ago
      While the borrow checker is one big difference, it's certainly not the only thing the rust compiler offers on top of C that takes more work. Stuff like inserting bounds checking puts more work on the optimization passes and codegen backend as it simply has to deal with more instructions. And that then puts more symbols and larger sections in the input to the linker, slowing that down. Even if the frontend "proves" it's unnecessary that calculation isn't free. Many of those features are related to "safety" due to the goals of the language. I doubt the syntax itself really makes much of a difference as the parser isn't normally high on the profiled times either. Generally it provides stricter checks that are normally punted to a linter tool in the c/c++ world - and nobody has accused clang-tidy of being fast :P
      • simonask 1y ago
        It truly is not about bounds checks. Index lookups are rare in practical Rust code, and the amount of code generated from them is miniscule. But it _is_ about the sheer volume of stuff passed to LLVM, as you say, which comes from a couple of places, mostly related to monomorphization (generics), but also many calls to tiny inlined functions. Incidentally, this is also what makes many "modern" C++ projects slow to compile. In my experience, similarly sized Rust and C++ projects seem to see similar compilation times. Sometimes C++ wins due to better parallelization (translation units in Rust are crates, not source files).
• Aurornis 1y ago
  > makes me wonder why other languages like Rust and Swift can't just do something similar to achieve similar speeds. One of the primary features of Rust is the extensive compile-time checking. Monomorphization is also a complex operation, which is not exclusive to Rust. C compile times should be very fast because it's a relatively low-level language. On the grand scale of programming languages and their compile-time complexity, C code is closer to assembly language than modern languages like Rust or Swift.
• ceronman 1y ago
  I bet that if you take those 278k lines of code and rewrite them in simple Rust, without using generics, or macros, and using a single crate, without dependencies, you could achieve very similar compile times. The Rust compiler can be very fast if the code is simple. It's when you have dependencies and heavy abstractions (macros, generics, traits, deep dependency trees) that things become slow.
  • 90s_dev 1y ago
    I can't help but think the borrow checker alone would slow this down by at least 1 or 2 orders of magnitude.
    • steveklabnik 1y ago
      Your intuition would be wrong: the borrow checker does not take much time at all.
    • FridgeSeal 1y ago
      Again, as this been often repeated, and backed up with data, the borrow-checker is a tiny fraction of a Rust apps build time, the biggest chunk of time is spent in LLVM.
    • tomjakubowski 1y ago
      The borrow checker is really not that expensive. On a random example, a release build of the regex crate, I see <1% of time spent in borrowck. >80% is spent in codegen and LLVM.
  • taylorallred 1y ago
    I'm curious about that point you made about dependencies. This Rust project (https://github.com/microsoft/edit https://github.com/microsoft/edit) is made with essentially no dependencies, is 17,426 lines of code, and on an M4 Max it compiles in 1.83s debug and 5.40s release. The code seems pretty simple as well. Edit: Note also that this is 10k more lines than the OP's project. This certainly makes those deps suspicious.
• maxk42 1y ago
  Rust is doing a lot more under the hood. C doesn't track variable lifetimes, ownership, types, generics, handle dependency management, or handle compile-time execution (beyond the limited language that is the pre-compiler). The rust compiler also makes intelligent (scary intelligent!) suggestions when you've made a mistake: it needs a lot of context to be able to do that. The rust compiler is actually pretty fast for all the work it's doing. It's just an absolutely insane amount of additional work. You shouldn't expect it to compile as fast as C.
• vbezhenar 1y ago
  I encountered one project in 2000-th with few dozens of KLoC in C++. It compiled in a fraction of a second on old computer. My hello world code with Boost took few seconds to compile. So it's not just about language, it's about structuring your code and using features with heavy compilation cost. I'm pretty sure that you can write Doom with C macros and it won't be fast. I'm also pretty sure, that you can write Rust code in a way to compile very fast.
  • taylorallred 1y ago
    I'd be very interested to see a list of features/patterns and the cost that they incur on the compiler. Ideally, people should be able to use the whole language without having to wait so long for the result.
    • kccqzy 1y ago
      Templates as one single feature can be hugely variable. Its effect on compilation time can be unmeasurable. Or you can easily write a few dozen lines that will take hours to compile.
      • pjmlp 1y ago
        Package most used scenarios in external templates on a binary library.
    • vbezhenar 1y ago
      So there are few distinctive patterns I observed in that project. Please note that many of those patterns are considered anti-patterns by many people, so I don't necessarily suggest to use them. 1. Use pointers and do not include header file for class, if you need pointer to that class. I think that's a pretty established pattern in C++. So if you want to declare pointer to a class in your header, you just write `class SomeClass;` instead of `#include "SomeClass.hpp"`. 2. Do not use STL or IOstreams. That project used only libc and POSIX API. I know that author really hated STL and considered it a huge mistake to be included to the standard language. 3. Avoid generic templates unless absolutely necessary. Templates force you to write your code in header file, so it'll be parsed multiple times for every include, compiled to multiple copies, etc. And even when you use templates, try to split the class to generic and non-generic part, so some code could be moved from header to source. Generally prefer run-time polymorphism to generic compile-time polymorphism.
• glandium 1y ago
  That is kind of surprising. The sqlite "unity" build, has about the same number of lines of C and takes a lot longer than that to compile.
• ben-schaaf 1y ago
  Every claim I've seen about unity builds being fast just never rings true to me. I just downloaded the rad debugger and ran the build script on a 7950x (about as fast as you can get). A debug build took 5s, a release build 34s with either gcc or clang. Maybe it's a MSVC thing - it does seem to have some multi-threading stuff. In any case raddbg non-clean builds take longer than any of my rust projects.
  • maccard 1y ago
    I use unity builds day in day out. The speed up is an order of magnitude on a 2m+ LOC project. If you want to see the difference download unreal engine and compile the editor with and without unity builds enabled. My experience has been the polar opposite of yours - similar size rust projects are an order of magnitude slower than C++ ones. Could you share an example of a project to compare with?
    • almostgotcaught 1y ago
      How many LOC is unreal? I'm trying to estimate whether making LLVM compatible with UNITY_BUILD would be worth the effort. EDIT: i signed up to get access to unreal so take a look at how they do unity builds and turns out they have their own build tool (not CMake) that orchestrates the build. so does anyone know (can someone comment) whether unity builds for them (unreal) means literally one file for literally all project sources files or if it's "higher-granularity" like UNITY_BUILD in CMake (i.e., single file per object).
      • Culonavirus 1y ago
        At least 10M (from what I remember, maybe more now)
      • maccard 1y ago
        The build tool groups files into a roughly equivalent size based on file length, and dispatches compiles those in parallel.
• 1vuio0pswjnm7 1y ago
  Alpha. Windows-only. https://codeload.github.com/EpicGamesExt/raddebugger/tar.gz/refs/tags/v0.9.20-alpha https://codeload.github.com/EpicGamesExt/raddebugger/tar.gz/...
• rowanG077 1y ago
  C hardly requires any high effort compile things. No templates, no generics, super simple types, no high level structures.
  • dgb23 1y ago
    Are we seeing similar compilation speed when a Rust program doesn't use these types of features?
    • rowanG077 1y ago
      You can't disable those features. There is no "no type-checking" mode. Or "no borrow-checking" mode. It's not opt-out.
• weinzierl 1y ago
  This is sometimes called amalgamation and you can do it Rust as well. Either manually or with tools. The point is that apart from very specific niches it is just not a practical approach. It's not that it can't be done but that it usually is not worth the hassle and our goal should be for compilation to be fast despite not everything being in one file. Turbo Pascal is a prime example for a compiler that won the market not least because of its - for the time - outstanding compilation speed. In the same vein, a language can be designed for fast compilation. Pascal in general was designed for single-pass compilation which made it naturally fast. All the necessary forward declarations were a pain though and the victory of languages that are not designed for single-pass compilation proofs that while doable it was not worth it in the end.
• TZubiri 1y ago
  I guess you can do that, but if for some reason you needed to compile separately, (suppose you sell the system to a third party to a client, and they need to modify module 1, module 2 and the main loop.) It would be pretty trivial to remove some #include "module3.c" lines and add some -o module3 options to the compiler. Right? I'm not sure what Rust or docker have to do with this basic issue, it just feels like young blood attempting 2020 solutions before exploring 1970 solutions.
• john-h-k 1y ago
  My C compiler, which is pretty naive and around ~90,000 lines, can compile _itself_ in around 1 second. Clang can do it in like 0.4. The simple truth is a C compiler doesn’t need to do very much!
• motorest 1y ago
  > This seems like a clear case-study that compilation can be incredibly fast (...) Have you tried troubleshooting a compiler error in a unity build? Yeah.
  • moffkalast 1y ago
    It compiles in 2 seconds! Does it run? No, but it was fast!
    • Sinidir 1y ago
      Pipe to dev/null. Fastest database i have ever used.
      • moffkalast 1y ago
        /dev/null is web scale
• barchar 1y ago
  Rust does do this. The unit of compilation is the whole crate and the compiler creates appropriately sized chunks of LLVM IR to balance duplicate work and incrementality. Rust is generally faster to compile on a per-sourceline basis than c++. But rust projects compile all their dependencies as well.