29 ms·

Thoughts on Rust bloat

443 points by xenocratus 7y ago

nerdponx 7y ago
It's a little unfortunate that some of the cool features of Rust need to be put into the "use sparingly" category. I know that polymorphism, async, etc. should be used judiciously anyway, but still. One recent case we saw a similar tradeoff was the observation that the unicase dep adds 50k to the binary size for pulldown-cmark. In this case, the CommonMark spec demands Unicode case-folding, and without that, it’s no longer complying with the standard. I understand the temptation to cut this corner, but I think having versions out there that are not spec-compliant is a bad thing, especially unfriendly to the majority of people in the world whose native language is other than English. In the Python world, you can install an "extra" along with a package. So you can make the deliberate decision to omit Unicode case folding from your CommonMark parser. Maybe something like that is possible with a crate? That said, I think this is a non-feature in the spec, if anything. I see the value in recommending (but not requiring) Unicode normalization, but I don't see the added value of Unicode-aware case-insensitivity. Maybe it's more important in non-Latin text.
- romaaeterna 7y ago
  > I don't see the added value of Unicode-aware case-insensitivity. Maybe it's more important in non-Latin text I believe that -- to give one example -- this is something that Chrome implements, but Firefox does not. It's a huge pain to have to match accents perfectly in a text search on a page when you often want your search to be accent and case insensitive. This sort of thing is very important for any text search workload, I imagine.
- dbaupp 7y ago
  Python extras are pretty much equivalent to cargo "features": https://doc.rust-lang.org/cargo/reference/manifest.html#the-features-section https://doc.rust-lang.org/cargo/reference/manifest.html#the-... The article mentions a few places that use cargo "features" for exactly that sort of thing.
- pornel 7y ago
  I don't see the "use sparingly" conclusion. It's a complex trade-off of runtime performance, code clarity, productivity, safety, binary size, and compile times. Rust tries hard to eat all the cakes and have them too, but it can't do miracles. And looks like the author is on the right track to tackling the problem — moving less common functionality behind feature flags, pregenerating data "offline", profiling and trimming code with the help of cargo-bloat.
- msbarnett 7y ago
  > It's a little unfortunate that some of the cool features of Rust need to be put into the "use sparingly" category. This is just the reality of engineering -- there are no silver bullets. Sure, it'd be nice if any language could give you the space efficiency of dynamic dispatch with the runtime efficiency of monomorphized generics, but those two things are fundamentally in tension. Neither Rust nor any other language can fix that. Rust at least gives you a fairly easy choice of which you want in any given circumstance. Most languages just pick one or the other universally.
  - heavenlyblue 7y ago
    One way to fix it is jitting. I am wondering if Rust has a project of shipping Rust compiler embedded in your executable, so that one could compile sym into code.
    - msbarnett 7y ago
      > One way to fix it is jitting. Also not a silver bullet, because there's non-zero costs in memory, CPU overhead, I$, etc to the statistics gathering, stop-and-jit-the-dynamic-call-and-change-the-call-sites and so forth. In long running server processes this can mostly amortize out nicely over a very long run, but for interactive applications it can add noticeable lag.
- pierreyoda 7y ago
  > In the Python world, you can install an "extra" along with a package. So you can make the deliberate decision to omit Unicode case folding from your CommonMark parser. Maybe something like that is possible with a crate? This is definitely possible in Rust with Cargo crate features! [0] For instance in a game library like quicksilver you can opt in to WebAssembly support, images, fonts, audio handling, etc. Then the image or audio library themselves can put formats and codecs behind feature flags for instance. Each crate then uses conditional compilation to branch out these features at compilation time. More commonly as an end-user, test modules use a special conditional compilation macro [1]. [0] https://doc.rust-lang.org/cargo/reference/manifest.html#the-features-section https://doc.rust-lang.org/cargo/reference/manifest.html#the-... [1] https://doc.rust-lang.org/stable/rust-by-example/testing/unit_testing.html https://doc.rust-lang.org/stable/rust-by-example/testing/uni...
- leshow 7y ago
  > I know that polymorphism, async, etc. should be used judiciously anyway, but still. I don't really see this recommended anywhere, especially re: polymorphism. Async _maybe_, but it's pretty situational.
neonate 7y ago
In case anyone else was wondering what "druid" is, it's https://github.com/xi-editor/druid https://github.com/xi-editor/druid.
Qasaur 7y ago
Use polymorphism sparingly I think it is a little ironic that he speaks of performance culture but simutaneously advises to use dynamic dispatch and avoid polymorphism. I can see the justification in non-critical code paths, but serialisation is a pretty important part of most networked software nowadays so I do not think that smaller binaries and faster compilation times (better developer experience) justifies a performance hit in the form of dynamic dispatch through crates like miniserde.
- andolanra 7y ago
  One thing I like about Rust a lot is that it lets you choose which hit you want to take when it comes to polymorphism. You can manually (and without much difficulty!) prefer dynamic dispatch if it's important to you to keep your binary size small, but you can also choose static dispatch and allow some replicated copies of your parametric code if that's what you want to optimize for. It's also worth noting that if you are using polymorphic functions that are only ever called in your code with a single known type parameter, then your program should be just as efficient as if you wrote a monomorphic version with that fixed type in both runtime _and_ binary size, which means the only disadvantage to the polymorphism there is compile time.
  - Koshkin 7y ago
    I’ve been using C and C++ (sorry Rust), like, forever, and I think should hate them with all fibers of my soul. A high-level programming language is “supposed” to let me forget, for the higher being’s sake, all machine-specific details and focus on the logic of the problem at hand. (Hello FORTRAN.)
    - dingo_bat 7y ago
      If you, as the programmer, would prefer not to know and remember machine specific details, you can use rust or c++ easily. Just don't think you will always get the best performance possible for your hardware. And I think that is justified and reasonable.
  - bogdanoff_2 7y ago
svnpenn 7y ago
Rust bloat is a serious issue that is not being taken seriously I think. I raised the issue about platform size in June: https://github.com/rust-lang/rust/issues/61978 https://github.com/rust-lang/rust/issues/61978 and its actually gotten worse since then, significantly worse. In the 2 months since then the installer has increased from 203 MB to 299 MB. Also unbelieveably, Rust has failed to address package balkanization which I would say has ruined the Node community. A popular package is "cargo-edit", which currently pulls in 239 other crates: https://github.com/rust-lang/cargo/issues/2179#issuecomment-523567384 https://github.com/rust-lang/cargo/issues/2179#issuecomment-...
- Qasaur 7y ago
  Also unbelieveably, Rust has failed to address package balkanization which I would say has ruined the Node community. I'd say that this is more of a culture thing rather than a language thing.
  - thesuperbigfrog 7y ago
    Whether culture or language, if it is not addressed then expect similar results :(
    - dom96 7y ago
      The primary way I can see to address this is to make packaging more difficult. Are there other steps that a language can take to avoid this problem?
      - thesuperbigfrog 7y ago
        A few ideas: 1) Remove the ability to unpublish / yank crates. A published crate should be immutable, but the crate's metadata should always be updateable by the maintainer. 2) Improve the metadata that describes a crate so that it is easy to tell if a crate should be used. For example, is the crate beta quality? Was a serious error found in the crate and it needs to be marked as "not safe"? Is it a Long-Term Support release? Etc. 3) As a culture, disallow trivial crates. No "is-odd" or similarly low effort crates. These just add bloat since the have so little functionality compared to their overhead. If your crate's toml is larger than the crate's code, you are doing it wrong.
pornel 7y ago
It looks like the author is most bothered by compile times of dependencies. Cargo needs to do better with shared caches (so you compile each dep at most once per machine) or ability to get precompiled crates (so you don't even compile it). Incremental improvements of compiler speed or trimming of individual dependencies won't bring the 10x improvement it needs.
- holy_city 7y ago
  You can kind of hack a shared cache using cargo workspaces (put all your rust projects in one directory as part of a workspace) but that is far from ideal. All your projects will share a target directory and Cargo.lock. I agree though. I don't mind binary sizes in the single megabytes for a release build, and I don't mind clean/rebuild build times that are a few dozen seconds. I do mind the 500MB of build artifacts in a single target directory, multiplied by all the projects I have that share the same version of serde/winapi/syn/quote/insert-common-dependency-here.
- leggomylibro 7y ago
  One of the biggest issues that I face with Rust is that its builds are enormous, and I often work on machines with limited disc space. The actual binary sizes are fine - even with embedded devices that have <1MB of program space, there doesn't seem to be much bloat. But I need to remember to clean every project when I finish working on it, because otherwise the build folders will eat 100s-1000s of megabytes each on a 16-32GB partition. I really appreciate Rust's inclusive approach to learning and teaching, but I can't justify using it for education on ultra-affordable machines for that reason. People often scatter one-off projects all over the place as they learn, and when they run out of space it takes a long time to clean everything up. So I would also be very in favor of some sort of simple shared package cache.
  - asgeir 7y ago
    Have you considered doing something like this: cargo build --target-dir ~/build-artifacts/$(basename $(pwd)) Then you can have something like this in your ~/.config/user-tmpfiles.d/clean_build.conf d /home/user/build-artifacts - - - 1d - You can then do cleanup by calling systemd-tmpfiles --user Or you can use the tmpfiles timer systemctl enable --user --now systemd-tmpfiles-clean.timer
qaq 7y ago
It highly depends on use case. If I have web application running on 100 nodes 5mb vs even 10mb has such a minuscule cost that it not even worth calculating. Now benefits for ops from just having to push a single static binary without external dependencies are fairly decent.
mapgrep 7y ago
>the release binary is now 5.9M A typical smartphone ships with around 10,000 times this much storage capacity and enough RAM to hold it 100 times over. This is bloat? I mean, I get it, the binary used to be only 2MB, 1/3rd the size. But are numbers this low really worth worrying about? I think a GUI app in 6MB is hugely impressive. I genuinely thought he was going to say it was 100MB or something higher.
- dmix 7y ago
  Even Raspberry Pis have many GBs of both disk and RAM and super fast USB, MicroSD, and network transfer speeds. I don't really see the difference between 5mb and 20mb bins. And I'm a fan of minimalist OS systems like Archlinux with a slim set of running services. Disk space isn't really the main concern besides as a symbolic measure of cruft. No system is ever going to fill up space by having too many OS/terminal programs installed. But I also don't come from the nostalgic Unixy C programming world. Just a Unix user. So I may be biased.
  - heavenlyblue 7y ago
    >> Even Raspberry Pis have many GBs of both disk and RAM and super fast USB, MicroSD, and network transfer speeds. You are speaking about _latest_ Pis.
    - dmix 7y ago
      No I'm not. Pi 2 meets all of those requirements and has MicroSDHC which offer tons of high speed space even at entry level. Besides that was an example, I have plenty of other IoT and embedded style hardware and they are all fitting this criteria. Simply put the size of binaries has never been a real concern for some time now and the few rare cases where it might matter probably aren't worth all the effort made obsessing over sub 50mb bin size. 99%+ of it will be people who "feel" like it should be small, without any good reason.
- xyzzyz 7y ago
  I think a GUI app in 6MB is hugely impressive. You think that because nobody cares about bloat anymore, so everything else is large.
weinzierl 7y ago
> There’s also an effort to analyze binary sizes more systematically. I applaud such efforts and would love it if they were even more visible. Ideally, crates.io would include some kind of bloat report along with its other metadata, [...] This is what I always wanted (for Rust as well as for C) but never got around to hack together myself. I dreamt it up more as a feature of cargo though, something like 'cargo stats' or so. Shouldn't be to hard and cargo is extensible.
- dmix 7y ago
  For NPM they should make it optionally inverted to highlight the one line node packages that you can get rid of!
- sitkack 7y ago
  You might be looking for `cargo install cargo-bloat` $ cargo bloat Compiling ... Analyzing target/debug/mdbook File .text Size Crate Name 0.5% 1.3% 166.4KiB regex <regex::exec::ExecNoSync as regex::re_trait::RegularExpression>::captures_read_at 0.5% 1.3% 165.2KiB idna unicode_normalization::tables::compatibility_fully_decomposed 0.3% 0.7% 95.0KiB ammonia html5ever::tree_builder::TreeBuilder<Handle,Sink>::step 0.3% 0.7% 92.5KiB idna unicode_normalization::tables::canonical_fully_decomposed 0.2% 0.5% 64.6KiB unicase unicase::unicode::map::lookup 0.2% 0.5% 63.4KiB idna unicode_normalization::tables::is_combining_mark 0.2% 0.4% 55.8KiB regex <regex::re_trait::Matches<R> as core::iter::traits::iterator::Iterator>::next 0.2% 0.4% 55.5KiB regex regex::re_unicode::Regex::find_at 0.1% 0.3% 41.7KiB unicode_normalization unicode_normalization::tables::composition_table 0.1% 0.3% 36.2KiB rand_hc rand_hc::hc128::Hc128Core::sixteen_steps 0.1% 0.3% 33.8KiB regex regex::re_unicode::Regex::shortest_match_at 0.1% 0.3% 32.4KiB rand_hc <rand_hc::hc128::Hc128Core as rand_core::block::BlockRngCore>::generate 0.1% 0.2% 31.4KiB ammonia html5ever::tokenizer::Tokenizer<Sink>::step 0.1% 0.2% 24.5KiB idna unicode_normalization::tables::canonical_combining_class 0.1% 0.2% 21.8KiB regex aho_corasick::ahocorasick::AhoCorasick<S>::find 0.1% 0.2% 21.6KiB regex aho_corasick::ahocorasick::AhoCorasick<S>::find 0.1% 0.2% 21.0KiB clap clap::app::parser::Parser::get_matches_with 0.1% 0.2% 20.3KiB ws ws::io::Handler<F>::handle_queue 0.1% 0.1% 19.0KiB ws ws::connection::Connection<H>::read_frames 0.1% 0.1% 18.7KiB env_logger termcolor::Ansi<W>::write_color 35.3% 91.6% 11.4MiB And 58189 smaller methods. Use -n N to show more. 38.6% 100.0% 12.5MiB .text section size, the file size is 32.4MiB
carterschonwald 7y ago
Seems like a lot of this comes from rust defaulting to allowing several different versions of a library to be linked in ... there’s def some other pieces. But that seems like a biggie
- lwhsiao 7y ago
  Do you know how to not do this "default"? I would love to try it.
  - MaulingMonkey 7y ago
    So far in my toy projects I've mostly seen this crop up from depending on a lot of 0.x crates still going through significant version churn and legitimate breaking changes, where A and B legitimately can't use the same version of C as-is due to API changes, because they haven't been keeping their dependencies up to date. The fix is simple, when it happens: Patch A/B to use the latest major version of C, fixing the source code as necessary. You can [patch] locally until upstream accepts your Pull Request - which might include a https://deps.rs/repo/github/rust-lang/cargo https://deps.rs/repo/github/rust-lang/cargo badge in their README.md, to encourage them to continue to keep C up-to-date.
  - JeanMertz 7y ago
    Clippy has an (off by default) lint to warn/deny using multiple versions of the same crate.
  - carterschonwald 7y ago
    i would love to, too! :)
ajxs 7y ago
At the risk of being slightly tangential, I've been sorely wanting to air this particular grievance with Rust for some time. It's somewhat related, since the author mentions their package system. Its package ecosystem isn't nearly in the horrible state that node's is, but having a package system shouldn't be a substitute for designing a useful standard library for a language. I think that the attraction to 'small languages' is very much misplaced. If I can't get through Rust's official documentation without being recommended the use of third party packages for basic functionality (getopt, interfacing with static libraries... Etc) then the designers have made a terrible error.
- Myrmornis 7y ago
  Isn't it at least partly explained by wanting to allow implementations of this functionality to evolve more rapidly than they could if they were in the stdlib and thus reach good solutions more rapidly? getopt in particular seems like something that programmers haven't reached a consensus on despite 40 years of experimentation.
  - ajxs 7y ago
    I would beg to differ with this. The kind of functionality expected from Getopt is fairly standard these days. I also disagree that features like getopt really need to 'evolve' much at this point in time. At the risk of courting controversy here, I'm going to confess that often I just don't want the community involved in the development of a language. On a project I inherited recently ( with the mission to save ) I unfortunately had to suffer using Typescript. If you look at the community discussions regarding the language's future direction you'll be treated to the uninformed arguing with the ignorant about the language's direction. Every time I download an external crate, I have to learn a different developer's way of doing things, suffer their idiosyncratic ways of creating an API and potentially expose myself or my application to a new set of vulnerabilities. The more I can get away with not doing this, the better.
    - sanderjd 7y ago
      Who is "the community" and who isn't? Are you sure you trust the core language developers more than people writing libraries?
ijiiijji1 7y ago
Maybe rust core/std/alloc should built with many smaller crates that are cached better and have fewer dependencies? Also: https://github.com/johnthagen/min-sized-rust https://github.com/johnthagen/min-sized-rust
geofft 7y ago
> For one, it’s common that you get different versions anyway (the Zola build currently has two versions each of unicase, parking_lot, parking_lot_core, crossbeam-deque, toml, derive_more, lock_api, scopeguard, and winapi). This seems like a specific thing that would be a measurable win and a not uncommon problem (e.g., I have a test kernel module that doesn't have many dependencies, and it still has two generic-arrays, two proc-macro2s, and two unicode-xids). Is there something that could be done here technically, such as pull requests to bump common crates to using the same version of even-more-common dependencies?
- MaulingMonkey 7y ago
  > generic-array This is up to 0.13.2... plenty of supposedly breaking API churn, no wonder you have two copies. Yes, your dependencies probably rely on different 0.x versions, and a pull request could fix that. You can inspect your Cargo.lock file to figure out which ones are to blame. > Is there something that could be done here technically, such as pull requests to bump common crates to using the same version of even-more-common dependencies? Yep, that's the fix. README.md badges like https://deps.rs/repo/github/rust-lang/cargo https://deps.rs/repo/github/rust-lang/cargo can alert people to out-of-date dependencies. You can even automate some of the pull requests with something like https://github.com/marketplace/dependabot-preview https://github.com/marketplace/dependabot-preview if you control the repositories.
Animats 7y ago
Do dynamic libraries really help? They only save space in the case where 1) you are running many different programs and 2) they all use the same libraries in the same versions. If you're running multiple copies of the same program, they share code, at least on Linux.
- raphlinus 7y ago
  Depends on the use case. I'm thinking about GUI, where on Linux you can write code that links against (say) Gtk, and thus can pull in lots of libraries while the executable itself is tiny. With the current state of the Rust ecosystem, you basically have to build the GUI toolkit and bundle it with your app.
- deafcalculus 7y ago
  Dynamic libs on the file system are awful. It's OK if the kernel wants to conserve RAM by sharing code pages that are identical, but saving hard disk space with dynamic libs is so painful.
- erik_seaberg 7y ago
  > running many different programs and they all use the same libraries in the same versions It's distro maintainers' job to test packaged apps and converge on a known-good version of a shared lib. This is only a problem if your distro doesn't package the apps you need, and you build from upstream source ignoring the distro and using all the random versions upstream happened to test.
leeoniya 7y ago
> Digging into xi-editor, the biggest single source of bloat is serde flatbuffers seems to be the go-to serialization lib, in terms of speed and mem efficiency: https://docs.rs/flatbuffers/0.6.0/flatbuffers/ https://docs.rs/flatbuffers/0.6.0/flatbuffers/
je42 7y ago
I don't understand the advice: "Use async sparingly" It doesn't make sense. Either your complete code base is async or it is not. If you have a single blocking code point in it. It is not async anymore, since it can't handle any other schedule async tasks while waiting for the blocking call.
- raphlinus 7y ago
  Probably could have worded it better. I didn't mean, "only use a little bit of async," I agree that doesn't make sense. I meant, "use async if your problem really needs it, otherwise avoid it."
  - je42 7y ago
    The main thing it is you can't really add async when you need it, because if you have written enough sync code, turning it into async amounts to a major rewrite/refactor of the code including choices for dependencies, code for parallel code execution are different in async etc.
ogoffart 7y ago
It looks like the `fluent` library is to blame for the bloat here. I have myself made another localization library that use a simpler model and compiles quite fast when using the rust `gettext` backend: https://github.com/woboq/tr https://github.com/woboq/tr
jokoon 7y ago
I just don't feel like rust can replace C/C++, because the syntax is not simple enough. Rust has a lot of cool things, but to me syntax simplicity is more important. Maybe I have a hard time adapting myself to rust? Maybe my brain is too much "wired" to a C style syntax. It still seems that to me, C-style syntax is just better. I would rather prefer a C++-like language that breaks down from backward compatibility with C while keeping its simplicity, has STL containers, and is simpler to read and use. Rust is cool, but I'm just curious if it can really be adopted for large project to justify rewriting existing code.
- timw4mail 7y ago
  C++ has simple syntax? Since when?
  - jokoon 7y ago
    Since you're not really required to use templates or inheritance at every corner.
- floatboth 7y ago
  Maybe you're looking for D? :) Anyway, what is "not simple" about Rust's syntax?
  - jokoon 7y ago
    Using "let thing: type" seems bloated, I prefer the C style "type thing;" ; can also make a lot of difference, which is a little weird. I don't understand why there is both str and String, seems complicated for nothing. Option things are not really clear yet, and I don't understand what is their use, it seems like an alternative to unions, but few developers use unions anyway... Pattern matching seems powerful but I fail to understand its usage, and I'm a little skeptical about the machine code it generates. D is fine, but I think that even D is not simple enough. Rust seems like it's awesome, but it requires to entirely rethink how you write code, and I've never been a fan of high level abstraction.
kissgyorgy 7y ago
Serialization will always be slow for obvious reasons, that's why binary protocols and messages started to emerge (HTTP/2, gRPC) instead of serializing/deserializing everything to JSON and back.
- floatboth 7y ago
  grpc uses protobuf which still does traditional serialization. The cool stuff is capnp/flatbuffers, where you write to and read from the "serial" memory directly.
axilmar 7y ago
Since it's not possible for a Standard Library to have everything a programmer needs, programming languages shouldn't have Standard Libraries but Standard Repositories.
home_project123 7y ago
How many people would use a Rust cloud compiler? Suppose it cuts build time from 5 minutes to 30 seconds.... Technically, a transparently mirrored file system, a strong compiler cluster (memory, cores, etc). And some predictive ML. But you end up with a binary-equivalent (verifiable) output. Any thoughts ?
- Hortinstein 7y ago
  I would likely use this depending on how transparent it was and the pricing and if it supported RLS etc. I was writing a rust project on a terrible old laptop that took 2-5 minutes for compilation time. I ended up standing up a Digital Ocean instance and used VScode (via Coder https://github.com/cdr/code-server https://github.com/cdr/code-server ). And this ended up being the most workable solution. Cut down my compilation times significantly and also helped w/ VScode RAM usage. Not sure how the buisness model for a cloud compiler would work, but I would be interested.
- Macha 7y ago
  I wouldn't use it. Incremental builds aren't that bad, and I'd have a hard time trusting third party compiled libs enough to include in a release from a new service that hasn't built up trust the way that say a Linux distro has. While you could assume any maliciousness or security compromise would be caught, as you can see from the rubygems news today this is not instant and it adds another point of failure.
kanishkarj 7y ago
I'm naive to rust, but I do get your concern. For a small part in a project, I used a tokio based library. And it just exponentially increased my build time. What about if rust could support something like dynamic Linking/loading. Like have some crates globally installed and while building we could link to the global one instead of locally getting all the crates. Like C/C++does it right?
ameixaseca 7y ago
Good points, but I'd like to point out that nothing on this post is Rust-specific. You can have the same issue if programming in any other language, including C: excessive indirections, inefficient algorithms, bad abstractions, excessive use of unnecessary libraries, etc. It's indeed easier to "bloat" your resulting binary in C++ or Rust given the easiness to do higher-level abstractions; since you can more easily program complex solutions you also need to consider your design and the trade-offs on your code. I'd also like to point out that, in comparison: * Rust bloat is a speck compared to hundreds of megabytes for a similar Python program + runtime including the same amount of code. * Rust bloat can be mostly optimized away for systems that really care about excess/unused code - you have #![no_std], disabling of backtraces, aborting on panic, and a lot of other optimizations that throw away a big portion of extra functionality not needed for things like embedded. You have little alternative on things like Go besides removing debugging symbols and doing tricks like "dynamic decompression" (which could also be applied to Rust programs to further reduce their size, btw). Bottom line is: Rust makes it easier for you to "just add a new library" and it also make you more mindful of the bloat, but we need to keep it in perspective.
xpe 7y ago
To respond to one particular point from the post: > Once you accept bloat, it’s very hard to claw it back. If your project has multi-minute compiles, people won’t even notice a 10s regression in compile time. Then these pile up, and it gets harder and harder to motivate the work to reduce bloat, because each second gained in compile time becomes such a small fraction of the total. This particular problem can be addressed head-on, I think. It would seem feasible to have the compiler to distinguish between the target application (library) and the dependencies. Then it could report the compile times as separate values. This could be built into CI/CD tools as a way to catch application level compile time changes. Of course, this approach wouldn’t tell the entire story, but it would likely serve as a canary in the coal mine at least.

443 points by xenocratus 7y ago

It's a little unfortunate that some of the cool features of Rust need to be put into the "use sparingly" category. I know that polymorphism, async, etc. should be used judiciously anyway, but still. One recent case we saw a similar tradeoff was the observation that the unicase dep adds 50k to the binary size for pulldown-cmark. In this case, the CommonMark spec demands Unicode case-folding, and without that, it’s no longer complying with the standard. I understand the temptation to cut this corner, but I think having versions out there that are not spec-compliant is a bad thing, especially unfriendly to the majority of people in the world whose native language is other than English. In the Python world, you can install an "extra" along with a package. So you can make the deliberate decision to omit Unicode case folding from your CommonMark parser. Maybe something like that is possible with a crate? That said, I think this is a non-feature in the spec, if anything. I see the value in recommending (but not requiring) Unicode normalization, but I don't see the added value of Unicode-aware case-insensitivity. Maybe it's more important in non-Latin text.

romaaeterna 7y ago
> I don't see the added value of Unicode-aware case-insensitivity. Maybe it's more important in non-Latin text I believe that -- to give one example -- this is something that Chrome implements, but Firefox does not. It's a huge pain to have to match accents perfectly in a text search on a page when you often want your search to be accent and case insensitive. This sort of thing is very important for any text search workload, I imagine.
dbaupp 7y ago
Python extras are pretty much equivalent to cargo "features": https://doc.rust-lang.org/cargo/reference/manifest.html#the-features-section https://doc.rust-lang.org/cargo/reference/manifest.html#the-... The article mentions a few places that use cargo "features" for exactly that sort of thing.
pornel 7y ago
I don't see the "use sparingly" conclusion. It's a complex trade-off of runtime performance, code clarity, productivity, safety, binary size, and compile times. Rust tries hard to eat all the cakes and have them too, but it can't do miracles. And looks like the author is on the right track to tackling the problem — moving less common functionality behind feature flags, pregenerating data "offline", profiling and trimming code with the help of cargo-bloat.
msbarnett 7y ago
> It's a little unfortunate that some of the cool features of Rust need to be put into the "use sparingly" category. This is just the reality of engineering -- there are no silver bullets. Sure, it'd be nice if any language could give you the space efficiency of dynamic dispatch with the runtime efficiency of monomorphized generics, but those two things are fundamentally in tension. Neither Rust nor any other language can fix that. Rust at least gives you a fairly easy choice of which you want in any given circumstance. Most languages just pick one or the other universally.
- heavenlyblue 7y ago
  One way to fix it is jitting. I am wondering if Rust has a project of shipping Rust compiler embedded in your executable, so that one could compile sym into code.
  - msbarnett 7y ago
    > One way to fix it is jitting. Also not a silver bullet, because there's non-zero costs in memory, CPU overhead, I$, etc to the statistics gathering, stop-and-jit-the-dynamic-call-and-change-the-call-sites and so forth. In long running server processes this can mostly amortize out nicely over a very long run, but for interactive applications it can add noticeable lag.
pierreyoda 7y ago
> In the Python world, you can install an "extra" along with a package. So you can make the deliberate decision to omit Unicode case folding from your CommonMark parser. Maybe something like that is possible with a crate? This is definitely possible in Rust with Cargo crate features! [0] For instance in a game library like quicksilver you can opt in to WebAssembly support, images, fonts, audio handling, etc. Then the image or audio library themselves can put formats and codecs behind feature flags for instance. Each crate then uses conditional compilation to branch out these features at compilation time. More commonly as an end-user, test modules use a special conditional compilation macro [1]. [0] https://doc.rust-lang.org/cargo/reference/manifest.html#the-features-section https://doc.rust-lang.org/cargo/reference/manifest.html#the-... [1] https://doc.rust-lang.org/stable/rust-by-example/testing/unit_testing.html https://doc.rust-lang.org/stable/rust-by-example/testing/uni...
leshow 7y ago
> I know that polymorphism, async, etc. should be used judiciously anyway, but still. I don't really see this recommended anywhere, especially re: polymorphism. Async _maybe_, but it's pretty situational.

neonate 7y ago

In case anyone else was wondering what "druid" is, it's https://github.com/xi-editor/druid https://github.com/xi-editor/druid.

Qasaur 7y ago

Use polymorphism sparingly I think it is a little ironic that he speaks of performance culture but simutaneously advises to use dynamic dispatch and avoid polymorphism. I can see the justification in non-critical code paths, but serialisation is a pretty important part of most networked software nowadays so I do not think that smaller binaries and faster compilation times (better developer experience) justifies a performance hit in the form of dynamic dispatch through crates like miniserde.

andolanra 7y ago
One thing I like about Rust a lot is that it lets you choose which hit you want to take when it comes to polymorphism. You can manually (and without much difficulty!) prefer dynamic dispatch if it's important to you to keep your binary size small, but you can also choose static dispatch and allow some replicated copies of your parametric code if that's what you want to optimize for. It's also worth noting that if you are using polymorphic functions that are only ever called in your code with a single known type parameter, then your program should be just as efficient as if you wrote a monomorphic version with that fixed type in both runtime _and_ binary size, which means the only disadvantage to the polymorphism there is compile time.
- Koshkin 7y ago
  I’ve been using C and C++ (sorry Rust), like, forever, and I think should hate them with all fibers of my soul. A high-level programming language is “supposed” to let me forget, for the higher being’s sake, all machine-specific details and focus on the logic of the problem at hand. (Hello FORTRAN.)
  - dingo_bat 7y ago
    If you, as the programmer, would prefer not to know and remember machine specific details, you can use rust or c++ easily. Just don't think you will always get the best performance possible for your hardware. And I think that is justified and reasonable.
- bogdanoff_2 7y ago

svnpenn 7y ago

Rust bloat is a serious issue that is not being taken seriously I think. I raised the issue about platform size in June: https://github.com/rust-lang/rust/issues/61978 https://github.com/rust-lang/rust/issues/61978 and its actually gotten worse since then, significantly worse. In the 2 months since then the installer has increased from 203 MB to 299 MB. Also unbelieveably, Rust has failed to address package balkanization which I would say has ruined the Node community. A popular package is "cargo-edit", which currently pulls in 239 other crates: https://github.com/rust-lang/cargo/issues/2179#issuecomment-523567384 https://github.com/rust-lang/cargo/issues/2179#issuecomment-...

Qasaur 7y ago
Also unbelieveably, Rust has failed to address package balkanization which I would say has ruined the Node community. I'd say that this is more of a culture thing rather than a language thing.
- thesuperbigfrog 7y ago
  Whether culture or language, if it is not addressed then expect similar results :(
  - dom96 7y ago
    The primary way I can see to address this is to make packaging more difficult. Are there other steps that a language can take to avoid this problem?
    - thesuperbigfrog 7y ago
      A few ideas: 1) Remove the ability to unpublish / yank crates. A published crate should be immutable, but the crate's metadata should always be updateable by the maintainer. 2) Improve the metadata that describes a crate so that it is easy to tell if a crate should be used. For example, is the crate beta quality? Was a serious error found in the crate and it needs to be marked as "not safe"? Is it a Long-Term Support release? Etc. 3) As a culture, disallow trivial crates. No "is-odd" or similarly low effort crates. These just add bloat since the have so little functionality compared to their overhead. If your crate's toml is larger than the crate's code, you are doing it wrong.

pornel 7y ago

It looks like the author is most bothered by compile times of dependencies. Cargo needs to do better with shared caches (so you compile each dep at most once per machine) or ability to get precompiled crates (so you don't even compile it). Incremental improvements of compiler speed or trimming of individual dependencies won't bring the 10x improvement it needs.

holy_city 7y ago
You can kind of hack a shared cache using cargo workspaces (put all your rust projects in one directory as part of a workspace) but that is far from ideal. All your projects will share a target directory and Cargo.lock. I agree though. I don't mind binary sizes in the single megabytes for a release build, and I don't mind clean/rebuild build times that are a few dozen seconds. I do mind the 500MB of build artifacts in a single target directory, multiplied by all the projects I have that share the same version of serde/winapi/syn/quote/insert-common-dependency-here.
leggomylibro 7y ago
One of the biggest issues that I face with Rust is that its builds are enormous, and I often work on machines with limited disc space. The actual binary sizes are fine - even with embedded devices that have <1MB of program space, there doesn't seem to be much bloat. But I need to remember to clean every project when I finish working on it, because otherwise the build folders will eat 100s-1000s of megabytes each on a 16-32GB partition. I really appreciate Rust's inclusive approach to learning and teaching, but I can't justify using it for education on ultra-affordable machines for that reason. People often scatter one-off projects all over the place as they learn, and when they run out of space it takes a long time to clean everything up. So I would also be very in favor of some sort of simple shared package cache.
- asgeir 7y ago
  Have you considered doing something like this: cargo build --target-dir ~/build-artifacts/$(basename $(pwd)) Then you can have something like this in your ~/.config/user-tmpfiles.d/clean_build.conf d /home/user/build-artifacts - - - 1d - You can then do cleanup by calling systemd-tmpfiles --user Or you can use the tmpfiles timer systemctl enable --user --now systemd-tmpfiles-clean.timer

qaq 7y ago

It highly depends on use case. If I have web application running on 100 nodes 5mb vs even 10mb has such a minuscule cost that it not even worth calculating. Now benefits for ops from just having to push a single static binary without external dependencies are fairly decent.

mapgrep 7y ago

>the release binary is now 5.9M A typical smartphone ships with around 10,000 times this much storage capacity and enough RAM to hold it 100 times over. This is bloat? I mean, I get it, the binary used to be only 2MB, 1/3rd the size. But are numbers this low really worth worrying about? I think a GUI app in 6MB is hugely impressive. I genuinely thought he was going to say it was 100MB or something higher.

dmix 7y ago
Even Raspberry Pis have many GBs of both disk and RAM and super fast USB, MicroSD, and network transfer speeds. I don't really see the difference between 5mb and 20mb bins. And I'm a fan of minimalist OS systems like Archlinux with a slim set of running services. Disk space isn't really the main concern besides as a symbolic measure of cruft. No system is ever going to fill up space by having too many OS/terminal programs installed. But I also don't come from the nostalgic Unixy C programming world. Just a Unix user. So I may be biased.
- heavenlyblue 7y ago
  >> Even Raspberry Pis have many GBs of both disk and RAM and super fast USB, MicroSD, and network transfer speeds. You are speaking about _latest_ Pis.
  - dmix 7y ago
    No I'm not. Pi 2 meets all of those requirements and has MicroSDHC which offer tons of high speed space even at entry level. Besides that was an example, I have plenty of other IoT and embedded style hardware and they are all fitting this criteria. Simply put the size of binaries has never been a real concern for some time now and the few rare cases where it might matter probably aren't worth all the effort made obsessing over sub 50mb bin size. 99%+ of it will be people who "feel" like it should be small, without any good reason.
xyzzyz 7y ago
I think a GUI app in 6MB is hugely impressive. You think that because nobody cares about bloat anymore, so everything else is large.

weinzierl 7y ago

> There’s also an effort to analyze binary sizes more systematically. I applaud such efforts and would love it if they were even more visible. Ideally, crates.io would include some kind of bloat report along with its other metadata, [...] This is what I always wanted (for Rust as well as for C) but never got around to hack together myself. I dreamt it up more as a feature of cargo though, something like 'cargo stats' or so. Shouldn't be to hard and cargo is extensible.

dmix 7y ago
For NPM they should make it optionally inverted to highlight the one line node packages that you can get rid of!
sitkack 7y ago
You might be looking for `cargo install cargo-bloat` $ cargo bloat Compiling ... Analyzing target/debug/mdbook File .text Size Crate Name 0.5% 1.3% 166.4KiB regex <regex::exec::ExecNoSync as regex::re_trait::RegularExpression>::captures_read_at 0.5% 1.3% 165.2KiB idna unicode_normalization::tables::compatibility_fully_decomposed 0.3% 0.7% 95.0KiB ammonia html5ever::tree_builder::TreeBuilder<Handle,Sink>::step 0.3% 0.7% 92.5KiB idna unicode_normalization::tables::canonical_fully_decomposed 0.2% 0.5% 64.6KiB unicase unicase::unicode::map::lookup 0.2% 0.5% 63.4KiB idna unicode_normalization::tables::is_combining_mark 0.2% 0.4% 55.8KiB regex <regex::re_trait::Matches<R> as core::iter::traits::iterator::Iterator>::next 0.2% 0.4% 55.5KiB regex regex::re_unicode::Regex::find_at 0.1% 0.3% 41.7KiB unicode_normalization unicode_normalization::tables::composition_table 0.1% 0.3% 36.2KiB rand_hc rand_hc::hc128::Hc128Core::sixteen_steps 0.1% 0.3% 33.8KiB regex regex::re_unicode::Regex::shortest_match_at 0.1% 0.3% 32.4KiB rand_hc <rand_hc::hc128::Hc128Core as rand_core::block::BlockRngCore>::generate 0.1% 0.2% 31.4KiB ammonia html5ever::tokenizer::Tokenizer<Sink>::step 0.1% 0.2% 24.5KiB idna unicode_normalization::tables::canonical_combining_class 0.1% 0.2% 21.8KiB regex aho_corasick::ahocorasick::AhoCorasick<S>::find 0.1% 0.2% 21.6KiB regex aho_corasick::ahocorasick::AhoCorasick<S>::find 0.1% 0.2% 21.0KiB clap clap::app::parser::Parser::get_matches_with 0.1% 0.2% 20.3KiB ws ws::io::Handler<F>::handle_queue 0.1% 0.1% 19.0KiB ws ws::connection::Connection<H>::read_frames 0.1% 0.1% 18.7KiB env_logger termcolor::Ansi<W>::write_color 35.3% 91.6% 11.4MiB And 58189 smaller methods. Use -n N to show more. 38.6% 100.0% 12.5MiB .text section size, the file size is 32.4MiB

carterschonwald 7y ago

Seems like a lot of this comes from rust defaulting to allowing several different versions of a library to be linked in ... there’s def some other pieces. But that seems like a biggie

lwhsiao 7y ago
Do you know how to not do this "default"? I would love to try it.
- MaulingMonkey 7y ago
  So far in my toy projects I've mostly seen this crop up from depending on a lot of 0.x crates still going through significant version churn and legitimate breaking changes, where A and B legitimately can't use the same version of C as-is due to API changes, because they haven't been keeping their dependencies up to date. The fix is simple, when it happens: Patch A/B to use the latest major version of C, fixing the source code as necessary. You can [patch] locally until upstream accepts your Pull Request - which might include a https://deps.rs/repo/github/rust-lang/cargo https://deps.rs/repo/github/rust-lang/cargo badge in their README.md, to encourage them to continue to keep C up-to-date.
- JeanMertz 7y ago
  Clippy has an (off by default) lint to warn/deny using multiple versions of the same crate.
- carterschonwald 7y ago
  i would love to, too! :)

ajxs 7y ago

At the risk of being slightly tangential, I've been sorely wanting to air this particular grievance with Rust for some time. It's somewhat related, since the author mentions their package system. Its package ecosystem isn't nearly in the horrible state that node's is, but having a package system shouldn't be a substitute for designing a useful standard library for a language. I think that the attraction to 'small languages' is very much misplaced. If I can't get through Rust's official documentation without being recommended the use of third party packages for basic functionality (getopt, interfacing with static libraries... Etc) then the designers have made a terrible error.

Myrmornis 7y ago
Isn't it at least partly explained by wanting to allow implementations of this functionality to evolve more rapidly than they could if they were in the stdlib and thus reach good solutions more rapidly? getopt in particular seems like something that programmers haven't reached a consensus on despite 40 years of experimentation.
- ajxs 7y ago
  I would beg to differ with this. The kind of functionality expected from Getopt is fairly standard these days. I also disagree that features like getopt really need to 'evolve' much at this point in time. At the risk of courting controversy here, I'm going to confess that often I just don't want the community involved in the development of a language. On a project I inherited recently ( with the mission to save ) I unfortunately had to suffer using Typescript. If you look at the community discussions regarding the language's future direction you'll be treated to the uninformed arguing with the ignorant about the language's direction. Every time I download an external crate, I have to learn a different developer's way of doing things, suffer their idiosyncratic ways of creating an API and potentially expose myself or my application to a new set of vulnerabilities. The more I can get away with not doing this, the better.
  - sanderjd 7y ago
    Who is "the community" and who isn't? Are you sure you trust the core language developers more than people writing libraries?

ijiiijji1 7y ago

Maybe rust core/std/alloc should built with many smaller crates that are cached better and have fewer dependencies? Also: https://github.com/johnthagen/min-sized-rust https://github.com/johnthagen/min-sized-rust

geofft 7y ago

> For one, it’s common that you get different versions anyway (the Zola build currently has two versions each of unicase, parking_lot, parking_lot_core, crossbeam-deque, toml, derive_more, lock_api, scopeguard, and winapi). This seems like a specific thing that would be a measurable win and a not uncommon problem (e.g., I have a test kernel module that doesn't have many dependencies, and it still has two generic-arrays, two proc-macro2s, and two unicode-xids). Is there something that could be done here technically, such as pull requests to bump common crates to using the same version of even-more-common dependencies?

MaulingMonkey 7y ago
> generic-array This is up to 0.13.2... plenty of supposedly breaking API churn, no wonder you have two copies. Yes, your dependencies probably rely on different 0.x versions, and a pull request could fix that. You can inspect your Cargo.lock file to figure out which ones are to blame. > Is there something that could be done here technically, such as pull requests to bump common crates to using the same version of even-more-common dependencies? Yep, that's the fix. README.md badges like https://deps.rs/repo/github/rust-lang/cargo https://deps.rs/repo/github/rust-lang/cargo can alert people to out-of-date dependencies. You can even automate some of the pull requests with something like https://github.com/marketplace/dependabot-preview https://github.com/marketplace/dependabot-preview if you control the repositories.

Animats 7y ago

Do dynamic libraries really help? They only save space in the case where 1) you are running many different programs and 2) they all use the same libraries in the same versions. If you're running multiple copies of the same program, they share code, at least on Linux.

raphlinus 7y ago
Depends on the use case. I'm thinking about GUI, where on Linux you can write code that links against (say) Gtk, and thus can pull in lots of libraries while the executable itself is tiny. With the current state of the Rust ecosystem, you basically have to build the GUI toolkit and bundle it with your app.
deafcalculus 7y ago
Dynamic libs on the file system are awful. It's OK if the kernel wants to conserve RAM by sharing code pages that are identical, but saving hard disk space with dynamic libs is so painful.
erik_seaberg 7y ago
> running many different programs and they all use the same libraries in the same versions It's distro maintainers' job to test packaged apps and converge on a known-good version of a shared lib. This is only a problem if your distro doesn't package the apps you need, and you build from upstream source ignoring the distro and using all the random versions upstream happened to test.

leeoniya 7y ago

> Digging into xi-editor, the biggest single source of bloat is serde flatbuffers seems to be the go-to serialization lib, in terms of speed and mem efficiency: https://docs.rs/flatbuffers/0.6.0/flatbuffers/ https://docs.rs/flatbuffers/0.6.0/flatbuffers/

je42 7y ago

I don't understand the advice: "Use async sparingly" It doesn't make sense. Either your complete code base is async or it is not. If you have a single blocking code point in it. It is not async anymore, since it can't handle any other schedule async tasks while waiting for the blocking call.

raphlinus 7y ago
Probably could have worded it better. I didn't mean, "only use a little bit of async," I agree that doesn't make sense. I meant, "use async if your problem really needs it, otherwise avoid it."
- je42 7y ago
  The main thing it is you can't really add async when you need it, because if you have written enough sync code, turning it into async amounts to a major rewrite/refactor of the code including choices for dependencies, code for parallel code execution are different in async etc.

ogoffart 7y ago

It looks like the `fluent` library is to blame for the bloat here. I have myself made another localization library that use a simpler model and compiles quite fast when using the rust `gettext` backend: https://github.com/woboq/tr https://github.com/woboq/tr

jokoon 7y ago

I just don't feel like rust can replace C/C++, because the syntax is not simple enough. Rust has a lot of cool things, but to me syntax simplicity is more important. Maybe I have a hard time adapting myself to rust? Maybe my brain is too much "wired" to a C style syntax. It still seems that to me, C-style syntax is just better. I would rather prefer a C++-like language that breaks down from backward compatibility with C while keeping its simplicity, has STL containers, and is simpler to read and use. Rust is cool, but I'm just curious if it can really be adopted for large project to justify rewriting existing code.

timw4mail 7y ago
C++ has simple syntax? Since when?
- jokoon 7y ago
  Since you're not really required to use templates or inheritance at every corner.
floatboth 7y ago
Maybe you're looking for D? :) Anyway, what is "not simple" about Rust's syntax?
- jokoon 7y ago
  Using "let thing: type" seems bloated, I prefer the C style "type thing;" ; can also make a lot of difference, which is a little weird. I don't understand why there is both str and String, seems complicated for nothing. Option things are not really clear yet, and I don't understand what is their use, it seems like an alternative to unions, but few developers use unions anyway... Pattern matching seems powerful but I fail to understand its usage, and I'm a little skeptical about the machine code it generates. D is fine, but I think that even D is not simple enough. Rust seems like it's awesome, but it requires to entirely rethink how you write code, and I've never been a fan of high level abstraction.

kissgyorgy 7y ago

Serialization will always be slow for obvious reasons, that's why binary protocols and messages started to emerge (HTTP/2, gRPC) instead of serializing/deserializing everything to JSON and back.

floatboth 7y ago
grpc uses protobuf which still does traditional serialization. The cool stuff is capnp/flatbuffers, where you write to and read from the "serial" memory directly.

axilmar 7y ago

Since it's not possible for a Standard Library to have everything a programmer needs, programming languages shouldn't have Standard Libraries but Standard Repositories.

home_project123 7y ago

How many people would use a Rust cloud compiler? Suppose it cuts build time from 5 minutes to 30 seconds.... Technically, a transparently mirrored file system, a strong compiler cluster (memory, cores, etc). And some predictive ML. But you end up with a binary-equivalent (verifiable) output. Any thoughts ?

Hortinstein 7y ago
I would likely use this depending on how transparent it was and the pricing and if it supported RLS etc. I was writing a rust project on a terrible old laptop that took 2-5 minutes for compilation time. I ended up standing up a Digital Ocean instance and used VScode (via Coder https://github.com/cdr/code-server https://github.com/cdr/code-server ). And this ended up being the most workable solution. Cut down my compilation times significantly and also helped w/ VScode RAM usage. Not sure how the buisness model for a cloud compiler would work, but I would be interested.
Macha 7y ago
I wouldn't use it. Incremental builds aren't that bad, and I'd have a hard time trusting third party compiled libs enough to include in a release from a new service that hasn't built up trust the way that say a Linux distro has. While you could assume any maliciousness or security compromise would be caught, as you can see from the rubygems news today this is not instant and it adds another point of failure.

kanishkarj 7y ago

I'm naive to rust, but I do get your concern. For a small part in a project, I used a tokio based library. And it just exponentially increased my build time. What about if rust could support something like dynamic Linking/loading. Like have some crates globally installed and while building we could link to the global one instead of locally getting all the crates. Like C/C++does it right?

ameixaseca 7y ago

Good points, but I'd like to point out that nothing on this post is Rust-specific. You can have the same issue if programming in any other language, including C: excessive indirections, inefficient algorithms, bad abstractions, excessive use of unnecessary libraries, etc. It's indeed easier to "bloat" your resulting binary in C++ or Rust given the easiness to do higher-level abstractions; since you can more easily program complex solutions you also need to consider your design and the trade-offs on your code. I'd also like to point out that, in comparison: * Rust bloat is a speck compared to hundreds of megabytes for a similar Python program + runtime including the same amount of code. * Rust bloat can be mostly optimized away for systems that really care about excess/unused code - you have #![no_std], disabling of backtraces, aborting on panic, and a lot of other optimizations that throw away a big portion of extra functionality not needed for things like embedded. You have little alternative on things like Go besides removing debugging symbols and doing tricks like "dynamic decompression" (which could also be applied to Rust programs to further reduce their size, btw). Bottom line is: Rust makes it easier for you to "just add a new library" and it also make you more mindful of the bloat, but we need to keep it in perspective.

xpe 7y ago

To respond to one particular point from the post: > Once you accept bloat, it’s very hard to claw it back. If your project has multi-minute compiles, people won’t even notice a 10s regression in compile time. Then these pile up, and it gets harder and harder to motivate the work to reduce bloat, because each second gained in compile time becomes such a small fraction of the total. This particular problem can be addressed head-on, I think. It would seem feasible to have the compiler to distinguish between the target application (library) and the dependencies. Then it could report the compile times as separate values. This could be built into CI/CD tools as a way to catch application level compile time changes. Of course, this approach wouldn’t tell the entire story, but it would likely serve as a canary in the coal mine at least.