5 ms·

> Another obvious advice is to use fewer, smaller dependencies. As a library author, I sometimes wonder if I should be introducing more feature flags into my c

> Another obvious advice is to use fewer, smaller dependencies. As a library author, I sometimes wonder if I should be introducing more feature flags into my crate to let people only use the bits that they want, in order to help compile times. The whole thing doesn't take very long to compile (~20 seconds), but maybe the added complexity of juggling feature flags might be worth it.

nicoburns 5y ago
You think 20 seconds isn't a long compile time!?
- axiosgunnar 5y ago
  I don't see the problem. Since it's an external crate, it will compile exactly once after adding it to Cargo.toml, and will be cached forever.
  - unanswered 5y ago
    I'm convinced that most of the people complaining about build times are using docker for some incomprehensible reason and are blowing away the dependency cache with every build.
    - rowanG077 5y ago
      This. It's annoying to compile from scratch. But I almost never have to do that. So I don't really have a problem with compile times.
    - nicoburns 5y ago
      Nope. FWIW, I conpile on a MacBook. So I only have 2 cores, and laptop ones at that. Perhaps that's affecting our difference in perspective? I also had a project a couple of years ago where I had to compile it in release mode to get reasonable performance for testing (Iw as processing GB's of data), which meant 2-3min incremental compile times. Not fun!
  - kzrdude 5y ago
    forever, or until rust version update (might be often for nightly builds), or until build flag change or cargo clean, whichever happens first. It's also only a per-project target directory (not across all projects you compile).
    - kibwen 5y ago
      Even for people on nightly builds of the compiler, I would be quite surprised if nearly anyone actually bothered to upgrade their compiler every night. Most people on nightly are just going to be there for the use of a specific unstable feature.
  - finnb 5y ago
    Cached locally, sure. The point of bazel with remote execution and remote caching is that it's then available in the cloud for everyone to access.
- jeroenhd 5y ago
  This depends on what the library does, of course. From a quick Google I believe the parent has made several packages, like a front-end to help with XML trees or a full wrapper around procfs. Libraries that complex (especially with their dependencies) can take significant time to compile, it's just a consequence of their complexity. Some smaller libraries I've seen that just do some annoying-to-test but simple string manipulations have had similar compile times, mostly because they include specific versions of advanced logging or error reporting libraries for no obvious reason.
  - eminence32 5y ago
    Yup, that's me :) For example in the `procfs` crate, there's a dependency on `flate2` just in case a /boot/config.gz file exists (in order to see what options the kernel was built with). But if a consumer never needs to inspect the kernel options, this dependency (and related code) is just dead-weight. The nice thing about rust is that this code will be removed from the final executable if it's unused, but maybe it's time to start thinking more carefully about compile times.
- deleted 5y ago
  [deleted]
- alkonaut 5y ago
  It all depends on how many times you do it! 20 seconds for an edit-compile loop is huge. 20 seconds for a CI build is not as bad, but still bad (if you have dozens more). 20 seconds once per machine per dependency version (i.e. there is no rebuild of the dependency until you change compiler flags or update the library) is OK.
jeroenhd 5y ago
I'm just one voice (and I'm probably not someone who uses your specific library), but this does sound like a good idea. I'd appreciate it (though I understand that this takes effort and maintenance, so I understand if you don't want to!) Your 20 second library is not too bad when it's used on its own, but add 5 or 6 libraries that all "only" take a few seconds and you're waiting a solid five minutes compiling other people's code and their dependencies. Even simple Rust programs quickly end up compiling several versions of several libraries because of the dependency tree, some of which can include large libraries like Tokio and Serde. I love libraries that make use of feature flags, they massively cut down on my compile time. This isn't really any library developers' fault, it's a problem with the entire ecosystem. Cargo is not dissimilar from NPM where you can quickly end up with an exploded dependency chain of hundreds of megabytes in supporting code before you even realize.
mplanchard 5y ago
Fwiw unless the flags really represent independent bits of functionality, I find many features more of a pain to deal with than dependency compile times, given that even in our big workspace, a combination of incremental builds and sccache makes compile times mostly a non issue.
- eminence32 5y ago
  Yeah, the position you articulated is my main concern. Do you think it makes any differences if the features are enabled-by-default? If so, the "default" experience isn't any different than it is today, but discerning consumers can disable things if they need it. But also, an "everything-by-default" approach doesn't seem that helpful to the global ecosystem.
  - mplanchard 5y ago
    Yeah I think default features resolve a fair bit of the pain. I wonder though how many consumers are actually going to go through the hassle of slicing just what they need out of a library. For something as large and featureful as Tokio it makes sense, or for optional additions to core functionality, but we at least haven’t yet found ourselves going around and looking for places to trim the fat. One dependency like diesel or something similar eats up easily 10x the compile time of any other single dependency, so we’ve been more focused on rooting out those big offenders when we can. I definitely wouldn’t want to ask that OSS maintainers take on extra maintenance burden unless there’s a good reason. Features are super neat, but it’s easy to see them overused. It’s also a bit hard to adequately test arbitrary combinations of features. It’s a different concern from a library, but we saw this with one of our applications. We had a situation where we were using features where we should have just been using env vars for feature flagging, and it caused a fair bit of hassle. We’ve since ripped out that usage, because we never in practice tested or deployed with anything other than “all features,” and even in the case where we want to use some functionality to upsell, enabling an env var and restarting is way easier than recompiling the project.
dochtman 5y ago
I only use feature flags for features that require an additional dependency, or for features that really require a whole lot of extra code.
edflsafoiewq 5y ago
Doesn't this basically require buy-in across the entire crate tree? Eg even if I disable all the default features I don't need, all it takes is a single crate anywhere in the dep tree that uses the same crate with the default features to turn them on again?
stormbrew 5y ago
As someone who works on a large rust codebase with a lot of moving parts, I'd ask you to really think hard about how your use of features might affect people down the line in undesireable ways if you go down this path. In particular, please for the love of god don't put the whole kitchen sink into the default feature. I'd even argue don't ever have a default feature at all. In practice, once a codebase gets above a specific size, it becomes completely impossible to exclude any features in a crate's default list because `default-features=false` only applies locally. Something 5 levels deep will bring those features in whether I like it or not just by not specifying anything. Also, please don't do things like how slog uses a feature flag to control minimum log levels, or k8s-openapi uses it for version selection. These kinds of usages create extremely frustrating problems. Honestly, features -- and especially the way default features work -- are the worst thing about cargo to me. I think they should be a lot more restricted than they currently are. If I had my way you'd only be able to use `#[cfg(feature = X)]` at module level to include or exclude code, never to select actual behaviour. To me they're filling a hole that crates themselves should be filling. Certain language features aren't really rich enough to make really effective 'additive' crates though, unfortunately.
- matklad 5y ago
  > In practice, once a codebase gets above a specific size, it becomes completely impossible to exclude any features in a crate's default list because `default-features=false` only applies locally Curious, what that specific size approximately is? For rust-analyzer, getting all the deps to use features correctly is rather annoying, but isn’t particularly time-consuming.
  - stormbrew 5y ago
    I think we must be talking about different things because rust-analyzer doesn't enter into or help the issue I'm talking about? If you depend on crates A and B, and B depends on A, and you specify `default-features=false` on A but B doesn't, you will always get A's default features included in your compile. There is no way to prevent this.

nicoburns 5y ago

You think 20 seconds isn't a long compile time!?

axiosgunnar 5y ago
I don't see the problem. Since it's an external crate, it will compile exactly once after adding it to Cargo.toml, and will be cached forever.
- unanswered 5y ago
  I'm convinced that most of the people complaining about build times are using docker for some incomprehensible reason and are blowing away the dependency cache with every build.
  - rowanG077 5y ago
    This. It's annoying to compile from scratch. But I almost never have to do that. So I don't really have a problem with compile times.
  - nicoburns 5y ago
    Nope. FWIW, I conpile on a MacBook. So I only have 2 cores, and laptop ones at that. Perhaps that's affecting our difference in perspective? I also had a project a couple of years ago where I had to compile it in release mode to get reasonable performance for testing (Iw as processing GB's of data), which meant 2-3min incremental compile times. Not fun!
- kzrdude 5y ago
  forever, or until rust version update (might be often for nightly builds), or until build flag change or cargo clean, whichever happens first. It's also only a per-project target directory (not across all projects you compile).
  - kibwen 5y ago
    Even for people on nightly builds of the compiler, I would be quite surprised if nearly anyone actually bothered to upgrade their compiler every night. Most people on nightly are just going to be there for the use of a specific unstable feature.
- finnb 5y ago
  Cached locally, sure. The point of bazel with remote execution and remote caching is that it's then available in the cloud for everyone to access.
jeroenhd 5y ago
This depends on what the library does, of course. From a quick Google I believe the parent has made several packages, like a front-end to help with XML trees or a full wrapper around procfs. Libraries that complex (especially with their dependencies) can take significant time to compile, it's just a consequence of their complexity. Some smaller libraries I've seen that just do some annoying-to-test but simple string manipulations have had similar compile times, mostly because they include specific versions of advanced logging or error reporting libraries for no obvious reason.
- eminence32 5y ago
  Yup, that's me :) For example in the `procfs` crate, there's a dependency on `flate2` just in case a /boot/config.gz file exists (in order to see what options the kernel was built with). But if a consumer never needs to inspect the kernel options, this dependency (and related code) is just dead-weight. The nice thing about rust is that this code will be removed from the final executable if it's unused, but maybe it's time to start thinking more carefully about compile times.
deleted 5y ago
[deleted]
alkonaut 5y ago
It all depends on how many times you do it! 20 seconds for an edit-compile loop is huge. 20 seconds for a CI build is not as bad, but still bad (if you have dozens more). 20 seconds once per machine per dependency version (i.e. there is no rebuild of the dependency until you change compiler flags or update the library) is OK.

jeroenhd 5y ago

I'm just one voice (and I'm probably not someone who uses your specific library), but this does sound like a good idea. I'd appreciate it (though I understand that this takes effort and maintenance, so I understand if you don't want to!) Your 20 second library is not too bad when it's used on its own, but add 5 or 6 libraries that all "only" take a few seconds and you're waiting a solid five minutes compiling other people's code and their dependencies. Even simple Rust programs quickly end up compiling several versions of several libraries because of the dependency tree, some of which can include large libraries like Tokio and Serde. I love libraries that make use of feature flags, they massively cut down on my compile time. This isn't really any library developers' fault, it's a problem with the entire ecosystem. Cargo is not dissimilar from NPM where you can quickly end up with an exploded dependency chain of hundreds of megabytes in supporting code before you even realize.

mplanchard 5y ago

Fwiw unless the flags really represent independent bits of functionality, I find many features more of a pain to deal with than dependency compile times, given that even in our big workspace, a combination of incremental builds and sccache makes compile times mostly a non issue.

eminence32 5y ago
Yeah, the position you articulated is my main concern. Do you think it makes any differences if the features are enabled-by-default? If so, the "default" experience isn't any different than it is today, but discerning consumers can disable things if they need it. But also, an "everything-by-default" approach doesn't seem that helpful to the global ecosystem.
- mplanchard 5y ago
  Yeah I think default features resolve a fair bit of the pain. I wonder though how many consumers are actually going to go through the hassle of slicing just what they need out of a library. For something as large and featureful as Tokio it makes sense, or for optional additions to core functionality, but we at least haven’t yet found ourselves going around and looking for places to trim the fat. One dependency like diesel or something similar eats up easily 10x the compile time of any other single dependency, so we’ve been more focused on rooting out those big offenders when we can. I definitely wouldn’t want to ask that OSS maintainers take on extra maintenance burden unless there’s a good reason. Features are super neat, but it’s easy to see them overused. It’s also a bit hard to adequately test arbitrary combinations of features. It’s a different concern from a library, but we saw this with one of our applications. We had a situation where we were using features where we should have just been using env vars for feature flagging, and it caused a fair bit of hassle. We’ve since ripped out that usage, because we never in practice tested or deployed with anything other than “all features,” and even in the case where we want to use some functionality to upsell, enabling an env var and restarting is way easier than recompiling the project.

dochtman 5y ago

I only use feature flags for features that require an additional dependency, or for features that really require a whole lot of extra code.

edflsafoiewq 5y ago

Doesn't this basically require buy-in across the entire crate tree? Eg even if I disable all the default features I don't need, all it takes is a single crate anywhere in the dep tree that uses the same crate with the default features to turn them on again?

stormbrew 5y ago

As someone who works on a large rust codebase with a lot of moving parts, I'd ask you to really think hard about how your use of features might affect people down the line in undesireable ways if you go down this path. In particular, please for the love of god don't put the whole kitchen sink into the default feature. I'd even argue don't ever have a default feature at all. In practice, once a codebase gets above a specific size, it becomes completely impossible to exclude any features in a crate's default list because `default-features=false` only applies locally. Something 5 levels deep will bring those features in whether I like it or not just by not specifying anything. Also, please don't do things like how slog uses a feature flag to control minimum log levels, or k8s-openapi uses it for version selection. These kinds of usages create extremely frustrating problems. Honestly, features -- and especially the way default features work -- are the worst thing about cargo to me. I think they should be a lot more restricted than they currently are. If I had my way you'd only be able to use `#[cfg(feature = X)]` at module level to include or exclude code, never to select actual behaviour. To me they're filling a hole that crates themselves should be filling. Certain language features aren't really rich enough to make really effective 'additive' crates though, unfortunately.

matklad 5y ago
> In practice, once a codebase gets above a specific size, it becomes completely impossible to exclude any features in a crate's default list because `default-features=false` only applies locally Curious, what that specific size approximately is? For rust-analyzer, getting all the deps to use features correctly is rather annoying, but isn’t particularly time-consuming.
- stormbrew 5y ago
  I think we must be talking about different things because rust-analyzer doesn't enter into or help the issue I'm talking about? If you depend on crates A and B, and B depends on A, and you specify `default-features=false` on A but B doesn't, you will always get A's default features included in your compile. There is no way to prevent this.