7 ms·

> 2) Rust's ecosystem makes it easy to find and integrate solutions not offered by the standard API This is a double-edged sword. I'm a huge fan of Rust's too

by skohan 6y ago

> 2) Rust's ecosystem makes it easy to find and integrate solutions not offered by the standard API This is a double-edged sword. I'm a huge fan of Rust's tooling, but when it's so easy to add dependencies, this inevitably leads to deep dependency trees, slower compile times, and less knowledge about what's actually happening inside most rust code bases. Often times when asking about how to solve a problem in Rust, the answer comes in the form of "use crate X" rather than an explanation of how to solve the problem using the language itself.

serial_dev 6y ago
I think it's important to say the silent part out loud: "just use crate x and you can look up how the crate works behind the scenes if you are curious".
hannofcart 6y ago
This statement is a bit confusing: are you saying that there is such a thing as too much code reuse? That it should be necessarily a bit difficult to integrate a third party library in your codebase?
- skohan 6y ago
  There's always a balance point. For something like JSON parsing, or an HTTP client, for sure I want to use a library. But when code reuse means pulling in a giant library to do one specific thing, or else introducing a runtime like tokio where it otherwise might not be needed, I would probably rather implement a small tailored solution specifically for my use-case.
  - Measter 6y ago
    In one sense, the way that larger crates in Rust are fragmented does address some of that. This does result in large dependency trees, but it does also mean that if you need that one specific functionality, you may not need to pull in the full library, but only that smaller section.
  - mypalmike 6y ago
    Ideally, there should be no harm in pulling in a library for just 1% of its functionally. A combination of careful library design and tooling should take care of dependency bloat ("shaking the tree"). In practice of course, libraries don't always consider this aspect, and the tooling never seems quite up to the task.
    - skohan 6y ago
      Yeah I think it’s also a matter of API design. Like a crate which by necessity has to handle the “general case” to some degree will often have more API complexity than a solution which is tailored to one specific use-case. For instance, you might have to concern yourself with configuration parameters which have zero relevance to your use-case.
fractionalhare 6y ago
> Often times when asking about how to solve a problem in Rust, the answer comes in the form of "use crate X" rather than an explanation of how to solve the problem using the language itself. Do you think this is a bad thing? How frequently would you say it's a better use of a software engineer's time to carefully reimplement a commoditized library that already exists for a speed improvement or to understand it? I don't really understand arguments that you shouldn't introduce convenience and quality of life features because programmers will lean on them too much. Leaning on them is the point; the implicit thesis is that programmers no longer have to understand what's happening under the hood. It's specialization and division of labor applied to programming. And if you really care, you can probably just look at the source code or pick up a book to learn how it works?
- colmmacc 6y ago
  For almost any real world application, where people's data and businesses are at stake, developer convenience is a distant priority to security. Of course it's great when we can have both, and making secure solutions the most convenient is key, but Rust doesn't quite get this right yet. If there's a security issue in one of those dependencies you are importing, how do you even know that you are impacted? What about when it's a dependency of a dependency? Making it easy for developers to suck in a huge TCB at build time has some serious downsides.
  - jamesmishra 6y ago
    When security is a concern, developers can simply avoid using untrusted crates. But concerns over security are not reason for a package manager to not even exist. Pretty much every major language and operating system has a package manager. While supply chain attacks are real, they are not unique to Rust in any way.
  - mplanchard 6y ago
    The ecosystem also includes stuff like cargo-audit[0], with which you can easily add dependency auditing to your CI pipelines or whatever. People are definitely thinking about this problem. [0]: https://github.com/rustsec/cargo-audit https://github.com/rustsec/cargo-audit
vvanders 6y ago
As someone who's spent a few decades working in C++ codebases I'm not sure how this is a Rust specific problem. Last large-scale C++ codebase I worked on took 60+ minutes to compile when you touched the precompiled header. In fact, I've found it to be common to have binary-only library dependencies in C++ which makes it harder to tell what those downstream dependencies do.
- allo37 6y ago
  I think you just run into it faster in Rust: I tried making a simple REST HTTP server (like 60 LoC) using actix-web and it pulled in over 200 sub-dependencies and took something like 10 minutes to compile. This toy program written using Boost.Beast or something similar would compile much faster. Of course I would have spent all day setting up the library instead of waiting for it to compile...
akiselev 6y ago
The worst part is how easy it is to add derive and proc macros that kill compile times. I have a ~3k LOC project that blows up to over 45k LOC after `cargo expand` due to a dozen or so macros and the compile time is really starting to hurt iteration speed. Sadly, macros are by far one of my favorite Rust features. I ended up paying for Clion just to get better debugger integration and a quick action for cargo expanding a file in the UI so I can copy paste the macro results into my codebase. I'm hoping that will improve incremental compilation times until I buy a Threadripper workstation.
- brundolf 6y ago
  Does the expanded code take longer to compile than the equivalent handwritten code? Is it just the expansion itself that takes a long time? In the latter case, maybe cargo needs an intermediate macro-expansion cache (instead of just a crate-level build cache)
  - TheCoelacanth 6y ago
    Often, you would write a lot less code if you were doing it by hand.
- mrlonglong 6y ago
  I adore CLion, it's brilliant for coding. At the moment I'm using it with Python and Git.
brundolf 6y ago
> this inevitably leads to deep dependency trees, slower compile times I've never thought that "having more friction when adding dependencies" is a valid strategy for preventing overuse of third-party code. Unlike other "trust the developer" scenarios, dependency bloat is a minor issue at best and is very easy to detect and diagnose. IMO you should cut down your dependencies when they actually become a problem that's bigger than the one they're solving. > and less knowledge about what's actually happening inside most rust code bases I actually think the package ecosystem (along with macros) is crucial for the breadth and accessibility that Rust provides. Rust is a low-level language, but you have high-level libraries at your fingertips if that's the level where you need to be working. This means that instead of writing hot-paths in a low level language and application code in a high-level language (and dealing with the added complexity in terms of FFI, builds, tooling, developer knowledge, etc), it's very possible to just write the entire thing in Rust. Again in this case, I think "forcing devs to learn how everything works" is a weak argument for increasing the friction for adding dependencies. I'm pretty sure that everything on crates.io is required to include source, not just a binary, so it's trivial to dive in and learn how it works if you're so inclined.
agumonkey 6y ago
I'm not a rust/cpp guy but I have a feeling that people skilled enough to need rust or cpp can probably use public libs for rapid prototyping and then recreate a smaller version if needs be.
throwaway189262 6y ago
Compile times in Rust are really the only problem with this. Rust with LTO on is really good about removing unused code, so it doesn't hurt you during runtime Whenever you run anything there's millions of lines involved across the OS API's directly or indirectly, so in practice it doesn't matter what's happening behind the scenes. In languages with fast compile times like Go and Java people add dependencies without even caring. And I think that's how it should be. All our Java backends at work are like 200MB and it doesn't matter and nobody cares. It still compiles in less than ten seconds, and all that code barely affects startup time.

by skohan 6y ago

serial_dev 6y ago

I think it's important to say the silent part out loud: "just use crate x and you can look up how the crate works behind the scenes if you are curious".

hannofcart 6y ago

This statement is a bit confusing: are you saying that there is such a thing as too much code reuse? That it should be necessarily a bit difficult to integrate a third party library in your codebase?

skohan 6y ago
There's always a balance point. For something like JSON parsing, or an HTTP client, for sure I want to use a library. But when code reuse means pulling in a giant library to do one specific thing, or else introducing a runtime like tokio where it otherwise might not be needed, I would probably rather implement a small tailored solution specifically for my use-case.
- Measter 6y ago
  In one sense, the way that larger crates in Rust are fragmented does address some of that. This does result in large dependency trees, but it does also mean that if you need that one specific functionality, you may not need to pull in the full library, but only that smaller section.
- mypalmike 6y ago
  Ideally, there should be no harm in pulling in a library for just 1% of its functionally. A combination of careful library design and tooling should take care of dependency bloat ("shaking the tree"). In practice of course, libraries don't always consider this aspect, and the tooling never seems quite up to the task.
  - skohan 6y ago
    Yeah I think it’s also a matter of API design. Like a crate which by necessity has to handle the “general case” to some degree will often have more API complexity than a solution which is tailored to one specific use-case. For instance, you might have to concern yourself with configuration parameters which have zero relevance to your use-case.

fractionalhare 6y ago

> Often times when asking about how to solve a problem in Rust, the answer comes in the form of "use crate X" rather than an explanation of how to solve the problem using the language itself. Do you think this is a bad thing? How frequently would you say it's a better use of a software engineer's time to carefully reimplement a commoditized library that already exists for a speed improvement or to understand it? I don't really understand arguments that you shouldn't introduce convenience and quality of life features because programmers will lean on them too much. Leaning on them is the point; the implicit thesis is that programmers no longer have to understand what's happening under the hood. It's specialization and division of labor applied to programming. And if you really care, you can probably just look at the source code or pick up a book to learn how it works?

colmmacc 6y ago
For almost any real world application, where people's data and businesses are at stake, developer convenience is a distant priority to security. Of course it's great when we can have both, and making secure solutions the most convenient is key, but Rust doesn't quite get this right yet. If there's a security issue in one of those dependencies you are importing, how do you even know that you are impacted? What about when it's a dependency of a dependency? Making it easy for developers to suck in a huge TCB at build time has some serious downsides.
- jamesmishra 6y ago
  When security is a concern, developers can simply avoid using untrusted crates. But concerns over security are not reason for a package manager to not even exist. Pretty much every major language and operating system has a package manager. While supply chain attacks are real, they are not unique to Rust in any way.
- mplanchard 6y ago
  The ecosystem also includes stuff like cargo-audit[0], with which you can easily add dependency auditing to your CI pipelines or whatever. People are definitely thinking about this problem. [0]: https://github.com/rustsec/cargo-audit https://github.com/rustsec/cargo-audit

vvanders 6y ago

As someone who's spent a few decades working in C++ codebases I'm not sure how this is a Rust specific problem. Last large-scale C++ codebase I worked on took 60+ minutes to compile when you touched the precompiled header. In fact, I've found it to be common to have binary-only library dependencies in C++ which makes it harder to tell what those downstream dependencies do.

allo37 6y ago
I think you just run into it faster in Rust: I tried making a simple REST HTTP server (like 60 LoC) using actix-web and it pulled in over 200 sub-dependencies and took something like 10 minutes to compile. This toy program written using Boost.Beast or something similar would compile much faster. Of course I would have spent all day setting up the library instead of waiting for it to compile...

akiselev 6y ago

The worst part is how easy it is to add derive and proc macros that kill compile times. I have a ~3k LOC project that blows up to over 45k LOC after `cargo expand` due to a dozen or so macros and the compile time is really starting to hurt iteration speed. Sadly, macros are by far one of my favorite Rust features. I ended up paying for Clion just to get better debugger integration and a quick action for cargo expanding a file in the UI so I can copy paste the macro results into my codebase. I'm hoping that will improve incremental compilation times until I buy a Threadripper workstation.

brundolf 6y ago
Does the expanded code take longer to compile than the equivalent handwritten code? Is it just the expansion itself that takes a long time? In the latter case, maybe cargo needs an intermediate macro-expansion cache (instead of just a crate-level build cache)
- TheCoelacanth 6y ago
  Often, you would write a lot less code if you were doing it by hand.
mrlonglong 6y ago
I adore CLion, it's brilliant for coding. At the moment I'm using it with Python and Git.

brundolf 6y ago

> this inevitably leads to deep dependency trees, slower compile times I've never thought that "having more friction when adding dependencies" is a valid strategy for preventing overuse of third-party code. Unlike other "trust the developer" scenarios, dependency bloat is a minor issue at best and is very easy to detect and diagnose. IMO you should cut down your dependencies when they actually become a problem that's bigger than the one they're solving. > and less knowledge about what's actually happening inside most rust code bases I actually think the package ecosystem (along with macros) is crucial for the breadth and accessibility that Rust provides. Rust is a low-level language, but you have high-level libraries at your fingertips if that's the level where you need to be working. This means that instead of writing hot-paths in a low level language and application code in a high-level language (and dealing with the added complexity in terms of FFI, builds, tooling, developer knowledge, etc), it's very possible to just write the entire thing in Rust. Again in this case, I think "forcing devs to learn how everything works" is a weak argument for increasing the friction for adding dependencies. I'm pretty sure that everything on crates.io is required to include source, not just a binary, so it's trivial to dive in and learn how it works if you're so inclined.

agumonkey 6y ago

I'm not a rust/cpp guy but I have a feeling that people skilled enough to need rust or cpp can probably use public libs for rapid prototyping and then recreate a smaller version if needs be.

throwaway189262 6y ago

Compile times in Rust are really the only problem with this. Rust with LTO on is really good about removing unused code, so it doesn't hurt you during runtime Whenever you run anything there's millions of lines involved across the OS API's directly or indirectly, so in practice it doesn't matter what's happening behind the scenes. In languages with fast compile times like Go and Java people add dependencies without even caring. And I think that's how it should be. All our Java backends at work are like 200MB and it doesn't matter and nobody cares. It still compiles in less than ten seconds, and all that code barely affects startup time.