5 ms·

I'm not sure "new tooling" (if you're contrasting language-specific with linux distro tooling) does give more visibility though. How well does "new tooling" tra

I'm not sure "new tooling" (if you're contrasting language-specific with linux distro tooling) does give more visibility though. How well does "new tooling" track root certificates being up-to-date and available for example (ca-certificates is explicitly mentioned in the vlc dependencies in the link as an example)? Or across language/ecosystem boundaries? It would be interesting to compare vlc dependencies on Windows, MacOS and one of the BSDs, as unlike linux distros, there is a explicit and somewhat complete core (which would likely remove or at least reduce the number of "system" dependencies on the vlc list). I do think better tooling can exist (and is being developed), but it needs to track the whole system, not just a single language.

pornel 4y ago
Yes, I mean something like Cargo vs apt. If you depend on libcurl, you count that as 1 (one) dependency. Nice, and restrained. But curl has everything from gopher, FTP to SMTP and IMAP! It has a half a dozen TLS libraries, SSH, auth methods, multiple compression algorithms, socks proxies, etc, It's a lot of code, and a lot of transitive dependencies that you don't even think about. OTOH when you add `reqwest` in Cargo, Cargo will visibly fetch everything needed to build it from scratch, and it will look like 140 dependencies, not 1. But in total it's still only a subset of what libcurl contains. There's no Samba or POP3. It's just a very detailed list showing you it contains mundane things like a CLI arg parser and base64 encoder, and a wrapper for ca-certificates too. Please don't interpret it as a criticism of curl. It's just to contrast that you may see libcurl as 1 dependency, because it's one precompiled package, but you may see a smaller amount of code as extravagantly bloated shocking 140 dependencies, because it's just been split into tiny pieces and rubbed in your face.
- nonameiguess 4y ago
  Curl, even compiled against all of its optional dependencies to support every possible kind of auth, encryption scheme, compression, etc. is nowhere 140 dependencies, even transitively. All of Linux from Scratch is only 75 packages. In contrast, when I tried to add a token-replacement library to mdbook, Cargo installed 187 crates. There is a real and legitimate difference in the size and functionality scope of older C and C++ libraries with the type of single-function packages you see from NPM and Cargo that has made full supply-chain audits far more difficult. To get functionality equivalent to something like Boost, you might need to vet a thousand separate vendors, the majority of which are pseudonymous Github accounts with no physical address or backing organization. I'm not just speculating about this, either. I used to be a platform lead on fully-disconnected, classified information systems where we needed to bring in and compile every dependency from source via DVDs hand-carried to a classified workstation. Somewhere around 2015 or so, this stopped being even a remotely feasible job and created multi-year backlogs of teams that had something fully working in their unclassified development environment with Internet access suddenly finding themselves staring at a minimum 6-month, more likely indefinite wait to get to production when they discovered security teams needed to individually approve every package and every developer they wanted to bring onto a classified system.
- aragilar 4y ago
  But does not cargo bottom out at libraries like https://crates.io/crates/openssl-sys/0.9.75 https://crates.io/crates/openssl-sys/0.9.75, where you do need addition (non-managed) dependencies? I would be surprised that the more "conservative" linux distros didn't have a complete, deduplicated and unvendored tree (only encountering issues around binary firmware and similar), whereas from my minimal rust knowledge, cargo's reqwest metadata is incomplete. It's the incompleteness I'm suggesting is a problem (I'm more familiar with Python and wheels, where there are substantial non-Python dependencies people forget about), not the number of dependencies (though that's not great either).
  - pornel 4y ago
    The "system dependency" wrapper crates are unfortunately dependent on the environment outside of Cargo's view. The Rewrite-In-Rust Strike Force is working on eliminating this problem ;) Cargo/crates.io has a publicly available index of every crate with every dependency. `cargo tree` lists everything in your project. `Cargo.lock` contains inventory of every package used in the build, including their checksums. See https://lib.rs/stats https://lib.rs/stats

pornel 4y ago

Yes, I mean something like Cargo vs apt. If you depend on libcurl, you count that as 1 (one) dependency. Nice, and restrained. But curl has everything from gopher, FTP to SMTP and IMAP! It has a half a dozen TLS libraries, SSH, auth methods, multiple compression algorithms, socks proxies, etc, It's a lot of code, and a lot of transitive dependencies that you don't even think about. OTOH when you add `reqwest` in Cargo, Cargo will visibly fetch everything needed to build it from scratch, and it will look like 140 dependencies, not 1. But in total it's still only a subset of what libcurl contains. There's no Samba or POP3. It's just a very detailed list showing you it contains mundane things like a CLI arg parser and base64 encoder, and a wrapper for ca-certificates too. Please don't interpret it as a criticism of curl. It's just to contrast that you may see libcurl as 1 dependency, because it's one precompiled package, but you may see a smaller amount of code as extravagantly bloated shocking 140 dependencies, because it's just been split into tiny pieces and rubbed in your face.

nonameiguess 4y ago
Curl, even compiled against all of its optional dependencies to support every possible kind of auth, encryption scheme, compression, etc. is nowhere 140 dependencies, even transitively. All of Linux from Scratch is only 75 packages. In contrast, when I tried to add a token-replacement library to mdbook, Cargo installed 187 crates. There is a real and legitimate difference in the size and functionality scope of older C and C++ libraries with the type of single-function packages you see from NPM and Cargo that has made full supply-chain audits far more difficult. To get functionality equivalent to something like Boost, you might need to vet a thousand separate vendors, the majority of which are pseudonymous Github accounts with no physical address or backing organization. I'm not just speculating about this, either. I used to be a platform lead on fully-disconnected, classified information systems where we needed to bring in and compile every dependency from source via DVDs hand-carried to a classified workstation. Somewhere around 2015 or so, this stopped being even a remotely feasible job and created multi-year backlogs of teams that had something fully working in their unclassified development environment with Internet access suddenly finding themselves staring at a minimum 6-month, more likely indefinite wait to get to production when they discovered security teams needed to individually approve every package and every developer they wanted to bring onto a classified system.
aragilar 4y ago
But does not cargo bottom out at libraries like https://crates.io/crates/openssl-sys/0.9.75 https://crates.io/crates/openssl-sys/0.9.75, where you do need addition (non-managed) dependencies? I would be surprised that the more "conservative" linux distros didn't have a complete, deduplicated and unvendored tree (only encountering issues around binary firmware and similar), whereas from my minimal rust knowledge, cargo's reqwest metadata is incomplete. It's the incompleteness I'm suggesting is a problem (I'm more familiar with Python and wheels, where there are substantial non-Python dependencies people forget about), not the number of dependencies (though that's not great either).
- pornel 4y ago
  The "system dependency" wrapper crates are unfortunately dependent on the environment outside of Cargo's view. The Rewrite-In-Rust Strike Force is working on eliminating this problem ;) Cargo/crates.io has a publicly available index of every crate with every dependency. `cargo tree` lists everything in your project. `Cargo.lock` contains inventory of every package used in the build, including their checksums. See https://lib.rs/stats https://lib.rs/stats