7 ms·

Google open-sources Rust crate audits

256 points by taintegral 3y ago

mr_00ff00 3y ago
Curious if any senior devs on HN can comment on the importance/effectiveness of audits for crates? I’m a junior C++ dev that dabbles with rust in my free time, and I always feel a bit nervous when pulling huge dependency trees with tons of crates into projects. I would assume most places would turn away from the “node.js” way of doing these things and would just write internal versions of things they need. Again I am junior, so maybe my worries are way over blown.
- lesuorac 3y ago
  > I would assume most places would turn away from the “node.js” way of doing these things and would just write internal versions of things they need. Incorrect assumption, look up the left pad fiasco [1]. Its importance is really a personal opinion; convince nearly always trumps security so if the NPM way allows you to increase sales by ~10% you'll see people continuing to do it. Google is fairly principled though, all of the 3p code is internally vendored and supposed to be audited by the people pulling in that code/update. [1]: https://www.google.com/search?q=leftpad+broke+the+internet https://www.google.com/search?q=leftpad+broke+the+internet
- pclmulqdq 3y ago
  I think in a lot of C++ and ex-C++ orgs you see this sentiment a lot, and sometimes for good reason. Sometimes that code has security or performance reasons to worry about this. On the other hand, it often doesn't. On the other hand, Python folks and JavaScript users (which make up a lot of emigres to Rust) probably don't care enough about their supply chain. That's how you end up with misspelled packages causing viruses in production and other disasters. The short answer to this is that it actually depends a lot on what you are doing.
  - mr_00ff00 3y ago
    This surprises me that most people that use rust come from python and JavaScript. I would think the reason rust is so popular is from people moving from C and C++ and getting all the nice modern features to do systems with. Python and JavaScript people I would imagine find rust annoying since it’s all the niceties they are use to but with a bunch of rules on top.
    - efnx 3y ago
      I see people coming to Rust from all angles. It’s a nice sweet spot. I came to it from Haskell and on my team of three the other two devs came to Rust from C++. I can opine the motivation looks something like this: * From Haskell - looking for a strong type system (with sum types, typeclasses) but is “widely accepted”. No GC. * From C++ - looking for low-level capabilities (pointers, references) with improved safety. Improved manual memory management. * From Python/JS - looking for performance with a familiar feeling ecosystem and a welcoming community I think the Python and JS folks will have the hardest go of it, but they also have the most to gain.
    - david2ndaccount 3y ago
      Python and Javascript programmers so massively outnumber all other programmers that they are the majority of people converting to any language.
    - deathanatos 3y ago
      I went Python → Rust. Rust is high-level enough for me to remain competitively productive with Python, and the type-system just helps so much. I can't tell you how nice it has been to not have to see "Object of type NoneType has no attribute 'foo'" anymore. Also, def foo(obj): # ... And me wondering "what is 'obj'?" and then pulling that root … and pulling … and pulling …. Also "data", another just fabulous name for a variable, really narrows the possibilities. Even once you know the "type" of the variable, oftentimes I'd find that the type definition would subtle shift and morph in different parts of the program: they'd all want a Duck, but have varying opinions on what a Duck actually is. You cannot commit such BS with an actual type-checker. There is more up-front work with the compiler, but it pays off: the code that passes the compiler is of much better quality. Also, Option<T> is very nice to have when you need an actual Option<T> (i.e., generically), which Python lacks. (No, `None` is not it: You run into problems when T == Option<U>, and you have None and Some(None) — Python's cannot differentiate between the two.) Also sum types in general.
- nemothekid 3y ago
  The "node.js" way of doing things, and it's dysfunction, is nearly exclusive to node because Javascript lacks a standard library and npm's haphazard way of running things. Java, Ruby, Python, even my grandfather's Perl have had "modules" for years with none of the fear that is typically associated with Node. Personally, C++ aversion to sane dependency management is more about C++'s "I know better than you" culture and legacy cruft (packages are usually managed by the distro, not the language) than actually having any serious security implications.
  - bombolo 3y ago
    in a way rust's standard library is close to node's than python's. You can't really do much without getting some crates in.
  - pjmlp 3y ago
    This is slowly changing wiht conan and vcpkg increasing adoption. Still most environments I worked on, always had internal repos for packages, no CI/CD server talks to the outside world and vendoring isn't allowed.
- wongarsu 3y ago
  Writing your own version of everything means it's probably more tuned to your needs. But unless it's a core part of your software it will also be worse because you can't justify putting many resources into it. It also means new hires will have to learn a lot more. It's one of the (many) reasons why it's so hard to onboard into C/C++ projects, because every standard building block is bespoke and somehow different than what everyone else does. Of course if you are really big you just have those resources, which is why Meta or Google can have bespoke everything. On security it's a tradeoff. The open-source version is an easier target for attackers, but might be much more battle-tested and thus more bug-free. Audits are the attempt to have the best of both worlds here, and since they again can be crowd-sourced (with cargo-vet and cargo-cev both working on this) it scales even for companies that aren't Google-sized.
- fcantournet 3y ago
  Dependencies are dependencies in rust as in C++. I found it's extremely rare that homegrown library that have similar functionality to (used) open-source libraries are better from a security stand-point. At least in Rust a large part of the security issues that would be VERY time consuming to audit at scale through your dependency tree (whether internal or public) are covered by the compiler/borrow checker/type-system. In that sense I would take on an larger amount of dependency in Rust than I would in C++ while sleeping better.
- bombolo 3y ago
  > I would assume most places would turn away from the “node.js” way of doing these things and would just write internal versions of things they need. I assume most places don't care.
- pornel 3y ago
  I've reviewed hundreds of Rust crates. It's tedious and boring. The results are boring too — their code is mostly good! Big dependency trees have a reputation for being hot garbage, but that's not my experience. In Rust the small focused crates tend to do one thing, and do it well.
vitorsr 3y ago
See also Google Cloud’s Assured Open Source Software service: https://cloud.google.com/assured-open-source-software https://cloud.google.com/assured-open-source-software
sneed_chucker 3y ago
What's the state of supply chain auditing in general these days?
- adolph 3y ago
  SE Daily podcast episode "CAP Theorem 23 Years Later with Eric Brewer" touches on it a bit. On the whole an excellent episode. https://softwareengineeringdaily.com/2023/05/12/cap-theorem-23-years-later/ https://softwareengineeringdaily.com/2023/05/12/cap-theorem-...
progbits 3y ago
Can someone explain why cargo-vet doesn't include a cryptographic hash of the crate contents? My understanding is that this repository, and similar ones from Mozilla and others, says: "I, person X from trustworthy organization Y, have reviewed version 1.0 of crate foo and deemed it legit" (for a definition of trustworthy and legit). But now how does that help me if I want to be careful about what I depend on and supply-chain attacks? I ask for version 1.0 of crate foo but might get some malicious payload without knowing it.
- hobofan 3y ago
  That's already prevented by the checksum which is present for all crate versions in the registry index, which is set in stone on publish and verified by cargo on download. See e.g. https://github.com/rust-lang/crates.io-index/blob/74f1b1e064bc63a3be7026165d0b39ea88baf515/se/rd/serde https://github.com/rust-lang/crates.io-index/blob/74f1b1e064...
  - progbits 3y ago
    Hmm, but then you have to trust 1) github, 2) anyone with commit access to that repository. It's not the worst thing I suppose: #1 is a problem anyway for trusting Google/Mozilla's repo of audits, and #2 can be noticed by others so hard to pull of some supply chain attack that way. But I would still feel more confident if the audit log contained a copy of the checksum, and ideally itself was signed with author's keys.
- pornel 3y ago
  https://lib.rs/cargo-crev https://lib.rs/cargo-crev does this, with the entire chain from the crate data to the reviewer's trusted identity. However, this adds a lot of complexity. cargo-vet went for the other extreme of being super simple. To fill in their review report you don't even need any tooling.
deleted 3y ago
[deleted]
baby 3y ago
Interesting stuff! I think everyone seems to come up with their own solutions. I think security in general is a matter of who you trust, and things only work when we build a network of trust. Imagine if all companies and rust developers started sharing what crates they were confident in + what other organizations they trust as well. If you could then create your own set of such companies, and then choose a dependency depth you were willing to go down to, you might be able to quickly vet a number of crates this way, or at least see the weird crates that demand a bit more attention. If this could be added to whackadep[1] then you'd be able to monitor your Rust repo pretty solidly! [1]: https://www.cryptologie.net/article/550/supply-chain-attacks-are-the-new-big-thing/ https://www.cryptologie.net/article/550/supply-chain-attacks...
SilverBirch 3y ago
>Before a project starts using a new crate, members usually perform a thorough audit to measure it against their standards for security, correctness, testing, and more. Do they? I mean really? Let's lay aside the fact that it's almost impossible to eyeball security. I just cannot imagine that Google works so differently to every company I've ever worked at that they actually carefully check the stuff they use. Every company I've worked at has had a process for using external code. Some have been stricter than others, none have meaningfully required engineers to make a judgement on the security of code. All of them boil down to speed-running a pointless process. And that leaves apart the obvious question: I want to use a crate, I check it 'works' for what I need. Some middle manager type has mandated I have to now add it to crate audit (FYI, this is the point I dropped importing the library and just wrote it myself) so I add it to crate audit. Some other poor sap comes along and uses it because I audited it, but he's working on VR goggles and I was working on in-vitro fertilization of cats and he's using a whole set of functions that I didn't even realise were there. When his VR Goggles fertilize his beta testers eyes with cat sperm due to a buffer overflow, which of us get fired?
- vasco 3y ago
  > When his VR Goggles fertilize his beta testers eyes with cat sperm due to a buffer overflow, which of us get fired? The PM gets promoted for encouraging fast experimentation!
- zeroxfe 3y ago
  > All of them boil down to speed-running a pointless process. There's a pretty large gap between auditing every line of code and doing nothing. Google does a good job managing external dependencies within their monorepo. There's dedicated tooling, infrastructure, and processes for this.
- jupp0r 3y ago
  Don't forget that you need to do this not only for the crate you depend on, but the whole dependency subtree that comes with it as well.
- yablak 3y ago
  Some useful context here: https://chromium.googlesource.com/chromiumos/third_party/rust_crates/+/refs/heads/main/cargo-vet/#How-do-I-audit https://chromium.googlesource.com/chromiumos/third_party/rus... Seems there are 3-4 folks who helped build this and spent a lot of time doing initial audits; they outsource crypto algorithm audits to specialists.
wly_cdgr 3y ago
Hmmm, is it really a good idea to de-duplicate audits? This is a situation where I want multiple parties to each separately do their own audits
pabs3 3y ago
I wonder why they went with cargo vet instead of the more cross-language crev: https://github.com/crev-dev/ https://github.com/crev-dev/
thenerdhead 3y ago
Ah yes, security by another person who happens to work at Google saying "lgtm".

256 points by taintegral 3y ago

Curious if any senior devs on HN can comment on the importance/effectiveness of audits for crates? I’m a junior C++ dev that dabbles with rust in my free time, and I always feel a bit nervous when pulling huge dependency trees with tons of crates into projects. I would assume most places would turn away from the “node.js” way of doing these things and would just write internal versions of things they need. Again I am junior, so maybe my worries are way over blown.

lesuorac 3y ago
> I would assume most places would turn away from the “node.js” way of doing these things and would just write internal versions of things they need. Incorrect assumption, look up the left pad fiasco [1]. Its importance is really a personal opinion; convince nearly always trumps security so if the NPM way allows you to increase sales by ~10% you'll see people continuing to do it. Google is fairly principled though, all of the 3p code is internally vendored and supposed to be audited by the people pulling in that code/update. [1]: https://www.google.com/search?q=leftpad+broke+the+internet https://www.google.com/search?q=leftpad+broke+the+internet
pclmulqdq 3y ago
I think in a lot of C++ and ex-C++ orgs you see this sentiment a lot, and sometimes for good reason. Sometimes that code has security or performance reasons to worry about this. On the other hand, it often doesn't. On the other hand, Python folks and JavaScript users (which make up a lot of emigres to Rust) probably don't care enough about their supply chain. That's how you end up with misspelled packages causing viruses in production and other disasters. The short answer to this is that it actually depends a lot on what you are doing.
- mr_00ff00 3y ago
  This surprises me that most people that use rust come from python and JavaScript. I would think the reason rust is so popular is from people moving from C and C++ and getting all the nice modern features to do systems with. Python and JavaScript people I would imagine find rust annoying since it’s all the niceties they are use to but with a bunch of rules on top.
  - efnx 3y ago
    I see people coming to Rust from all angles. It’s a nice sweet spot. I came to it from Haskell and on my team of three the other two devs came to Rust from C++. I can opine the motivation looks something like this: * From Haskell - looking for a strong type system (with sum types, typeclasses) but is “widely accepted”. No GC. * From C++ - looking for low-level capabilities (pointers, references) with improved safety. Improved manual memory management. * From Python/JS - looking for performance with a familiar feeling ecosystem and a welcoming community I think the Python and JS folks will have the hardest go of it, but they also have the most to gain.
  - david2ndaccount 3y ago
    Python and Javascript programmers so massively outnumber all other programmers that they are the majority of people converting to any language.
  - deathanatos 3y ago
    I went Python → Rust. Rust is high-level enough for me to remain competitively productive with Python, and the type-system just helps so much. I can't tell you how nice it has been to not have to see "Object of type NoneType has no attribute 'foo'" anymore. Also, def foo(obj): # ... And me wondering "what is 'obj'?" and then pulling that root … and pulling … and pulling …. Also "data", another just fabulous name for a variable, really narrows the possibilities. Even once you know the "type" of the variable, oftentimes I'd find that the type definition would subtle shift and morph in different parts of the program: they'd all want a Duck, but have varying opinions on what a Duck actually is. You cannot commit such BS with an actual type-checker. There is more up-front work with the compiler, but it pays off: the code that passes the compiler is of much better quality. Also, Option<T> is very nice to have when you need an actual Option<T> (i.e., generically), which Python lacks. (No, `None` is not it: You run into problems when T == Option<U>, and you have None and Some(None) — Python's cannot differentiate between the two.) Also sum types in general.
nemothekid 3y ago
The "node.js" way of doing things, and it's dysfunction, is nearly exclusive to node because Javascript lacks a standard library and npm's haphazard way of running things. Java, Ruby, Python, even my grandfather's Perl have had "modules" for years with none of the fear that is typically associated with Node. Personally, C++ aversion to sane dependency management is more about C++'s "I know better than you" culture and legacy cruft (packages are usually managed by the distro, not the language) than actually having any serious security implications.
- bombolo 3y ago
  in a way rust's standard library is close to node's than python's. You can't really do much without getting some crates in.
- pjmlp 3y ago
  This is slowly changing wiht conan and vcpkg increasing adoption. Still most environments I worked on, always had internal repos for packages, no CI/CD server talks to the outside world and vendoring isn't allowed.
wongarsu 3y ago
Writing your own version of everything means it's probably more tuned to your needs. But unless it's a core part of your software it will also be worse because you can't justify putting many resources into it. It also means new hires will have to learn a lot more. It's one of the (many) reasons why it's so hard to onboard into C/C++ projects, because every standard building block is bespoke and somehow different than what everyone else does. Of course if you are really big you just have those resources, which is why Meta or Google can have bespoke everything. On security it's a tradeoff. The open-source version is an easier target for attackers, but might be much more battle-tested and thus more bug-free. Audits are the attempt to have the best of both worlds here, and since they again can be crowd-sourced (with cargo-vet and cargo-cev both working on this) it scales even for companies that aren't Google-sized.
fcantournet 3y ago
Dependencies are dependencies in rust as in C++. I found it's extremely rare that homegrown library that have similar functionality to (used) open-source libraries are better from a security stand-point. At least in Rust a large part of the security issues that would be VERY time consuming to audit at scale through your dependency tree (whether internal or public) are covered by the compiler/borrow checker/type-system. In that sense I would take on an larger amount of dependency in Rust than I would in C++ while sleeping better.
bombolo 3y ago
> I would assume most places would turn away from the “node.js” way of doing these things and would just write internal versions of things they need. I assume most places don't care.
pornel 3y ago
I've reviewed hundreds of Rust crates. It's tedious and boring. The results are boring too — their code is mostly good! Big dependency trees have a reputation for being hot garbage, but that's not my experience. In Rust the small focused crates tend to do one thing, and do it well.

vitorsr 3y ago

See also Google Cloud’s Assured Open Source Software service: https://cloud.google.com/assured-open-source-software https://cloud.google.com/assured-open-source-software

sneed_chucker 3y ago

What's the state of supply chain auditing in general these days?

adolph 3y ago
SE Daily podcast episode "CAP Theorem 23 Years Later with Eric Brewer" touches on it a bit. On the whole an excellent episode. https://softwareengineeringdaily.com/2023/05/12/cap-theorem-23-years-later/ https://softwareengineeringdaily.com/2023/05/12/cap-theorem-...

progbits 3y ago

Can someone explain why cargo-vet doesn't include a cryptographic hash of the crate contents? My understanding is that this repository, and similar ones from Mozilla and others, says: "I, person X from trustworthy organization Y, have reviewed version 1.0 of crate foo and deemed it legit" (for a definition of trustworthy and legit). But now how does that help me if I want to be careful about what I depend on and supply-chain attacks? I ask for version 1.0 of crate foo but might get some malicious payload without knowing it.

hobofan 3y ago
That's already prevented by the checksum which is present for all crate versions in the registry index, which is set in stone on publish and verified by cargo on download. See e.g. https://github.com/rust-lang/crates.io-index/blob/74f1b1e064bc63a3be7026165d0b39ea88baf515/se/rd/serde https://github.com/rust-lang/crates.io-index/blob/74f1b1e064...
- progbits 3y ago
  Hmm, but then you have to trust 1) github, 2) anyone with commit access to that repository. It's not the worst thing I suppose: #1 is a problem anyway for trusting Google/Mozilla's repo of audits, and #2 can be noticed by others so hard to pull of some supply chain attack that way. But I would still feel more confident if the audit log contained a copy of the checksum, and ideally itself was signed with author's keys.
pornel 3y ago
https://lib.rs/cargo-crev https://lib.rs/cargo-crev does this, with the entire chain from the crate data to the reviewer's trusted identity. However, this adds a lot of complexity. cargo-vet went for the other extreme of being super simple. To fill in their review report you don't even need any tooling.

deleted 3y ago

[deleted]

baby 3y ago

Interesting stuff! I think everyone seems to come up with their own solutions. I think security in general is a matter of who you trust, and things only work when we build a network of trust. Imagine if all companies and rust developers started sharing what crates they were confident in + what other organizations they trust as well. If you could then create your own set of such companies, and then choose a dependency depth you were willing to go down to, you might be able to quickly vet a number of crates this way, or at least see the weird crates that demand a bit more attention. If this could be added to whackadep[1] then you'd be able to monitor your Rust repo pretty solidly! [1]: https://www.cryptologie.net/article/550/supply-chain-attacks-are-the-new-big-thing/ https://www.cryptologie.net/article/550/supply-chain-attacks...

SilverBirch 3y ago

>Before a project starts using a new crate, members usually perform a thorough audit to measure it against their standards for security, correctness, testing, and more. Do they? I mean really? Let's lay aside the fact that it's almost impossible to eyeball security. I just cannot imagine that Google works so differently to every company I've ever worked at that they actually carefully check the stuff they use. Every company I've worked at has had a process for using external code. Some have been stricter than others, none have meaningfully required engineers to make a judgement on the security of code. All of them boil down to speed-running a pointless process. And that leaves apart the obvious question: I want to use a crate, I check it 'works' for what I need. Some middle manager type has mandated I have to now add it to crate audit (FYI, this is the point I dropped importing the library and just wrote it myself) so I add it to crate audit. Some other poor sap comes along and uses it because I audited it, but he's working on VR goggles and I was working on in-vitro fertilization of cats and he's using a whole set of functions that I didn't even realise were there. When his VR Goggles fertilize his beta testers eyes with cat sperm due to a buffer overflow, which of us get fired?

vasco 3y ago
> When his VR Goggles fertilize his beta testers eyes with cat sperm due to a buffer overflow, which of us get fired? The PM gets promoted for encouraging fast experimentation!
zeroxfe 3y ago
> All of them boil down to speed-running a pointless process. There's a pretty large gap between auditing every line of code and doing nothing. Google does a good job managing external dependencies within their monorepo. There's dedicated tooling, infrastructure, and processes for this.
jupp0r 3y ago
Don't forget that you need to do this not only for the crate you depend on, but the whole dependency subtree that comes with it as well.
yablak 3y ago
Some useful context here: https://chromium.googlesource.com/chromiumos/third_party/rust_crates/+/refs/heads/main/cargo-vet/#How-do-I-audit https://chromium.googlesource.com/chromiumos/third_party/rus... Seems there are 3-4 folks who helped build this and spent a lot of time doing initial audits; they outsource crypto algorithm audits to specialists.

wly_cdgr 3y ago

Hmmm, is it really a good idea to de-duplicate audits? This is a situation where I want multiple parties to each separately do their own audits

pabs3 3y ago

I wonder why they went with cargo vet instead of the more cross-language crev: https://github.com/crev-dev/ https://github.com/crev-dev/

thenerdhead 3y ago

Ah yes, security by another person who happens to work at Google saying "lgtm".