6 ms·

Vetting the Cargo

73 points by tta 4y ago

dane-pgp 4y ago
> Each new participant automatically contributes its audits back to the commons, making it progressively less work for everyone to secure their dependencies. This is really exciting and I hope it gets adopted by all package ecosystems. Of course audits can't guarantee to find the most underhanded "bugdoors", but it will still be a huge step forwards if third parties can vouch for various properties of the code you are about to install, such as it being reproducibly built from a tagged release on a public repository, with no Unicode homoglyphs or unexplained high-entropy strings in the code, and the unit tests all passing. This will naturally lead to the question of who can be trusted to provide these audits, but such automatable checks could be done by almost anyone and their reputation could grow with time (which might lead to second-layer systems which track which auditors make the most accurate claims). Perhaps there will be companies that offer cyber-insurance against these specific threats, and use the premiums from that to fund the audit checks.
hyperion2010 4y ago
If this does not have a way to track and filter based on who did the audit then it will wind up like the semantic web where anyone can tag a page with safeGoodQuality.
javert 4y ago
> Our dependency tree has steadily grown to almost four hundred third-party crates, and we have thus far lacked a mechanism to efficiently audit this code and ensure that we do so systematically. (-Firefox) Wow. This makes me feel like I have to stop using Firefox. I wonder if others feel the same, or have a different analysis. For example, is the situation with Chrome better?
- hsbauauvhabzb 4y ago
  It’s the current state of development as a whole. I assume most browsers and operating systems do this.
  - azakai 4y ago
    That is not the case. In fact, Firefox itself did not do this until recent years. It is true that modern JS development often works that way (as does Rust and some others), but it is not the norm in all ecosystems, and definitely wasn't in browsers.
    - sgift 4y ago
      I very much doubt that Firefox and most other internal software didn't do this until recent years though usually in a more ad-hoc manner, i.e. "oh look, that code does what I want ... but eh .. how can I add that as dependency, far too much work, let's copy it into our tree", which gives you all the downsides of the current state without any of the upsides.
      - azakai 4y ago
        I can tell you as a former Firefox developer that that was not the case. Yes, Firefox has some dependencies copied into the tree, but deciding to do it, and the process afterwards (including updating), were very careful and slow. Of course there may have been exceptions I am not aware of, and developers are humans that can make mistakes, but that is the overall culture I experienced. It is fundamentally different to the JS/Rust/etc. ecosystem models.
- pengaru 4y ago
  > Wow. This makes me feel like I have to stop using Firefox. I already had that urge in a powerful way after the last ESR update filled my browser with seemingly impossible to remove Bing and Google search hooks, among other obnoxious commercial b.s. I specifically use Firefox to not have shoved in my face at the browser internals level. I'd probably be using Epiphany/GNOME Web full-time if it provided a noscript analog.
- dredmorbius 4y ago
  It's quite likely that this is an accurate desctiption of much development generally, and that Mozilla are simply more transparent about the situation than most vendors. Though Google's deep pockets for security work do likely convey advantage.
- fulafel 4y ago
  If your standard is that third party code must be audited, you'll have quite a small selection of software to use. It's mostly all running on faith and reputation and counting on good intentions. (Like society generally!)
  - cxr 4y ago
    The specific issue described is a recent development. I'm a former Mozillian, and this is surprising to hear. In the pre-Rust days, this NPM-style fractal-of-dependencies approach wasn't a thing in Gecko/Firefox, and anyone naively insisting that this is just how things work in software development (and purportedly have to work) was someone who demonstrably didn't know what they were talking about. Looks like we've lost some of the "demonstrably" part.
    - fulafel 4y ago
      I wasn't talking about Firefox specifically, I could have made that more explicit... But it's good to hear that Mozilla has good culture for this. If there is something you can say or link to about systematic third party code auditing at Mozilla, eg are results public, it would be interesting to hear. Or about how many vulnerabilities are discovered in code audits vs post-shipping security testing like fuzzing and other pentesting-y activities. (Of course good control of versions is still a worthy goal for many situations even if you don't do this)
- hampereddustbin 4y ago
  At times like these I'm reminded of the phrase “If you wish to make an apple pie from scratch, you must first invent the universe” You have to rely on something in life, just like in an office building you can't realistically check the structure inside-out, or if you can, how can you make sure that the individual components are actually of the material they say they are? Have you double-checked your local water table yourself? Have you done geological studies to uncover vulnerabilities the subcontracting firm building the office may not have done correctly? What is the effect of local radio-interference or power quality on your equipment? Are your UPS-devices actually performing to spec?
  - II2II 4y ago
    > At times like these I'm reminded of the phrase “If you wish to make an apple pie from scratch, you must first invent the universe” Sure, but it may also be a comparison between grandma's apple pie and the apple pie from the supermarket. If you never looked at the ingredient list of the latter, you're bound to be surprised at what you're getting. If grandma hand picked her apples from the back yard, you may be unhappy to learn about the pesticides used for commercially grown apples. > You have to rely on something in life, just like in an office building you can't realistically check the structure inside-out, or if you can, how can you make sure that the individual components are actually of the material they say they are? In that case, there are building codes as well as checks to ensure they are followed. While it is possible for someone to ignore those codes, there is also a cost for doing so if you are caught. For the most part, the software industry doesn't have building codes. If something fails, software licenses are generally written to avoid accountability. About the only constraint is the negative response of the market, but even that can be managed to some degree.
hda2 4y ago
This sounds a lot like cargo-crev but without off-line cryptographic signatures, a significant downgrade in my view. Edit: Yep: https://mozilla.github.io/cargo-vet/design-choice-faq.html#how-does-this-relate-to-cargo-crev https://mozilla.github.io/cargo-vet/design-choice-faq.html#h... None of the reasons given by Mozilla seem to justify the downgrade in security, especially since most can be worked around with crev which already employs secure and well-tested authentication schemes. What also makes this situation peculiar to me is that it's being immediately rushed into Cargo proper instead of the usual way these tools are handled by the Cargo team (i.e. allowing multiple ideas to compete as third-party tools and maybe choosing one once a winner is clear). I understand the recent string of security issues might have played a role here, but I wouldn't expect their reaction to be steamrolling an inferior version of crev as a builtin tool. I would really like to know what happened here. Disclosure: I use neither tool, but I'm very interested in the security and health of the Rust ecosystem.
- LegionMammal978 4y ago
  > What also makes this situation peculiar to me is that it's being immediately rushed into Cargo proper instead of the usual way these tools are handled by the Cargo team (i.e. allowing multiple ideas to compete as third-party tools and maybe choosing one once a winner is clear). This tool isn't being pushed into Cargo proper, and I doubt it will be any time soon. Instead, it is a third-party crate called "cargo-vet". If an executable named "cargo-whatever" is in the search path, Cargo can invoke it as a subcommand "cargo whatever". See Cargo's docs for more info: https://doc.rust-lang.org/cargo/reference/external-tools.html#custom-subcommands https://doc.rust-lang.org/cargo/reference/external-tools.htm...
  - cozzyd 4y ago
    hopefully nobody adds a cargo-buidl package then!
    - nindalf 4y ago
      Why would a user install a cargo-buidl package when cargo build already works? Typosquatting is definitely an issue, but not in this specific case.
mattpallissard 4y ago
With out mandatory user fields optional or signed audits I wasn't convinced how useful this was. I originally read the post viewing this as a way of sharing audits amongst the general public, in similar fashion to how the packages themselves are distributed. However, once I RTFM[1] (well some of it at least) I realized that is not the intended use case and that I was mistaken. tl'dr * internal management of audits. * reading audits in from trusted third parties. I'm glad that someone is starting to tackle this terrifying problem. [1] https://mozilla.github.io/cargo-vet/importing-audits.html https://mozilla.github.io/cargo-vet/importing-audits.html
colonwqbang 4y ago
Has there been any study of the effectiveness of code audits? Sensitivity and specificity in trying to find security problems, unintentional or intentional.
- hampereddustbin 4y ago
  It's a question of whether you can have a process which can reliably detect if an arbitrary program will do X, and the answer is proved to be no: https://en.wikipedia.org/wiki/Halting_problem https://en.wikipedia.org/wiki/Halting_problem In real life however we have to accept imperfections, in which good-enough is often better than not at all
lawl 4y ago
This feels like the equivalent of AdressSanitizer and similar tools for C. They fix a problem that shouldn't exist. At least not this extreme. C has the excuse of being old, Rust does not have that excuse. Using npm as an inspiration for cargo is just really sad.
- diggan 4y ago
  > They fix a problem that shouldn't exist Shouldn't exists because why? Because people should just be nice? Supply-chain attacks are a real vector, one that seems to be able to have a larger impact by each day, as the OSS ecosystem grows and the dependency of using dependencies grow. People want to be able to use 3rd party packages, like we've been doing for quite some time now. But there are a lot of them, and no (easy) way to manage exactly which one are "good" vs not, without manually going through each one of them, for each project.
- bin_bash 4y ago
  Any ecosystem where you’re running code from unvetted third parties is susceptible to this problem. We either need solutions to improve the supply chain safety or never use third party dependencies.
diggan 4y ago
Alternatives to cargo-vet that has been mentioned before here on HN: - https://github.com/crev-dev/crev https://github.com/crev-dev/crev - https://github.com/vouch-dev/vouch https://github.com/vouch-dev/vouch Anyone know of any more alternatives or similar tools already available?
weinzierl 4y ago
Does anyone have a link to Mozilla's audits.toml? I found one in Mozilla's GitHub[1], but it only has five entries. Moreover all of the crates in the file were audited by their respective authors - which kind of goes against the whole idea of this thing. Overall GitHub seems to only have six audits.toml files[1], two of which are the Mozilla one mentioned above, the other four are empty. [1] https://github.com/mozilla/gecko-dev/blob/64f3b7d019700f4fe3e5c033986bf7d20b49ba1c/supply-chain/audits.toml https://github.com/mozilla/gecko-dev/blob/64f3b7d019700f4fe3... [2] https://github.com/search?q=filename%3Aaudits.toml&type=code https://github.com/search?q=filename%3Aaudits.toml&type=code
saagarjha 4y ago
I’m hesitant about this effort too, but for different reasons than the ones mentioned already. “Vetting” code for the kernel means something very different than vetting code for a normal application. I mean sure, you want to make sure that the code is functional and free of bad practices that may hide bugs or vulnerabilities, but that’s kind of where the mutual needs end. Kernel code in some cases may not allocate, or may have to avoid the use of certain vector registers. Locking and synchronization in the kernel usually looks pretty different from what you’d want to do in userspace. I’m not entirely against the idea of using third party crates in the kernel but it seems to me that you’d want something better than just “Mozilla ships this in Firefox and they got one of their security engineers to look at it so it’s probably good” that this vetting process seems to provide.
mrpotato 4y ago
Having heard about this for the first time, I think it's a great initiative. However, I'm concerned about the liabilities of having vetted a crate only to accidentally (ie non-maliciously) miss some kind of vulnerability. Could this lead to the auditor getting sued? I wouldn't mind getting a legal take on cargo-vet and similar tools.
4sak3n 4y ago
> A developer working on a function may suddenly discover the need to, say, left-pad a string with blanks. Rather than go though the pain of implementing this challenging functionality ... The irony is palpable.
brundolf 4y ago
The snide/sarcastic tone is surprising and unbecoming for what's normally a source of high-quality articles

73 points by tta 4y ago

> Each new participant automatically contributes its audits back to the commons, making it progressively less work for everyone to secure their dependencies. This is really exciting and I hope it gets adopted by all package ecosystems. Of course audits can't guarantee to find the most underhanded "bugdoors", but it will still be a huge step forwards if third parties can vouch for various properties of the code you are about to install, such as it being reproducibly built from a tagged release on a public repository, with no Unicode homoglyphs or unexplained high-entropy strings in the code, and the unit tests all passing. This will naturally lead to the question of who can be trusted to provide these audits, but such automatable checks could be done by almost anyone and their reputation could grow with time (which might lead to second-layer systems which track which auditors make the most accurate claims). Perhaps there will be companies that offer cyber-insurance against these specific threats, and use the premiums from that to fund the audit checks.

hyperion2010 4y ago

If this does not have a way to track and filter based on who did the audit then it will wind up like the semantic web where anyone can tag a page with safeGoodQuality.

javert 4y ago

> Our dependency tree has steadily grown to almost four hundred third-party crates, and we have thus far lacked a mechanism to efficiently audit this code and ensure that we do so systematically. (-Firefox) Wow. This makes me feel like I have to stop using Firefox. I wonder if others feel the same, or have a different analysis. For example, is the situation with Chrome better?

hsbauauvhabzb 4y ago
It’s the current state of development as a whole. I assume most browsers and operating systems do this.
- azakai 4y ago
  That is not the case. In fact, Firefox itself did not do this until recent years. It is true that modern JS development often works that way (as does Rust and some others), but it is not the norm in all ecosystems, and definitely wasn't in browsers.
  - sgift 4y ago
    I very much doubt that Firefox and most other internal software didn't do this until recent years though usually in a more ad-hoc manner, i.e. "oh look, that code does what I want ... but eh .. how can I add that as dependency, far too much work, let's copy it into our tree", which gives you all the downsides of the current state without any of the upsides.
    - azakai 4y ago
      I can tell you as a former Firefox developer that that was not the case. Yes, Firefox has some dependencies copied into the tree, but deciding to do it, and the process afterwards (including updating), were very careful and slow. Of course there may have been exceptions I am not aware of, and developers are humans that can make mistakes, but that is the overall culture I experienced. It is fundamentally different to the JS/Rust/etc. ecosystem models.
pengaru 4y ago
> Wow. This makes me feel like I have to stop using Firefox. I already had that urge in a powerful way after the last ESR update filled my browser with seemingly impossible to remove Bing and Google search hooks, among other obnoxious commercial b.s. I specifically use Firefox to not have shoved in my face at the browser internals level. I'd probably be using Epiphany/GNOME Web full-time if it provided a noscript analog.
dredmorbius 4y ago
It's quite likely that this is an accurate desctiption of much development generally, and that Mozilla are simply more transparent about the situation than most vendors. Though Google's deep pockets for security work do likely convey advantage.
fulafel 4y ago
If your standard is that third party code must be audited, you'll have quite a small selection of software to use. It's mostly all running on faith and reputation and counting on good intentions. (Like society generally!)
- cxr 4y ago
  The specific issue described is a recent development. I'm a former Mozillian, and this is surprising to hear. In the pre-Rust days, this NPM-style fractal-of-dependencies approach wasn't a thing in Gecko/Firefox, and anyone naively insisting that this is just how things work in software development (and purportedly have to work) was someone who demonstrably didn't know what they were talking about. Looks like we've lost some of the "demonstrably" part.
  - fulafel 4y ago
    I wasn't talking about Firefox specifically, I could have made that more explicit... But it's good to hear that Mozilla has good culture for this. If there is something you can say or link to about systematic third party code auditing at Mozilla, eg are results public, it would be interesting to hear. Or about how many vulnerabilities are discovered in code audits vs post-shipping security testing like fuzzing and other pentesting-y activities. (Of course good control of versions is still a worthy goal for many situations even if you don't do this)
hampereddustbin 4y ago
At times like these I'm reminded of the phrase “If you wish to make an apple pie from scratch, you must first invent the universe” You have to rely on something in life, just like in an office building you can't realistically check the structure inside-out, or if you can, how can you make sure that the individual components are actually of the material they say they are? Have you double-checked your local water table yourself? Have you done geological studies to uncover vulnerabilities the subcontracting firm building the office may not have done correctly? What is the effect of local radio-interference or power quality on your equipment? Are your UPS-devices actually performing to spec?
- II2II 4y ago
  > At times like these I'm reminded of the phrase “If you wish to make an apple pie from scratch, you must first invent the universe” Sure, but it may also be a comparison between grandma's apple pie and the apple pie from the supermarket. If you never looked at the ingredient list of the latter, you're bound to be surprised at what you're getting. If grandma hand picked her apples from the back yard, you may be unhappy to learn about the pesticides used for commercially grown apples. > You have to rely on something in life, just like in an office building you can't realistically check the structure inside-out, or if you can, how can you make sure that the individual components are actually of the material they say they are? In that case, there are building codes as well as checks to ensure they are followed. While it is possible for someone to ignore those codes, there is also a cost for doing so if you are caught. For the most part, the software industry doesn't have building codes. If something fails, software licenses are generally written to avoid accountability. About the only constraint is the negative response of the market, but even that can be managed to some degree.

hda2 4y ago

This sounds a lot like cargo-crev but without off-line cryptographic signatures, a significant downgrade in my view. Edit: Yep: https://mozilla.github.io/cargo-vet/design-choice-faq.html#how-does-this-relate-to-cargo-crev https://mozilla.github.io/cargo-vet/design-choice-faq.html#h... None of the reasons given by Mozilla seem to justify the downgrade in security, especially since most can be worked around with crev which already employs secure and well-tested authentication schemes. What also makes this situation peculiar to me is that it's being immediately rushed into Cargo proper instead of the usual way these tools are handled by the Cargo team (i.e. allowing multiple ideas to compete as third-party tools and maybe choosing one once a winner is clear). I understand the recent string of security issues might have played a role here, but I wouldn't expect their reaction to be steamrolling an inferior version of crev as a builtin tool. I would really like to know what happened here. Disclosure: I use neither tool, but I'm very interested in the security and health of the Rust ecosystem.

LegionMammal978 4y ago
> What also makes this situation peculiar to me is that it's being immediately rushed into Cargo proper instead of the usual way these tools are handled by the Cargo team (i.e. allowing multiple ideas to compete as third-party tools and maybe choosing one once a winner is clear). This tool isn't being pushed into Cargo proper, and I doubt it will be any time soon. Instead, it is a third-party crate called "cargo-vet". If an executable named "cargo-whatever" is in the search path, Cargo can invoke it as a subcommand "cargo whatever". See Cargo's docs for more info: https://doc.rust-lang.org/cargo/reference/external-tools.html#custom-subcommands https://doc.rust-lang.org/cargo/reference/external-tools.htm...
- cozzyd 4y ago
  hopefully nobody adds a cargo-buidl package then!
  - nindalf 4y ago
    Why would a user install a cargo-buidl package when cargo build already works? Typosquatting is definitely an issue, but not in this specific case.

mattpallissard 4y ago

With out mandatory user fields optional or signed audits I wasn't convinced how useful this was. I originally read the post viewing this as a way of sharing audits amongst the general public, in similar fashion to how the packages themselves are distributed. However, once I RTFM[1] (well some of it at least) I realized that is not the intended use case and that I was mistaken. tl'dr * internal management of audits. * reading audits in from trusted third parties. I'm glad that someone is starting to tackle this terrifying problem. [1] https://mozilla.github.io/cargo-vet/importing-audits.html https://mozilla.github.io/cargo-vet/importing-audits.html

colonwqbang 4y ago

Has there been any study of the effectiveness of code audits? Sensitivity and specificity in trying to find security problems, unintentional or intentional.

hampereddustbin 4y ago
It's a question of whether you can have a process which can reliably detect if an arbitrary program will do X, and the answer is proved to be no: https://en.wikipedia.org/wiki/Halting_problem https://en.wikipedia.org/wiki/Halting_problem In real life however we have to accept imperfections, in which good-enough is often better than not at all

lawl 4y ago

This feels like the equivalent of AdressSanitizer and similar tools for C. They fix a problem that shouldn't exist. At least not this extreme. C has the excuse of being old, Rust does not have that excuse. Using npm as an inspiration for cargo is just really sad.

diggan 4y ago
> They fix a problem that shouldn't exist Shouldn't exists because why? Because people should just be nice? Supply-chain attacks are a real vector, one that seems to be able to have a larger impact by each day, as the OSS ecosystem grows and the dependency of using dependencies grow. People want to be able to use 3rd party packages, like we've been doing for quite some time now. But there are a lot of them, and no (easy) way to manage exactly which one are "good" vs not, without manually going through each one of them, for each project.
bin_bash 4y ago
Any ecosystem where you’re running code from unvetted third parties is susceptible to this problem. We either need solutions to improve the supply chain safety or never use third party dependencies.

diggan 4y ago

Alternatives to cargo-vet that has been mentioned before here on HN: - https://github.com/crev-dev/crev https://github.com/crev-dev/crev - https://github.com/vouch-dev/vouch https://github.com/vouch-dev/vouch Anyone know of any more alternatives or similar tools already available?

weinzierl 4y ago

Does anyone have a link to Mozilla's audits.toml? I found one in Mozilla's GitHub[1], but it only has five entries. Moreover all of the crates in the file were audited by their respective authors - which kind of goes against the whole idea of this thing. Overall GitHub seems to only have six audits.toml files[1], two of which are the Mozilla one mentioned above, the other four are empty. [1] https://github.com/mozilla/gecko-dev/blob/64f3b7d019700f4fe3e5c033986bf7d20b49ba1c/supply-chain/audits.toml https://github.com/mozilla/gecko-dev/blob/64f3b7d019700f4fe3... [2] https://github.com/search?q=filename%3Aaudits.toml&type=code https://github.com/search?q=filename%3Aaudits.toml&type=code

saagarjha 4y ago

I’m hesitant about this effort too, but for different reasons than the ones mentioned already. “Vetting” code for the kernel means something very different than vetting code for a normal application. I mean sure, you want to make sure that the code is functional and free of bad practices that may hide bugs or vulnerabilities, but that’s kind of where the mutual needs end. Kernel code in some cases may not allocate, or may have to avoid the use of certain vector registers. Locking and synchronization in the kernel usually looks pretty different from what you’d want to do in userspace. I’m not entirely against the idea of using third party crates in the kernel but it seems to me that you’d want something better than just “Mozilla ships this in Firefox and they got one of their security engineers to look at it so it’s probably good” that this vetting process seems to provide.

mrpotato 4y ago

Having heard about this for the first time, I think it's a great initiative. However, I'm concerned about the liabilities of having vetted a crate only to accidentally (ie non-maliciously) miss some kind of vulnerability. Could this lead to the auditor getting sued? I wouldn't mind getting a legal take on cargo-vet and similar tools.

4sak3n 4y ago

> A developer working on a function may suddenly discover the need to, say, left-pad a string with blanks. Rather than go though the pain of implementing this challenging functionality ... The irony is palpable.

brundolf 4y ago

The snide/sarcastic tone is surprising and unbecoming for what's normally a source of high-quality articles