13 ms·

> If crates.io goes down or access is otherwise disrupted then the Rust community will stop work. As someone who has worked in a big corporate their entire car

> If crates.io goes down or access is otherwise disrupted then the Rust community will stop work. As someone who has worked in a big corporate their entire career this statement is bizarre. This has been solved for Java for over a decade. Builds don't access the internet directly. You host a proxy on-premise (Nexus/Artifactory/etc) and access that. If the internet location is down, you continue as normal. > You need only one author in your maybe-hundreds-of-dependencies tree to be hacked, coerced or in a malicious mood for you to have a really bad day. There are products (Nexus Firewall) that can check dependencies for vulnerabilities and either block them from entering the network or fail CI pipelines.

richardwhiuk 3y ago
Every serious enterprise runs a Rust proxy - and indeed Artifactory / Azure Artifacts and others support this.
- jrpelkonen 3y ago
  Sure, Artifactory, etc. have their places. But I am objecting to your "no true scotsman"[1] type of argument. Furthermore, the most bizarre setups I have seen have been in "serious" enterprises. Hardly an endorsement. 1: https://en.wikipedia.org/wiki/No_true_Scotsman https://en.wikipedia.org/wiki/No_true_Scotsman
  - devman0 3y ago
    I have to agree with the poster you are replying to. If an entity is not doing the bare minimum to QA software supply chain, then they are not "serious". That being said, I have seen many a large enterprise that do not treat software development as a serious endeavor and it shows.
brabel 3y ago
The main problem the author is talking about is actually about version updates, which in Maven as well as crates.io is up to each lib's author, and is not curated in any way. There's no technical solution to that, really. Do you think Nexus Firewall can pick up every exploit, or even most? How confident of that are you, and what data do you have to back that up? I don't have any myself, but would not be surprised at all if "hackers" can easily work around their scanning. However, I don't have a better approach than using scanning tools like Nexus, or as the author proposes, use a curated library repository like Debian is doing (which hopefully gets enough eyeballs to remain secure) or the https://github.com/crev-dev/cargo-crev https://github.com/crev-dev/cargo-crev project (manually reviewed code) also mentioned. It's interesting that they mention C/C++ just rely on distros providing dynamic libs instead which means you don't even control your dependencies versions, some distro does (how reliable is the distro?)... I wonder if that could work for other languages or if it's just as painful as it looks in the C world.
- jjgreen 3y ago
  Coding mainly in C for 20 years, I've never had a program break on minor version of system shared library conflict. Occasionally need to recompile on OS version update when the major version increments.
  - goodpoint 3y ago
    That does not solve supply chain attacks.
    - mid-kid 3y ago
      It solves most supply chain attacks. Distro maintainers are users, not developers. They put the needs of a user first, and their work is reviewed during submission. If one of them goes rogue, the damage is usually caught before reaching the user and otherwise minimal, as the update is simply reverted by other maintainers and a different maintainer can take over their packages.
      - goodpoint 3y ago
        Sorry, I misread jjgreen's comment. I 100% agree with what you are saying.
  - lmm 3y ago
    The C community basically doesn't bother with libraries. They're so cumbersome to do in C that you'll tend to only have, like, 20 giant libraries in your (non-vendored) dependencies.
- mid-kid 3y ago
  It works in the C world because everyone is super fixated on library compatibility, and the programs are expected to be rebuilt by the distribution when necessary. When an API is broken, this entails a new major version and installing both versions side by side until all maintained software is ported. This doesn't happen often enough to be an issue - that's the perk of having a really mature ecosystem.
brightball 3y ago
Gitlab has this built in as well. EDIT: Has a proxy for packages. https://docs.gitlab.com/ee/user/packages/package_registry/ https://docs.gitlab.com/ee/user/packages/package_registry/
- basedrum 3y ago
  Has which built in?
  - brightball 3y ago
    Package registry proxy (linked in original comment)
- basedrum 3y ago
  That isn't a proxy, it's a registry for packages you build
wongarsu 3y ago
And this is easy to do with Rust as well. The article mentions both major options: store your dependencies in your repo using the built-in `cargo vendor` command, or run a mirror like panamax that just mirrors all of crates.io (or just the packages you need, but then you run into issues with updating if crates.io is ever down). Mirroring all of crates.io takes about a terabyte of storage, not much bandwidth and is really easy to set up, but judging by the download numbers of obscure packages there are only about 20 places in the world doing that. Maybe people haven't bothered because crates.io is so stable, maybe everyone goes the vendoring route instead.
- api 3y ago
  These days some laptops have enough storage to comfortably mirror all of multiple language package repositories especially if you use a compressed volume for them.
- galangalalgol 3y ago
  Why would you ask panamax to mirror everything though? You are maintaining a manifest of all your organizations dependencies right? Why would I mirror ancient versionsbof clap and that crate that checks your aol mail? Also, nexus doesn't have a functioning cargo plugin, it is community made and stagnant. Artifactory does support it. And I'm guessing artifactory is what most people are using for a private registry too given the state of the others.
  - wongarsu 3y ago
    Mirroring everything gives me the ability to not just build existing software independently of crates.io outages or decisions, but also to not be impacted in software development by such incidents. To me development involves using packages or package versions I might not have used before. Of course if your development is already set up in a way that prevents you from easily adding, experimenting with or updating dependencies, maybe because of how the vetting process works, that might not be an issue for you. And many places are completely fine with just being able to build
    - 3y ago
hun3 3y ago
> There are products (Nexus Firewall) that can check dependencies for vulnerabilities Products can only do best effort scanning for known patterns of vulnerabilites. Those are only added after someone (or something) discovers it and verifies that it's not a false alarm. In between the time gap, scanners are ineffective and anything in your infrastructure can run the malicious code and get hacked. (Google supply chain incidents on npm or pypi.) In general, there is always a way to bypass it, since the language is turing complete and unsandboxed. Anything going beyond that is a lie, or simply impractical. Systems with latest software and antivirus get hacked all the time. Nothing can really stop them. Why? - https://web.archive.org/web/20160424133617/http://www.pcworld.com/article/130455/article.html https://web.archive.org/web/20160424133617/http://www.pcworl... - https://security.stackexchange.com/questions/201992/has-it-been-mathematically-proven-that-antivirus-cant-detect-all-viruses https://security.stackexchange.com/questions/201992/has-it-b... - https://en.wikipedia.org/wiki/Rice%27s_theorem https://en.wikipedia.org/wiki/Rice%27s_theorem > and either block them from entering the network or fail CI pipelines. Also, relying solely on an endpoint security product for protection is dangerous since antiviruses themselves get hacked all the time. Sonatype for an example: - https://nvd.nist.gov/vuln/search/results?form_type=Basic&results_type=overview&query=Sonatype&queryType=phrase&search_type=all&isCpeNameSearch=false https://nvd.nist.gov/vuln/search/results?form_type=Basic&res...
- hoosieree 3y ago
  My research is about detecting semantically similar executable code inside obfuscated and stripped programs. I don't know what commercial antivirus or vulnerability scanners use internally, but it's possible to generate similarity scores between an obfuscated/stripped unknown binary and a bunch of known binaries. I suspect commercial scanners use a lot of heuristics. I know IDA Pro has a plugin for "fingerprinting" but it's based on hashes of byte sequences and can be spoofed. My approach is basically: train a model on a large obfuscated dataset seeded with "known" examples. While you can't say with certainty what an unknown sample contains, you can determine how similar it is to a known sample, so you can spend more of your time analyzing the really weird stuff. The hardest part in my opinion is generating the training data. You need a good source code obfuscator for your language. I've seen a lot of papers that use obfuscator-llvm[1] to obfuscate the IR during compilation. I use Tigress[2] to obfuscate the source code because it provides more diversity, but it only supports C. [1]: https://github.com/obfuscator-llvm/obfuscator/wiki/Installation https://github.com/obfuscator-llvm/obfuscator/wiki/Installat... [2]: https://tigress.wtf/ https://tigress.wtf/
tgv 3y ago
> There are products (Nexus Firewall) that can check dependencies for vulnerabilities and either block them from entering the network or fail CI pipelines. Once they're known. I suppose nexus firewall let log4j pass.
- vilunov 3y ago
  Debian devs also let log4j pass.
  - frakkingcylons 3y ago
    It's pretty absurd to expect distribution maintainers to be discovering 0days.
- ashishbijlani 3y ago
  I’ve been building Packj [1] to detect publicly UNKNOWN dummy, malicious, abandoned, typo-squatting, and other "risky" PyPI/NPM/Ruby/PHP/Maven/Rust packages. It carries out static/dynamic/metadata analysis and scans for 40+ attributes such as num funcs/files, spawning of shell, use of SSH keys, network communication, use of decode+eval, etc. to flag risky packages. Packj Github action [2] can alert if a risky dependency is pulled into your build. 1. https://github.com/ossillate-inc/packj https://github.com/ossillate-inc/packj 2. https://github.com/ossillate-inc/packj-github-action https://github.com/ossillate-inc/packj-github-action
- oblio 3y ago
  > Once they're known. Isn't that how life works? Unknown unknowns? :-) > I suppose nexus firewall let log4j pass. It should be reasonable to assume that a product dedicated to something will, on average, do this job better/faster than a random developer, though. Since that's their job. So Nexus Firewall presumably blocks these vulnerabilities faster than the average app/system developer notices them. There's value in reducing the delta between the vulnerability exploit day and an entire ecosystem being patched/fixed.
- livealight 3y ago
  It let log4j pass for as long as it was known to be good. Within hours of the CVE opening the tool was blocking it. The purpose of dependency firewalls is to avoid two things: known badly vulnerable packages AND known malicious packages that serve no other purpose than to steal data or drop a trojan. No security is 100% bulletproof, but it's really surprising how much of the damage is done by 7 year old CVEs. Firewalls can be useful in exactly that.
KronisLV 3y ago
> As someone who has worked in a big corporate their entire career this statement is bizarre. This has been solved for Java for over a decade. > Builds don't access the internet directly. You host a proxy on-premise (Nexus/Artifactory/etc) and access that. If the internet location is down, you continue as normal. Right now I'm using Sonatype Nexus (the free version) for my personal needs: https://www.sonatype.com/products/sonatype-nexus-repository https://www.sonatype.com/products/sonatype-nexus-repository And I do mean most things: apt packages, Docker images, Helm charts, as well as packages for Go, Java, .NET, Node, Python and Ruby, maybe even more in the future. I'm very glad that it is possible to do something like that, but honestly it's hard to do and I can easily see why most wouldn't bother: even though the configuration UI is okay and you can create users with appropriate permissions, you'll still need to have plenty of storage, manage updates and backups, setup automated cleanup tasks (especially for Docker images), in addition to the whole process on the developer's side sometimes being a nuisance. With npm you get --registry which is one of the simpler approaches, Ruby Gems needed some CLI commands along the lines of "bundle config mirror" and other stuff, Maven needs an XML file with the configuration, in addition to publishing your own packages to those also needing more configuration. Sadly, it seems like this fragmentation of approaches is unavoidable and complexity increases with the amount of languages you use. On the bright side, I keep control over my data, as well as have really good download/upload speeds, even on a cheap VPS. Plus you learn some vaguely useful things the more you explore if you want to go further, like how I also start with Debian/Ubuntu base container images and provision whatever tools or runtimes I need for more specialized images (e.g. my own JDK, .NET, Node etc. images).
- oblio 3y ago
  I don't know if they ever implemented it, can it offload the storage to S3? The cost is peanuts and you stop bothering about the annoying recurrent storage issues.
  - KronisLV 3y ago
    > I don't know if they ever implemented it, can it offload the storage to S3? There are actually multiple different kinds of blob store backends: https://help.sonatype.com/repomanager3/nexus-repository-administration/repository-management/configuring-blob-stores https://help.sonatype.com/repomanager3/nexus-repository-admi... though so far I've only used the File one, because my servers have incremental backups that can be restored in a few clicks (that I've sometimes done in the past).
keep_reading 3y ago
I have CI pipelines fail when Github goes down because of crates.io being inaccessible. Does anyone have a proxy for Rust? I'd love to run it. Please. Someone... help edit: just learned Panamax exists for this... will try it

richardwhiuk 3y ago

Every serious enterprise runs a Rust proxy - and indeed Artifactory / Azure Artifacts and others support this.

jrpelkonen 3y ago
Sure, Artifactory, etc. have their places. But I am objecting to your "no true scotsman"[1] type of argument. Furthermore, the most bizarre setups I have seen have been in "serious" enterprises. Hardly an endorsement. 1: https://en.wikipedia.org/wiki/No_true_Scotsman https://en.wikipedia.org/wiki/No_true_Scotsman
- devman0 3y ago
  I have to agree with the poster you are replying to. If an entity is not doing the bare minimum to QA software supply chain, then they are not "serious". That being said, I have seen many a large enterprise that do not treat software development as a serious endeavor and it shows.

brabel 3y ago

The main problem the author is talking about is actually about version updates, which in Maven as well as crates.io is up to each lib's author, and is not curated in any way. There's no technical solution to that, really. Do you think Nexus Firewall can pick up every exploit, or even most? How confident of that are you, and what data do you have to back that up? I don't have any myself, but would not be surprised at all if "hackers" can easily work around their scanning. However, I don't have a better approach than using scanning tools like Nexus, or as the author proposes, use a curated library repository like Debian is doing (which hopefully gets enough eyeballs to remain secure) or the https://github.com/crev-dev/cargo-crev https://github.com/crev-dev/cargo-crev project (manually reviewed code) also mentioned. It's interesting that they mention C/C++ just rely on distros providing dynamic libs instead which means you don't even control your dependencies versions, some distro does (how reliable is the distro?)... I wonder if that could work for other languages or if it's just as painful as it looks in the C world.

jjgreen 3y ago
Coding mainly in C for 20 years, I've never had a program break on minor version of system shared library conflict. Occasionally need to recompile on OS version update when the major version increments.
- goodpoint 3y ago
  That does not solve supply chain attacks.
  - mid-kid 3y ago
    It solves most supply chain attacks. Distro maintainers are users, not developers. They put the needs of a user first, and their work is reviewed during submission. If one of them goes rogue, the damage is usually caught before reaching the user and otherwise minimal, as the update is simply reverted by other maintainers and a different maintainer can take over their packages.
    - goodpoint 3y ago
      Sorry, I misread jjgreen's comment. I 100% agree with what you are saying.
- lmm 3y ago
  The C community basically doesn't bother with libraries. They're so cumbersome to do in C that you'll tend to only have, like, 20 giant libraries in your (non-vendored) dependencies.
mid-kid 3y ago
It works in the C world because everyone is super fixated on library compatibility, and the programs are expected to be rebuilt by the distribution when necessary. When an API is broken, this entails a new major version and installing both versions side by side until all maintained software is ported. This doesn't happen often enough to be an issue - that's the perk of having a really mature ecosystem.

brightball 3y ago

Gitlab has this built in as well. EDIT: Has a proxy for packages. https://docs.gitlab.com/ee/user/packages/package_registry/ https://docs.gitlab.com/ee/user/packages/package_registry/

basedrum 3y ago
Has which built in?
- brightball 3y ago
  Package registry proxy (linked in original comment)
basedrum 3y ago
That isn't a proxy, it's a registry for packages you build

wongarsu 3y ago

And this is easy to do with Rust as well. The article mentions both major options: store your dependencies in your repo using the built-in `cargo vendor` command, or run a mirror like panamax that just mirrors all of crates.io (or just the packages you need, but then you run into issues with updating if crates.io is ever down). Mirroring all of crates.io takes about a terabyte of storage, not much bandwidth and is really easy to set up, but judging by the download numbers of obscure packages there are only about 20 places in the world doing that. Maybe people haven't bothered because crates.io is so stable, maybe everyone goes the vendoring route instead.

api 3y ago
These days some laptops have enough storage to comfortably mirror all of multiple language package repositories especially if you use a compressed volume for them.
galangalalgol 3y ago
Why would you ask panamax to mirror everything though? You are maintaining a manifest of all your organizations dependencies right? Why would I mirror ancient versionsbof clap and that crate that checks your aol mail? Also, nexus doesn't have a functioning cargo plugin, it is community made and stagnant. Artifactory does support it. And I'm guessing artifactory is what most people are using for a private registry too given the state of the others.
- wongarsu 3y ago
  Mirroring everything gives me the ability to not just build existing software independently of crates.io outages or decisions, but also to not be impacted in software development by such incidents. To me development involves using packages or package versions I might not have used before. Of course if your development is already set up in a way that prevents you from easily adding, experimenting with or updating dependencies, maybe because of how the vetting process works, that might not be an issue for you. And many places are completely fine with just being able to build
  - 3y ago

hun3 3y ago

> There are products (Nexus Firewall) that can check dependencies for vulnerabilities Products can only do best effort scanning for known patterns of vulnerabilites. Those are only added after someone (or something) discovers it and verifies that it's not a false alarm. In between the time gap, scanners are ineffective and anything in your infrastructure can run the malicious code and get hacked. (Google supply chain incidents on npm or pypi.) In general, there is always a way to bypass it, since the language is turing complete and unsandboxed. Anything going beyond that is a lie, or simply impractical. Systems with latest software and antivirus get hacked all the time. Nothing can really stop them. Why? - https://web.archive.org/web/20160424133617/http://www.pcworld.com/article/130455/article.html https://web.archive.org/web/20160424133617/http://www.pcworl... - https://security.stackexchange.com/questions/201992/has-it-been-mathematically-proven-that-antivirus-cant-detect-all-viruses https://security.stackexchange.com/questions/201992/has-it-b... - https://en.wikipedia.org/wiki/Rice%27s_theorem https://en.wikipedia.org/wiki/Rice%27s_theorem > and either block them from entering the network or fail CI pipelines. Also, relying solely on an endpoint security product for protection is dangerous since antiviruses themselves get hacked all the time. Sonatype for an example: - https://nvd.nist.gov/vuln/search/results?form_type=Basic&results_type=overview&query=Sonatype&queryType=phrase&search_type=all&isCpeNameSearch=false https://nvd.nist.gov/vuln/search/results?form_type=Basic&res...

hoosieree 3y ago
My research is about detecting semantically similar executable code inside obfuscated and stripped programs. I don't know what commercial antivirus or vulnerability scanners use internally, but it's possible to generate similarity scores between an obfuscated/stripped unknown binary and a bunch of known binaries. I suspect commercial scanners use a lot of heuristics. I know IDA Pro has a plugin for "fingerprinting" but it's based on hashes of byte sequences and can be spoofed. My approach is basically: train a model on a large obfuscated dataset seeded with "known" examples. While you can't say with certainty what an unknown sample contains, you can determine how similar it is to a known sample, so you can spend more of your time analyzing the really weird stuff. The hardest part in my opinion is generating the training data. You need a good source code obfuscator for your language. I've seen a lot of papers that use obfuscator-llvm[1] to obfuscate the IR during compilation. I use Tigress[2] to obfuscate the source code because it provides more diversity, but it only supports C. [1]: https://github.com/obfuscator-llvm/obfuscator/wiki/Installation https://github.com/obfuscator-llvm/obfuscator/wiki/Installat... [2]: https://tigress.wtf/ https://tigress.wtf/

tgv 3y ago

> There are products (Nexus Firewall) that can check dependencies for vulnerabilities and either block them from entering the network or fail CI pipelines. Once they're known. I suppose nexus firewall let log4j pass.

vilunov 3y ago
Debian devs also let log4j pass.
- frakkingcylons 3y ago
  It's pretty absurd to expect distribution maintainers to be discovering 0days.
ashishbijlani 3y ago
I’ve been building Packj [1] to detect publicly UNKNOWN dummy, malicious, abandoned, typo-squatting, and other "risky" PyPI/NPM/Ruby/PHP/Maven/Rust packages. It carries out static/dynamic/metadata analysis and scans for 40+ attributes such as num funcs/files, spawning of shell, use of SSH keys, network communication, use of decode+eval, etc. to flag risky packages. Packj Github action [2] can alert if a risky dependency is pulled into your build. 1. https://github.com/ossillate-inc/packj https://github.com/ossillate-inc/packj 2. https://github.com/ossillate-inc/packj-github-action https://github.com/ossillate-inc/packj-github-action
oblio 3y ago
> Once they're known. Isn't that how life works? Unknown unknowns? :-) > I suppose nexus firewall let log4j pass. It should be reasonable to assume that a product dedicated to something will, on average, do this job better/faster than a random developer, though. Since that's their job. So Nexus Firewall presumably blocks these vulnerabilities faster than the average app/system developer notices them. There's value in reducing the delta between the vulnerability exploit day and an entire ecosystem being patched/fixed.
livealight 3y ago
It let log4j pass for as long as it was known to be good. Within hours of the CVE opening the tool was blocking it. The purpose of dependency firewalls is to avoid two things: known badly vulnerable packages AND known malicious packages that serve no other purpose than to steal data or drop a trojan. No security is 100% bulletproof, but it's really surprising how much of the damage is done by 7 year old CVEs. Firewalls can be useful in exactly that.

KronisLV 3y ago

> As someone who has worked in a big corporate their entire career this statement is bizarre. This has been solved for Java for over a decade. > Builds don't access the internet directly. You host a proxy on-premise (Nexus/Artifactory/etc) and access that. If the internet location is down, you continue as normal. Right now I'm using Sonatype Nexus (the free version) for my personal needs: https://www.sonatype.com/products/sonatype-nexus-repository https://www.sonatype.com/products/sonatype-nexus-repository And I do mean most things: apt packages, Docker images, Helm charts, as well as packages for Go, Java, .NET, Node, Python and Ruby, maybe even more in the future. I'm very glad that it is possible to do something like that, but honestly it's hard to do and I can easily see why most wouldn't bother: even though the configuration UI is okay and you can create users with appropriate permissions, you'll still need to have plenty of storage, manage updates and backups, setup automated cleanup tasks (especially for Docker images), in addition to the whole process on the developer's side sometimes being a nuisance. With npm you get --registry which is one of the simpler approaches, Ruby Gems needed some CLI commands along the lines of "bundle config mirror" and other stuff, Maven needs an XML file with the configuration, in addition to publishing your own packages to those also needing more configuration. Sadly, it seems like this fragmentation of approaches is unavoidable and complexity increases with the amount of languages you use. On the bright side, I keep control over my data, as well as have really good download/upload speeds, even on a cheap VPS. Plus you learn some vaguely useful things the more you explore if you want to go further, like how I also start with Debian/Ubuntu base container images and provision whatever tools or runtimes I need for more specialized images (e.g. my own JDK, .NET, Node etc. images).

oblio 3y ago
I don't know if they ever implemented it, can it offload the storage to S3? The cost is peanuts and you stop bothering about the annoying recurrent storage issues.
- KronisLV 3y ago
  > I don't know if they ever implemented it, can it offload the storage to S3? There are actually multiple different kinds of blob store backends: https://help.sonatype.com/repomanager3/nexus-repository-administration/repository-management/configuring-blob-stores https://help.sonatype.com/repomanager3/nexus-repository-admi... though so far I've only used the File one, because my servers have incremental backups that can be restored in a few clicks (that I've sometimes done in the past).

keep_reading 3y ago

I have CI pipelines fail when Github goes down because of crates.io being inaccessible. Does anyone have a proxy for Rust? I'd love to run it. Please. Someone... help edit: just learned Panamax exists for this... will try it