10 ms·

100% this. My CSIO has been harping on this since 2018. I’ve been harping on this since 2019. Security vendors are JUST now starting to provide SCA offerings (s

by d4mi3n 4y ago

100% this. My CSIO has been harping on this since 2018. I’ve been harping on this since 2019. Security vendors are JUST now starting to provide SCA offerings (software composition analysis, for those not in infosec). Events like these don’t surprise anyone who’s been following this space. It’s been a rough ride that will get rougher. Consider: 1. NPM just recently started enforcing 2FA for some packages. The NPM has yet to support mechanisms like package signing. Post install scripts are still a thing. 2. Rubygems does not support package signing. 3. Go dependency management is a hot mess. 4. Clojure dependency management is still a hot mess. Initiatives like Google’s SLSA will help here, but there’s a whole ecosystem of stuff across multiple technical ecosystems they need to mature, and fast.

roblabla 4y ago
I agree with most of your post, but I do not understand this: > Post install scripts are still a thing I cannot find a situation where post-install script actually pose a security issue. I find that you'll always end up running the code you just installed on your machine (either through running unit tests, or running the actual code, or whatever). Hence, the only scenario where post-install scripts are a problem is: - You install deps on a machine that has access to security sensitive resources (think: your ssh keys) - BUT you do not run the resulting code on that machine. Are there really any use-case that satisfies those requirement where post-install scripts would actually result in a security issue?
- dec0dedab0de 4y ago
  NPM for compiled frontend JavaScript. The resulting code only gets run in your users browsers, but the post install script could be run on your Jenkins/whatever box. In general, anyone who gathers dependencies ahead of time before deployment, so they can just copy them over. So most people deploying to multiple machines/containers would be at a greater risk from a post install script
  - icambron 4y ago
    While this makes sense, my experience is that you end up with just a handful of dependencies for your actual product (“dependencies” in NPM parlance), but a whole herd of them for your build toolchain (“dev dependencies”). This happens because front end libraries don’t often have many, if any, dependencies themselves for fear bloating the JS slugs using them, whereas the Node libraries used to build it do not need to sweat that and each comes with a whole tree of transitive dependencies. So the exposure your build server has to supply chain attacks for code it runs directly is huge, much bigger than what’s just passing through. And for those toolchain reps, the GP’s point stands.
  - roblabla 4y ago
    Ah, yes, I had totally failed to consider that NPM is now used much more widely than just for nodejs server code, and can be used for browser deps. Thanks for this!
- ilammy 4y ago
  Maintainer scripts lower the bar for the attack. In order to get the code in scripts executed at a predictable time – you add it there and done. In order to get the code in the library executed on the build box, you have to add it somewhere that's actually used by the environment you want to attack. Scripts allow more coverage more easily. Plus, it's not unheard for people to run "npm" as root to run some scripts as root.
- larusso 4y ago
  Given the fact that deployments can also include a npm install, depending how you package and run your code. But that is not the point. An attacker can run code in the name of the current user. He can download and inject other code on the host or the project. Which means that even if you bundle your code in a docker image and you never run the install on the same machine, some code could have been injected. And I personally see any attack being against a production system or development machine relevant.
- theteapot 4y ago
  robabla, are you an NPM maintainer? They all seems to hold the same opinion despite the exploits being enumerated for them over and over again.
  - roblabla 4y ago
    No, I am not. I'm just an interested bystander. I don't even code much JavaScript anymore.
mperham 4y ago
Rubygems does support GPG signing but no one uses it.
- jacques_chester 4y ago
  It's actually x509 based, but otherwise very PGP-like in its trust dynamics. But as you say, basically nobody uses it.
skybrian 4y ago
What are you seeing with Go dependency management?
thenerdhead 4y ago
Check out some of the initiatives across the board: https://github.com/ossf https://github.com/ossf (Lots of WG and efforts such as package analysis, scorecards, etc) https://deps.dev/ https://deps.dev/ (Implements OSSF scorecard)
- jacques_chester 4y ago
  In particular, check out the Securing Software Repos WG: https://github.com/ossf/wg-securing-software-repos https://github.com/ossf/wg-securing-software-repos So far folks have turned up from RubyGems, PyPI, NPM, Maven Central, Gradle, Drupal and I'm probably forgotten someone.
dane-pgp 4y ago
> The NPM has yet to support mechanisms like package signing. Yes, but actually no. "To increase confidence in the npm public registry, we add our PGP signature to package metadata and publicize our public PGP key on Keybase."[0] Unfortunately (and perhaps unsurprisingly), this means NPM is still a single point of failure, and this doesn't add any defence in depth. I guess what you want is signatures by the package maintainers themselves, but key management is hard for both signers and verifiers. [0] https://docs.npmjs.com/about-pgp-signatures-for-packages-in-the-public-registry/ https://docs.npmjs.com/about-pgp-signatures-for-packages-in-...
- d4mi3n 4y ago
  IIRC when I examined this a couple years ago—it doesn’t seem to have changed—this is about NPM signing packages submitted to it’s registry. This can give confidence they packages are not tampered with when one downloads them from NPM. What this is not is support for package maintainers or CI/CD systems to sign artifacts they upload to NPM. Supporting this would give consumers of said packages the ability to detect if a package maintainer was compromised—the signature on a new version would either be invalid or different.
delusional 4y ago
I'm not in infosec (or any sec really, I'm a dev in a very immature organization), but I don't think it's a tooling problem, but rather a people problem. Software has reached a crazy amount of complexity. Layers are built to compensate for the layers that come before them, followed by new layers to undo that very layer. The hardware we run out software on has never been so uniform, yet our software has never been more abstract. That abstraction is then implemented in millions of lines of dependencies that people cargo cult into their projects. The problem with go packages isn't the security of the package manager, but rather the fact that I need to download some arbitrary go code from github to make uuids. "Pull this random package" has become the new "copy paste from stackoverflow without reading the snippet".
- d4mi3n 4y ago
  Security is absolutely a people problem, but in general good security tends to make it hard to do dangerous things. I do not believe most development tooling makes it easy to be safe while also making it hard to do generally dangerous things—like downloading unsigned, unverified code from an unknown third party.
halostatue 4y ago
Point 2 (Rubygems does not support package signing) is not true. Rubygems has supported package signing (`gem help cert`) since very early on, and it has an install flag `--trust-policy` which can be used to verify various things, including certs (https://github.com/rubygems/rubygems/blob/96e5cff3df491c4d943186804c6d03b57fcb459b/lib/rubygems/security/policies.rb https://github.com/rubygems/rubygems/blob/96e5cff3df491c4d94...). The experience in using it, however, sucks on every level. No one can really use the `High Security` policy level, because most gems aren’t signed. Most gems aren’t signed because there’s no clear benefit and it’s non-trivial to have shared certificates that can be used by multiple people authorized to release a particular gem. Most gems aren’t signed because there’s nowhere that public gem certs are published (there used to be with rubyforge), and you have to track down each cert you want to verify and download it separately. I used to sign my gems, but then stopped. Shopify has proposed a new RFC for signing gems based on sigstore. This RFC has many of the same points that I have already made as a reason for changing mechanisms. https://github.com/Shopify/rfcs/blob/new-signing-mechanism/text/0000-introduce-a-new-signing-mechanism.md https://github.com/Shopify/rfcs/blob/new-signing-mechanism/t... I’ve just discovered this, so I haven’t really evaluated it, but I would prefer to sign the gems I publish.
wbl 4y ago
How is go still a hot mess? Unlike the other systems you know exactly what went in by default and it's easy to vendor.
Thaxll 4y ago
3. Go mod is excellent and actually protect against supply chain issues, it's one of the few to do so. https://go.dev/blog/supply-chain https://go.dev/blog/supply-chain
scoot 4y ago
> Security vendors are JUST now starting to provide SCA offerings Sorry, but that just isn't true. Solutions have existed for years [1], but enterprises just weren't interested. [1] For example Sonatype who I know about from a former colleague (ex. Sonatype employee). They have existed for 14 years.
- d4mi3n 4y ago
  Could you point me at the product from Sonatype you have in mind? I’d genuinely love to see it. I’ve seen their SAST offerings, but not much around SCA/SBOM/supply chain analysis. It’s also worth pointing out that my original comment was probably less specific than it should have been—SCA isn’t new, but tools to help and mitigate risks around the digital supply chain have up until recently been fairly primitive as far as I’ve personally seen. Many vendors can tell me a package has a vuln. Very few can tell me which vulns can actually be exploited in the context of applications I secure. Fewer still can give me a cohesive view over time of what packages or maintainers start doing things out of the ordinary.
  - deleted 4y ago
    [deleted]
  - axsharma 4y ago
    Hi there, Ax Sharma here from Sonatype - I've written extensively about our malware/hijacked package findings almost every week now on the company blog. The automated malware detection bots flag anything that looks suspicious on npm/PyPI and "quarantine" the packages until a manual analysis from researchers is pending. Nexus Firewall automatically blocks malware and malicious typosquats, hijacked packages and dependency confusion attacks with algorithms now being expanded to cover self-sabotages (In fact, I first broke news on dependency confusion along with researcher Alex Birsan on the company blog and BleepingComputer). As such, before the attacks even picked up steam, Sonatype already had a solution for it and been blocking these for months - but coordinated disclosure agreement for PoC research delayed our public disclosure. Nexus IQ/Lifecycle is more for SBOM/vulnerabilities, including those without a CVE - e.g. reported via GitHub Issues and other sources. The vulnerability scanning looks for the exact occurrence of vulnerable code rather than just flagging any and all artifacts for a given component, which makes it quite precise imo. For SCA, there's Sonatype Lift that connects to your GitHub repo for free so you can test drive it before moving on to other offerings. Thanks, and I hope it helps.
zerkten 4y ago
I could only find this PyCon 2012 talk (https://pyvideo.org/pycon-us-2012/advanced-security-topics.html https://pyvideo.org/pycon-us-2012/advanced-security-topics.h...), but there was a session in 2010 that was perhaps more zeroed in on this topic. It was evident then that these problems affected every packaging and dependency ecosystem, but it was hard to get devs to take it seriously. The pattern plays out with every new wave in our industry. You can even see this with the k8s ecosystem where there is a ton of complexity and opportunity for attackers. It's only later that security is taken more seriously when it could be designed in from the start. We love to mitigate things that we could eliminate in original designs.

by d4mi3n 4y ago

roblabla 4y ago

I agree with most of your post, but I do not understand this: > Post install scripts are still a thing I cannot find a situation where post-install script actually pose a security issue. I find that you'll always end up running the code you just installed on your machine (either through running unit tests, or running the actual code, or whatever). Hence, the only scenario where post-install scripts are a problem is: - You install deps on a machine that has access to security sensitive resources (think: your ssh keys) - BUT you do not run the resulting code on that machine. Are there really any use-case that satisfies those requirement where post-install scripts would actually result in a security issue?

dec0dedab0de 4y ago
NPM for compiled frontend JavaScript. The resulting code only gets run in your users browsers, but the post install script could be run on your Jenkins/whatever box. In general, anyone who gathers dependencies ahead of time before deployment, so they can just copy them over. So most people deploying to multiple machines/containers would be at a greater risk from a post install script
- icambron 4y ago
  While this makes sense, my experience is that you end up with just a handful of dependencies for your actual product (“dependencies” in NPM parlance), but a whole herd of them for your build toolchain (“dev dependencies”). This happens because front end libraries don’t often have many, if any, dependencies themselves for fear bloating the JS slugs using them, whereas the Node libraries used to build it do not need to sweat that and each comes with a whole tree of transitive dependencies. So the exposure your build server has to supply chain attacks for code it runs directly is huge, much bigger than what’s just passing through. And for those toolchain reps, the GP’s point stands.
- roblabla 4y ago
  Ah, yes, I had totally failed to consider that NPM is now used much more widely than just for nodejs server code, and can be used for browser deps. Thanks for this!
ilammy 4y ago
Maintainer scripts lower the bar for the attack. In order to get the code in scripts executed at a predictable time – you add it there and done. In order to get the code in the library executed on the build box, you have to add it somewhere that's actually used by the environment you want to attack. Scripts allow more coverage more easily. Plus, it's not unheard for people to run "npm" as root to run some scripts as root.
larusso 4y ago
Given the fact that deployments can also include a npm install, depending how you package and run your code. But that is not the point. An attacker can run code in the name of the current user. He can download and inject other code on the host or the project. Which means that even if you bundle your code in a docker image and you never run the install on the same machine, some code could have been injected. And I personally see any attack being against a production system or development machine relevant.
theteapot 4y ago
robabla, are you an NPM maintainer? They all seems to hold the same opinion despite the exploits being enumerated for them over and over again.
- roblabla 4y ago
  No, I am not. I'm just an interested bystander. I don't even code much JavaScript anymore.

mperham 4y ago

Rubygems does support GPG signing but no one uses it.

jacques_chester 4y ago
It's actually x509 based, but otherwise very PGP-like in its trust dynamics. But as you say, basically nobody uses it.

skybrian 4y ago

What are you seeing with Go dependency management?

thenerdhead 4y ago

Check out some of the initiatives across the board: https://github.com/ossf https://github.com/ossf (Lots of WG and efforts such as package analysis, scorecards, etc) https://deps.dev/ https://deps.dev/ (Implements OSSF scorecard)

jacques_chester 4y ago
In particular, check out the Securing Software Repos WG: https://github.com/ossf/wg-securing-software-repos https://github.com/ossf/wg-securing-software-repos So far folks have turned up from RubyGems, PyPI, NPM, Maven Central, Gradle, Drupal and I'm probably forgotten someone.

dane-pgp 4y ago

> The NPM has yet to support mechanisms like package signing. Yes, but actually no. "To increase confidence in the npm public registry, we add our PGP signature to package metadata and publicize our public PGP key on Keybase."[0] Unfortunately (and perhaps unsurprisingly), this means NPM is still a single point of failure, and this doesn't add any defence in depth. I guess what you want is signatures by the package maintainers themselves, but key management is hard for both signers and verifiers. [0] https://docs.npmjs.com/about-pgp-signatures-for-packages-in-the-public-registry/ https://docs.npmjs.com/about-pgp-signatures-for-packages-in-...

d4mi3n 4y ago
IIRC when I examined this a couple years ago—it doesn’t seem to have changed—this is about NPM signing packages submitted to it’s registry. This can give confidence they packages are not tampered with when one downloads them from NPM. What this is not is support for package maintainers or CI/CD systems to sign artifacts they upload to NPM. Supporting this would give consumers of said packages the ability to detect if a package maintainer was compromised—the signature on a new version would either be invalid or different.

delusional 4y ago

I'm not in infosec (or any sec really, I'm a dev in a very immature organization), but I don't think it's a tooling problem, but rather a people problem. Software has reached a crazy amount of complexity. Layers are built to compensate for the layers that come before them, followed by new layers to undo that very layer. The hardware we run out software on has never been so uniform, yet our software has never been more abstract. That abstraction is then implemented in millions of lines of dependencies that people cargo cult into their projects. The problem with go packages isn't the security of the package manager, but rather the fact that I need to download some arbitrary go code from github to make uuids. "Pull this random package" has become the new "copy paste from stackoverflow without reading the snippet".

d4mi3n 4y ago
Security is absolutely a people problem, but in general good security tends to make it hard to do dangerous things. I do not believe most development tooling makes it easy to be safe while also making it hard to do generally dangerous things—like downloading unsigned, unverified code from an unknown third party.

halostatue 4y ago

Point 2 (Rubygems does not support package signing) is not true. Rubygems has supported package signing (`gem help cert`) since very early on, and it has an install flag `--trust-policy` which can be used to verify various things, including certs (https://github.com/rubygems/rubygems/blob/96e5cff3df491c4d943186804c6d03b57fcb459b/lib/rubygems/security/policies.rb https://github.com/rubygems/rubygems/blob/96e5cff3df491c4d94...). The experience in using it, however, sucks on every level. No one can really use the `High Security` policy level, because most gems aren’t signed. Most gems aren’t signed because there’s no clear benefit and it’s non-trivial to have shared certificates that can be used by multiple people authorized to release a particular gem. Most gems aren’t signed because there’s nowhere that public gem certs are published (there used to be with rubyforge), and you have to track down each cert you want to verify and download it separately. I used to sign my gems, but then stopped. Shopify has proposed a new RFC for signing gems based on sigstore. This RFC has many of the same points that I have already made as a reason for changing mechanisms. https://github.com/Shopify/rfcs/blob/new-signing-mechanism/text/0000-introduce-a-new-signing-mechanism.md https://github.com/Shopify/rfcs/blob/new-signing-mechanism/t... I’ve just discovered this, so I haven’t really evaluated it, but I would prefer to sign the gems I publish.

wbl 4y ago

How is go still a hot mess? Unlike the other systems you know exactly what went in by default and it's easy to vendor.

Thaxll 4y ago

3. Go mod is excellent and actually protect against supply chain issues, it's one of the few to do so. https://go.dev/blog/supply-chain https://go.dev/blog/supply-chain

scoot 4y ago

> Security vendors are JUST now starting to provide SCA offerings Sorry, but that just isn't true. Solutions have existed for years [1], but enterprises just weren't interested. [1] For example Sonatype who I know about from a former colleague (ex. Sonatype employee). They have existed for 14 years.

d4mi3n 4y ago
Could you point me at the product from Sonatype you have in mind? I’d genuinely love to see it. I’ve seen their SAST offerings, but not much around SCA/SBOM/supply chain analysis. It’s also worth pointing out that my original comment was probably less specific than it should have been—SCA isn’t new, but tools to help and mitigate risks around the digital supply chain have up until recently been fairly primitive as far as I’ve personally seen. Many vendors can tell me a package has a vuln. Very few can tell me which vulns can actually be exploited in the context of applications I secure. Fewer still can give me a cohesive view over time of what packages or maintainers start doing things out of the ordinary.
- deleted 4y ago
  [deleted]
- axsharma 4y ago
  Hi there, Ax Sharma here from Sonatype - I've written extensively about our malware/hijacked package findings almost every week now on the company blog. The automated malware detection bots flag anything that looks suspicious on npm/PyPI and "quarantine" the packages until a manual analysis from researchers is pending. Nexus Firewall automatically blocks malware and malicious typosquats, hijacked packages and dependency confusion attacks with algorithms now being expanded to cover self-sabotages (In fact, I first broke news on dependency confusion along with researcher Alex Birsan on the company blog and BleepingComputer). As such, before the attacks even picked up steam, Sonatype already had a solution for it and been blocking these for months - but coordinated disclosure agreement for PoC research delayed our public disclosure. Nexus IQ/Lifecycle is more for SBOM/vulnerabilities, including those without a CVE - e.g. reported via GitHub Issues and other sources. The vulnerability scanning looks for the exact occurrence of vulnerable code rather than just flagging any and all artifacts for a given component, which makes it quite precise imo. For SCA, there's Sonatype Lift that connects to your GitHub repo for free so you can test drive it before moving on to other offerings. Thanks, and I hope it helps.

zerkten 4y ago

I could only find this PyCon 2012 talk (https://pyvideo.org/pycon-us-2012/advanced-security-topics.html https://pyvideo.org/pycon-us-2012/advanced-security-topics.h...), but there was a session in 2010 that was perhaps more zeroed in on this topic. It was evident then that these problems affected every packaging and dependency ecosystem, but it was hard to get devs to take it seriously. The pattern plays out with every new wave in our industry. You can even see this with the k8s ecosystem where there is a ton of complexity and opportunity for attackers. It's only later that security is taken more seriously when it could be designed in from the start. We love to mitigate things that we could eliminate in original designs.