8 ms·

Malicious remote code execution backdoor discovered bootstrap-sass Ruby gem

147 points by qzio 8y ago

thibaut_barrere 8y ago
Relevant links: - https://github.com/twbs/bootstrap-sass/issues/1195 https://github.com/twbs/bootstrap-sass/issues/1195 - https://github.com/rubygems/rubygems.org/issues/1941 https://github.com/rubygems/rubygems.org/issues/1941
fxfan 8y ago
Rust has an effort going on (by a single unaffiliated person) to have a trust chain sort of thing per version. Cargo-cred. I hope others copy this. This is truly scary.
Foxboron 8y ago
I hope this will help rubygems adopting reproducible builds. Currently their issue with SOURCE_DATE_EPOCH is not yet supported properly. https://github.com/rubygems/rubygems/issues/2290 https://github.com/rubygems/rubygems/issues/2290 https://reproducible-builds.org/docs/source-date-epoch/ https://reproducible-builds.org/docs/source-date-epoch/
inglor 8y ago
Great writeup by Liran, he's also a member of the Node.js security team and I recommend checking his talks about package ecosystem security.
- thibaut_barrere 8y ago
  Ultimately, I hope we end up getting (in some future) some form of "cross language" global network of versions & vulnerabilities signaling, with e.g. alerts when you install something, or at runtime, and push notifications for all "subscribers" (outside of a paid product).
  - tannhaeuser 8y ago
    OWASP [1] maintains npm, maven (and nuget, I believe) plugins for detecting use of packages with known vulnerabilities, and a vulnerability db, of course. [1]: https://www.owasp.org/ https://www.owasp.org/
- the_duke 8y ago
  Talk here: https://vimeo.com/274629722 https://vimeo.com/274629722
jlangenauer 8y ago
For those who haven't yet come across it, bundler-audit is a very useful tool to pick up any old and/or insecure gems in your application. It should be part of your CI pipeline, or at a minimum, run locally every so often. https://github.com/rubysec/bundler-audit https://github.com/rubysec/bundler-audit
- thibaut_barrere 8y ago
  Note that it won't yet detect this specific vulnerability: https://github.com/rubysec/ruby-advisory-db/pull/386 https://github.com/rubysec/ruby-advisory-db/pull/386
- TicklishTiger 8y ago
  That's great. Is there also a bundler-meta-audit the somewhere, so I can audit bundler-audit and it's dependencies? If not, maybe we should start a kickstarter for it?
rajangdavis 8y ago
This is very timely: I was trying to start up an old Rails app today and when I ran the bundle install command, I got a warning that there were no sources for this gem... now I'm glad that this failed to install!
F117-DK 8y ago
This is scary. Gems don't have any audits?
- pmontra 8y ago
  Not this kind of audits. Is there any major language for which all third party modules are audited for security before they are released? I would be surprised.
- raesene9 8y ago
  gems don't, NPM doesn't, PyPI doesn't, NuGet doesn't.... you get the picture. Basically all apps using package repo's (i.e. all of them) are relying on massive piles of unaudited 3rd party code with usually no idea of provenance.
saurabhnanda 8y ago
> We assume that the attacker has obtained the credentials to publish the malicious RubyGems package from one of the two maintainers, but this has not been officially confirmed. Didn't this _also_ require a commit to the relevant Git repo? Or is it possible to upload a tarball directly to Rubygems without it being backed by a git repo?
- joekrill 8y ago
  5th Paragraph in: > The backdoor was wisely hidden in the 3.2.0.3 version that was only published to RubyGems and no source of the malicious version existed on the GitHub repository and allowed remote attackers to dynamically execute code on servers hosting the vulnerable versions.
deanclatworthy 8y ago
I really don't know what we can do as developers to prevent this. What else could be done? Is there any package manager system that isn't vulnerable to an individual being compromised? Having an audit-tool is only useful after you've been compromised and the vulnerability is discovered. Some kind of vulnerability scanner in all pulled in code might be one idea, but much like the cat-mouse battle with AV's, I don't think would be very successful. Ideas (shoot them down): - Somehow tie keys to location. - Require two-step verification for all commits for packages with over 1000 downloads. (Assuming their phone isn't compromised if the keys are).
- quietbritishjim 8y ago
  > Is there any package manager system that isn't vulnerable to an individual being compromised? One that comes closer is vcpkg [1], which is the Microsoft-run package manager for native C++ packages. It works by having the list of packages and recipes to build them (which it calls "ports") stored in a single repository. All requests to update them are submitted as pull requests and must be approved by Microsoft to enter the main repository. I'm not necessarily saying whether this is a good or bad idea. Certainly it is a very different situation: only libraries that are already popular are considered, often the ports are submitted/maintained by different people than the library authors, and writing a port to build a library on multiple platforms may require significantly more work than just pointing at an existing GitHub repo. I just point to it as an alternative idea that has had some success.
  - ttsda 8y ago
    That's how Homebrew[1] works as well. Any external file pulled during installation requires a matching hash in the main homebrew repo to prevent tampering after publishing. [1] https://brew.sh/ https://brew.sh/
- raesene9 8y ago
  Lots of things could be done, it's whether people are willing to accept the tradeoffs, which they haven't been (in general) up until now. Unfortunately, modern software development has heavily embedded the idea of using large numebers of un-reviewed open source libraries, changing that assumption would be very difficult. A non-exhaustive list of things that could be done... - Improve repository security (e.g. stronger authentication requirements). Problem here is most of the repositories are non-profit and this costs money. Also there's a risk that if you increase friction for developers, they'll go elsewhere. - Require library signing. This has some potential benefits (may even have stopped this attack) but again increased friction for developers and management overhead for repositories. - Curation of libraries. Actually pay some people to review the code in the libraries. This doesn't scale easily when you consider the volume that places like NPM have.
raesene9 8y ago
Another one in the line of inevitable supply chain attacks on software libraries. This isn't going to go away, it's just a question of if/when the damage caused will get bad enough that people start to accept some of the tradeoffs that would be required to improve matters...
- jimmychangas 8y ago
  Can you elaborate on what do you think would be a good solution to this problem? In my opinion, having big central sources of libraries makes it easier for security scanning services to operate. Companies hosting libraries locally (e.g. with Sinopia, Nexus, Artifactory) have to proactively monitor them, but most commonly will let them frozen at a specific version and miss all security patches.
  - raesene9 8y ago
    Really the only "solution" is to ensure that you have some way to establish trust in all the software libraries you use. For commercially provided libraries that could be a contract specifying security requirements, with some specific measures of how to do that. For Open source the only real option I can see is to curate your own package repositories and get a level of review that you're comfortable with. Definitely fixing versions and insisting on review before upgrade would help. The problem with automated scanning is that they can't really find backdoors or just generally insecure code, they can find known vulnerabilities. They could use static analysis to find insecure code, but my experience of SAST tooling is that it takes a lot of manual effort to tune, it's not a pure automated scan option. That's not to say that automated scanning provides nothing, but that it has limitations.
    - pjc50 8y ago
      > curate your own package repositories [and review them] This is a non-starter. This is even worse than "just run your own mail server".
      - 8y ago
RoryH 8y ago
Anyone maintaining a library with a large audience of consumers should be mandated to use 2FA/MFA, I can only assume from the information here that they only had a set of credentials for authing :-(
- ChrisCinelli 8y ago
  Having every package management system requiring 2FA to publish is a great idea. It may be a problem though when the build happens on a CI system.
intertextuality 8y ago
It's worse than I suspected. Rails 6.0, when generating a new app: - installs 102 ruby gems (without being asked) - installs yarn & 602 yarn dependencies (without being asked) The modern programming industry is a joke. We call ourselves "engineers" without any certification or oaths of responsibility, and basically ignore security beyond user credentials at best. Someone please tell me `yarn list | wc` showing 2,800 transient packages is incorrect. [0]: https://gist.github.com/azah/c219844f95243cdfdb3b352ad3dee2ff https://gist.github.com/azah/c219844f95243cdfdb3b352ad3dee2f...
- ChrisCinelli 8y ago
  We definitely trust more than we should. And it is worse that what you just found out. Did you think about all the software and dependencies that are installed just to get to the point where you can install Rails 6 (ex: Linux)? What about your coworkers? Are you sure that you can trust all of them and nobody will not sneak anything in? If you are too paranoid on who and what you can trust, you may end up tossing all your electronic devices and go living in a desert island.
  - intertextuality 8y ago
    Those attack vectors you mentioned are already known, and we essentially have to live with them. When it comes to "reflecting on trusting trust", we already know that it's basically impossible. However, this does not mean we should by default be installing heaps of packages upon generating new packages. This is practically begging for a security exploit, particularly the JS packages. [0]: https://www.archive.ece.cmu.edu/~ganger/712.fall02/papers/p761-thompson.pdf https://www.archive.ece.cmu.edu/~ganger/712.fall02/papers/p7...
    - ChrisCinelli 8y ago
      I am not sure why the npm packages in yarn should be more exploitable than Linux. - Linux is more ubiquitous than Node.js. If somebody exploits Linux, they get at least one order of magnitude more machines. So there is a higher motivation. - Funny code in C is harder to spot than in Javascript. Furthermore ... - A binary is harder to inspect than any module written in Javascript (even if minified). - The code in the Linux default installation has at least 2 orders of magnitudes of the code that yarn installs. I was very conservative in my estimations. Considering these kind of attacks, Linux seems more likely to be already exploited by a few organizations in multiple ways. We have been trusting too much. As more code gets written and many more people come to learn to program, the number of supply chain attacks will increase. And at the same time, as the security know-how is easier to be accessed on the Internet, we will get smarter exploiters capable of hiding their wrong doing.
deleted 8y ago
[deleted]
dpc_pw 8y ago
Relevant I'm working on a system for auditing open source libraries: https://github.com/dpc/crev https://github.com/dpc/crev . Currently focused on Rust/crates.io.
h1d 8y ago
Do people in mission critical systems (like life threatening) write code from scratch all the time without using any external dependencies? I'm sure they don't reinvent encryption. I'd like to know how they audit their system and the threshold of saying it is adequately secure.

147 points by qzio 8y ago

Relevant links: - https://github.com/twbs/bootstrap-sass/issues/1195 https://github.com/twbs/bootstrap-sass/issues/1195 - https://github.com/rubygems/rubygems.org/issues/1941 https://github.com/rubygems/rubygems.org/issues/1941

fxfan 8y ago

Rust has an effort going on (by a single unaffiliated person) to have a trust chain sort of thing per version. Cargo-cred. I hope others copy this. This is truly scary.

Foxboron 8y ago

I hope this will help rubygems adopting reproducible builds. Currently their issue with SOURCE_DATE_EPOCH is not yet supported properly. https://github.com/rubygems/rubygems/issues/2290 https://github.com/rubygems/rubygems/issues/2290 https://reproducible-builds.org/docs/source-date-epoch/ https://reproducible-builds.org/docs/source-date-epoch/

inglor 8y ago

Great writeup by Liran, he's also a member of the Node.js security team and I recommend checking his talks about package ecosystem security.

thibaut_barrere 8y ago
Ultimately, I hope we end up getting (in some future) some form of "cross language" global network of versions & vulnerabilities signaling, with e.g. alerts when you install something, or at runtime, and push notifications for all "subscribers" (outside of a paid product).
- tannhaeuser 8y ago
  OWASP [1] maintains npm, maven (and nuget, I believe) plugins for detecting use of packages with known vulnerabilities, and a vulnerability db, of course. [1]: https://www.owasp.org/ https://www.owasp.org/
the_duke 8y ago
Talk here: https://vimeo.com/274629722 https://vimeo.com/274629722

jlangenauer 8y ago

For those who haven't yet come across it, bundler-audit is a very useful tool to pick up any old and/or insecure gems in your application. It should be part of your CI pipeline, or at a minimum, run locally every so often. https://github.com/rubysec/bundler-audit https://github.com/rubysec/bundler-audit

thibaut_barrere 8y ago
Note that it won't yet detect this specific vulnerability: https://github.com/rubysec/ruby-advisory-db/pull/386 https://github.com/rubysec/ruby-advisory-db/pull/386
TicklishTiger 8y ago
That's great. Is there also a bundler-meta-audit the somewhere, so I can audit bundler-audit and it's dependencies? If not, maybe we should start a kickstarter for it?

rajangdavis 8y ago

This is very timely: I was trying to start up an old Rails app today and when I ran the bundle install command, I got a warning that there were no sources for this gem... now I'm glad that this failed to install!

F117-DK 8y ago

This is scary. Gems don't have any audits?

pmontra 8y ago
Not this kind of audits. Is there any major language for which all third party modules are audited for security before they are released? I would be surprised.
raesene9 8y ago
gems don't, NPM doesn't, PyPI doesn't, NuGet doesn't.... you get the picture. Basically all apps using package repo's (i.e. all of them) are relying on massive piles of unaudited 3rd party code with usually no idea of provenance.

saurabhnanda 8y ago

> We assume that the attacker has obtained the credentials to publish the malicious RubyGems package from one of the two maintainers, but this has not been officially confirmed. Didn't this _also_ require a commit to the relevant Git repo? Or is it possible to upload a tarball directly to Rubygems without it being backed by a git repo?

joekrill 8y ago
5th Paragraph in: > The backdoor was wisely hidden in the 3.2.0.3 version that was only published to RubyGems and no source of the malicious version existed on the GitHub repository and allowed remote attackers to dynamically execute code on servers hosting the vulnerable versions.

deanclatworthy 8y ago

I really don't know what we can do as developers to prevent this. What else could be done? Is there any package manager system that isn't vulnerable to an individual being compromised? Having an audit-tool is only useful after you've been compromised and the vulnerability is discovered. Some kind of vulnerability scanner in all pulled in code might be one idea, but much like the cat-mouse battle with AV's, I don't think would be very successful. Ideas (shoot them down): - Somehow tie keys to location. - Require two-step verification for all commits for packages with over 1000 downloads. (Assuming their phone isn't compromised if the keys are).

quietbritishjim 8y ago
> Is there any package manager system that isn't vulnerable to an individual being compromised? One that comes closer is vcpkg [1], which is the Microsoft-run package manager for native C++ packages. It works by having the list of packages and recipes to build them (which it calls "ports") stored in a single repository. All requests to update them are submitted as pull requests and must be approved by Microsoft to enter the main repository. I'm not necessarily saying whether this is a good or bad idea. Certainly it is a very different situation: only libraries that are already popular are considered, often the ports are submitted/maintained by different people than the library authors, and writing a port to build a library on multiple platforms may require significantly more work than just pointing at an existing GitHub repo. I just point to it as an alternative idea that has had some success.
- ttsda 8y ago
  That's how Homebrew[1] works as well. Any external file pulled during installation requires a matching hash in the main homebrew repo to prevent tampering after publishing. [1] https://brew.sh/ https://brew.sh/
raesene9 8y ago
Lots of things could be done, it's whether people are willing to accept the tradeoffs, which they haven't been (in general) up until now. Unfortunately, modern software development has heavily embedded the idea of using large numebers of un-reviewed open source libraries, changing that assumption would be very difficult. A non-exhaustive list of things that could be done... - Improve repository security (e.g. stronger authentication requirements). Problem here is most of the repositories are non-profit and this costs money. Also there's a risk that if you increase friction for developers, they'll go elsewhere. - Require library signing. This has some potential benefits (may even have stopped this attack) but again increased friction for developers and management overhead for repositories. - Curation of libraries. Actually pay some people to review the code in the libraries. This doesn't scale easily when you consider the volume that places like NPM have.

raesene9 8y ago

Another one in the line of inevitable supply chain attacks on software libraries. This isn't going to go away, it's just a question of if/when the damage caused will get bad enough that people start to accept some of the tradeoffs that would be required to improve matters...

jimmychangas 8y ago
Can you elaborate on what do you think would be a good solution to this problem? In my opinion, having big central sources of libraries makes it easier for security scanning services to operate. Companies hosting libraries locally (e.g. with Sinopia, Nexus, Artifactory) have to proactively monitor them, but most commonly will let them frozen at a specific version and miss all security patches.
- raesene9 8y ago
  Really the only "solution" is to ensure that you have some way to establish trust in all the software libraries you use. For commercially provided libraries that could be a contract specifying security requirements, with some specific measures of how to do that. For Open source the only real option I can see is to curate your own package repositories and get a level of review that you're comfortable with. Definitely fixing versions and insisting on review before upgrade would help. The problem with automated scanning is that they can't really find backdoors or just generally insecure code, they can find known vulnerabilities. They could use static analysis to find insecure code, but my experience of SAST tooling is that it takes a lot of manual effort to tune, it's not a pure automated scan option. That's not to say that automated scanning provides nothing, but that it has limitations.
  - pjc50 8y ago
    > curate your own package repositories [and review them] This is a non-starter. This is even worse than "just run your own mail server".
    - 8y ago

RoryH 8y ago

Anyone maintaining a library with a large audience of consumers should be mandated to use 2FA/MFA, I can only assume from the information here that they only had a set of credentials for authing :-(

ChrisCinelli 8y ago
Having every package management system requiring 2FA to publish is a great idea. It may be a problem though when the build happens on a CI system.

intertextuality 8y ago

It's worse than I suspected. Rails 6.0, when generating a new app: - installs 102 ruby gems (without being asked) - installs yarn & 602 yarn dependencies (without being asked) The modern programming industry is a joke. We call ourselves "engineers" without any certification or oaths of responsibility, and basically ignore security beyond user credentials at best. Someone please tell me `yarn list | wc` showing 2,800 transient packages is incorrect. [0]: https://gist.github.com/azah/c219844f95243cdfdb3b352ad3dee2ff https://gist.github.com/azah/c219844f95243cdfdb3b352ad3dee2f...

ChrisCinelli 8y ago
We definitely trust more than we should. And it is worse that what you just found out. Did you think about all the software and dependencies that are installed just to get to the point where you can install Rails 6 (ex: Linux)? What about your coworkers? Are you sure that you can trust all of them and nobody will not sneak anything in? If you are too paranoid on who and what you can trust, you may end up tossing all your electronic devices and go living in a desert island.
- intertextuality 8y ago
  Those attack vectors you mentioned are already known, and we essentially have to live with them. When it comes to "reflecting on trusting trust", we already know that it's basically impossible. However, this does not mean we should by default be installing heaps of packages upon generating new packages. This is practically begging for a security exploit, particularly the JS packages. [0]: https://www.archive.ece.cmu.edu/~ganger/712.fall02/papers/p761-thompson.pdf https://www.archive.ece.cmu.edu/~ganger/712.fall02/papers/p7...
  - ChrisCinelli 8y ago
    I am not sure why the npm packages in yarn should be more exploitable than Linux. - Linux is more ubiquitous than Node.js. If somebody exploits Linux, they get at least one order of magnitude more machines. So there is a higher motivation. - Funny code in C is harder to spot than in Javascript. Furthermore ... - A binary is harder to inspect than any module written in Javascript (even if minified). - The code in the Linux default installation has at least 2 orders of magnitudes of the code that yarn installs. I was very conservative in my estimations. Considering these kind of attacks, Linux seems more likely to be already exploited by a few organizations in multiple ways. We have been trusting too much. As more code gets written and many more people come to learn to program, the number of supply chain attacks will increase. And at the same time, as the security know-how is easier to be accessed on the Internet, we will get smarter exploiters capable of hiding their wrong doing.

deleted 8y ago

[deleted]

dpc_pw 8y ago

Relevant I'm working on a system for auditing open source libraries: https://github.com/dpc/crev https://github.com/dpc/crev . Currently focused on Rust/crates.io.

h1d 8y ago

Do people in mission critical systems (like life threatening) write code from scratch all the time without using any external dependencies? I'm sure they don't reinvent encryption. I'd like to know how they audit their system and the threshold of saying it is adequately secure.