4 ms·

I really don't know what we can do as developers to prevent this. What else could be done? Is there any package manager system that isn't vulnerable to an indiv

I really don't know what we can do as developers to prevent this. What else could be done? Is there any package manager system that isn't vulnerable to an individual being compromised? Having an audit-tool is only useful after you've been compromised and the vulnerability is discovered. Some kind of vulnerability scanner in all pulled in code might be one idea, but much like the cat-mouse battle with AV's, I don't think would be very successful. Ideas (shoot them down): - Somehow tie keys to location. - Require two-step verification for all commits for packages with over 1000 downloads. (Assuming their phone isn't compromised if the keys are).

quietbritishjim 8y ago
> Is there any package manager system that isn't vulnerable to an individual being compromised? One that comes closer is vcpkg [1], which is the Microsoft-run package manager for native C++ packages. It works by having the list of packages and recipes to build them (which it calls "ports") stored in a single repository. All requests to update them are submitted as pull requests and must be approved by Microsoft to enter the main repository. I'm not necessarily saying whether this is a good or bad idea. Certainly it is a very different situation: only libraries that are already popular are considered, often the ports are submitted/maintained by different people than the library authors, and writing a port to build a library on multiple platforms may require significantly more work than just pointing at an existing GitHub repo. I just point to it as an alternative idea that has had some success.
- ttsda 8y ago
  That's how Homebrew[1] works as well. Any external file pulled during installation requires a matching hash in the main homebrew repo to prevent tampering after publishing. [1] https://brew.sh/ https://brew.sh/
raesene9 8y ago
Lots of things could be done, it's whether people are willing to accept the tradeoffs, which they haven't been (in general) up until now. Unfortunately, modern software development has heavily embedded the idea of using large numebers of un-reviewed open source libraries, changing that assumption would be very difficult. A non-exhaustive list of things that could be done... - Improve repository security (e.g. stronger authentication requirements). Problem here is most of the repositories are non-profit and this costs money. Also there's a risk that if you increase friction for developers, they'll go elsewhere. - Require library signing. This has some potential benefits (may even have stopped this attack) but again increased friction for developers and management overhead for repositories. - Curation of libraries. Actually pay some people to review the code in the libraries. This doesn't scale easily when you consider the volume that places like NPM have.
- repolfx 8y ago
  Also (on some runtimes): - Sandbox code modules If a library doesn't need any permissions, and it's not granted any, then attempts to e.g. access the filesystem or network will throw exceptions. The JVM can do this, but it's not well documented how to use it.
  - nih0 8y ago
    The java security manager is a cool concept but sadly not very ergonomic to use.
- bfred_it 8y ago
  > This doesn't scale easily when you consider the volume that places like NPM have. Found the blanket statement for npm. How about instead of reviewing 1 library of 10k lines that does everything, you review 10 focused modules for a total of 1k lines? Also modules like left-pad don’t change that often so you probably have to review only changes in the top modules. We need to stop this mentality that dependency count should be low. Look at total LOC and you’ll probably find that it’s not as bad as people like to paint it.
  - nfm 8y ago
    A high dependency count does increase your vulnerability surface area. All it takes is one weak or reused password on an account without 2FA enabled for a malicious package to be uploaded. You're more vulnerable to this when you depend on more individual publishers to get it right.
adrianN 8y ago
I propose the radical idea of drastically reducing the number of transitive dependencies we rely on. I don't think that it's okay that I need hundreds of megabytes of unreviewed libraries for running relatively simple programs.
Foxboron 8y ago
For this problem, where the resulting code does not correspond to the published source code, reproducible builds is an adequate solution. https://reproducible-builds.org/docs/buy-in/ https://reproducible-builds.org/docs/buy-in/ When it comes to verifying this, we need to have independent rebuilders that build these packages reproducibly, and can attest these rebuilds in a secure manner. I have been working with NYU Secure System Labs to try provide this to debian packages and we currently have a PoC for such a system. https://reproducible-builds.org/docs/sharing-certifications/ https://reproducible-builds.org/docs/sharing-certifications/ https://ssl.engineering.nyu.edu/blog/2019-01-18-in-toto-paris https://ssl.engineering.nyu.edu/blog/2019-01-18-in-toto-pari...
- ChrisCinelli 8y ago
  Not trivial to do when the build changes each time. For example adding the build number or the date of when it is built. But I think this goes in the right direction.
  - ChrisCinelli 8y ago
    Unfortunately a few well know projects do not let you compile from source easily. Brew for example is going in that direction.
  - Foxboron 8y ago
    Already adresse in the documentation in the form of "SOURCE_DATE_EPOCH". Adding these things to projects in general is terrible.
    - ChrisCinelli 8y ago
      Sure there are ways to do it, but every piece of software needs to be aligned. What I was trying to say is that even if you have the tools you probably cannot "just use the tools". You will likely have to create patches for those piece of softwares that create a different artifact every time are built. There are a few other challenges but in my humble opinion, it is worthwhile to strive to write your own software with "reproducible builts" in mind so everybody can verify if something was altered.
pjc50 8y ago
Debian had mandatory use of OpenPGP by developers for twenty years, although I don't think it's linked to strict package signing. Of course, this adds a big deliberate barrier to participation - newcomers have to get their key signed by an existing Debian developers. There are real security/usability tradeoffs, and at the moment people seem to have default-chosen "anyone can add packages really easily", resulting in a Cambrian explosion of packages.
- deanclatworthy 8y ago
  Keys are somewhat useless in this scenario. Developers are hacked all the time and not all of them pass phrase their keys. And if someone is going through the effort to backflip a popular package via a developer they’d probably keylig the pass phrase too.
ttsda 8y ago
For a start, you could detect redflags like a new "eval" call in a minor update and force manual verification...
- IshKebab 8y ago
  That would be easy to workaround.
fxfan 8y ago
See my other comment, rust already has a decent system (unofficial)
viraptor 8y ago
> Require two-step verification for all commits for packages with over 1000 downloads. Just require 2fa everywhere. We're way past the point where password-only is enough if you distribute software. Package distribution platforms could also ensure that your package comes from a tagged/signed commit in a selected repo. Reproducible builds could help here too. Finally package signing itself and possibly WoT for authors. (Where either the author has to trust the dependencies or you have to explicitly trust all of them)
- ChrisCinelli 8y ago
  This is going to practically stop remote attacks with stolen credential. But there are a few other attacks that it will not stop. For example the NPM event-stream kind of attacks.
zzzcpan 8y ago
There are some things to do, but not if you want to trust random third parties. In which case nothing will save you, because not only you have to trust them, you also have to trust they are not compromised and have impenetrable security. So, not trusting third parties is the only way and with that language level capability based security can help. Although this is pretty far from common programming languages where any function can access any file, environment variable, global variable, constant, socket, etc.
ChrisCinelli 8y ago
The only way to completely prevent to get malicious code in your products is writing all code by yourself including the interpreter/compiler and maybe the operating system too. Machines should be off the Internet of course and security should be there 24/7. And of course you need to have a very high bar and a paranoid process to select the people you hire. But now you will move at the pace of a bacteria on a snail. But since the company will probably introduce vulnerability bugs in the code they write, it is probably not a very good idea. We are using open source and free packages to save time and move fast. Security and speed/usability have tradeoffs. The paranoid will do everything in their knowledge to protect themselves but they will move slowly. The naive will not do anything and sooner than later will get burnt. Assuming you trust your current modules and you can trust that they will not be altered, you can lock down the dependencies. But now monitor and updating a library when a bug or vulnerability is found in a modules become another burden. Those actors that have the budget, patience and the motivation will always find a way in. I sometimes feel quite sad about the real situation but there is not such a thing as perfect security. That said some of the suggestions I see here will dramatically improve the situation with a relatively small effort. 2FA and reproducible builds for example.
intertextuality 8y ago
2fa should always be on. 1) Signing should always happen. Publish time + hashes of external resources should be taken into account to prevent meddling with things after the fact. 2- a much harder solution) Don't use so many libraries. In this way I think it's borderline impossible. Of course this vuln targeted `bootstrap-sass`.. it's basically a default gem for new rails installations since bootstrap is so commonly used. Actually, rails 6.0 not only installs dependencies for you upon generating a new app, but it downloads yarn and installs 602 dependencies too. It's a miracle we do anything as programmers.
chaz6 8y ago
I really like the idea of a cryptographically provable signature for a set of coordinates in space and time, but not sure how such a thing is possible.

quietbritishjim 8y ago

> Is there any package manager system that isn't vulnerable to an individual being compromised? One that comes closer is vcpkg [1], which is the Microsoft-run package manager for native C++ packages. It works by having the list of packages and recipes to build them (which it calls "ports") stored in a single repository. All requests to update them are submitted as pull requests and must be approved by Microsoft to enter the main repository. I'm not necessarily saying whether this is a good or bad idea. Certainly it is a very different situation: only libraries that are already popular are considered, often the ports are submitted/maintained by different people than the library authors, and writing a port to build a library on multiple platforms may require significantly more work than just pointing at an existing GitHub repo. I just point to it as an alternative idea that has had some success.

ttsda 8y ago
That's how Homebrew[1] works as well. Any external file pulled during installation requires a matching hash in the main homebrew repo to prevent tampering after publishing. [1] https://brew.sh/ https://brew.sh/

raesene9 8y ago

Lots of things could be done, it's whether people are willing to accept the tradeoffs, which they haven't been (in general) up until now. Unfortunately, modern software development has heavily embedded the idea of using large numebers of un-reviewed open source libraries, changing that assumption would be very difficult. A non-exhaustive list of things that could be done... - Improve repository security (e.g. stronger authentication requirements). Problem here is most of the repositories are non-profit and this costs money. Also there's a risk that if you increase friction for developers, they'll go elsewhere. - Require library signing. This has some potential benefits (may even have stopped this attack) but again increased friction for developers and management overhead for repositories. - Curation of libraries. Actually pay some people to review the code in the libraries. This doesn't scale easily when you consider the volume that places like NPM have.

repolfx 8y ago
Also (on some runtimes): - Sandbox code modules If a library doesn't need any permissions, and it's not granted any, then attempts to e.g. access the filesystem or network will throw exceptions. The JVM can do this, but it's not well documented how to use it.
- nih0 8y ago
  The java security manager is a cool concept but sadly not very ergonomic to use.
bfred_it 8y ago
> This doesn't scale easily when you consider the volume that places like NPM have. Found the blanket statement for npm. How about instead of reviewing 1 library of 10k lines that does everything, you review 10 focused modules for a total of 1k lines? Also modules like left-pad don’t change that often so you probably have to review only changes in the top modules. We need to stop this mentality that dependency count should be low. Look at total LOC and you’ll probably find that it’s not as bad as people like to paint it.
- nfm 8y ago
  A high dependency count does increase your vulnerability surface area. All it takes is one weak or reused password on an account without 2FA enabled for a malicious package to be uploaded. You're more vulnerable to this when you depend on more individual publishers to get it right.

adrianN 8y ago

I propose the radical idea of drastically reducing the number of transitive dependencies we rely on. I don't think that it's okay that I need hundreds of megabytes of unreviewed libraries for running relatively simple programs.

Foxboron 8y ago

For this problem, where the resulting code does not correspond to the published source code, reproducible builds is an adequate solution. https://reproducible-builds.org/docs/buy-in/ https://reproducible-builds.org/docs/buy-in/ When it comes to verifying this, we need to have independent rebuilders that build these packages reproducibly, and can attest these rebuilds in a secure manner. I have been working with NYU Secure System Labs to try provide this to debian packages and we currently have a PoC for such a system. https://reproducible-builds.org/docs/sharing-certifications/ https://reproducible-builds.org/docs/sharing-certifications/ https://ssl.engineering.nyu.edu/blog/2019-01-18-in-toto-paris https://ssl.engineering.nyu.edu/blog/2019-01-18-in-toto-pari...

ChrisCinelli 8y ago
Not trivial to do when the build changes each time. For example adding the build number or the date of when it is built. But I think this goes in the right direction.
- ChrisCinelli 8y ago
  Unfortunately a few well know projects do not let you compile from source easily. Brew for example is going in that direction.
- Foxboron 8y ago
  Already adresse in the documentation in the form of "SOURCE_DATE_EPOCH". Adding these things to projects in general is terrible.
  - ChrisCinelli 8y ago
    Sure there are ways to do it, but every piece of software needs to be aligned. What I was trying to say is that even if you have the tools you probably cannot "just use the tools". You will likely have to create patches for those piece of softwares that create a different artifact every time are built. There are a few other challenges but in my humble opinion, it is worthwhile to strive to write your own software with "reproducible builts" in mind so everybody can verify if something was altered.

pjc50 8y ago

Debian had mandatory use of OpenPGP by developers for twenty years, although I don't think it's linked to strict package signing. Of course, this adds a big deliberate barrier to participation - newcomers have to get their key signed by an existing Debian developers. There are real security/usability tradeoffs, and at the moment people seem to have default-chosen "anyone can add packages really easily", resulting in a Cambrian explosion of packages.

deanclatworthy 8y ago
Keys are somewhat useless in this scenario. Developers are hacked all the time and not all of them pass phrase their keys. And if someone is going through the effort to backflip a popular package via a developer they’d probably keylig the pass phrase too.

ttsda 8y ago

For a start, you could detect redflags like a new "eval" call in a minor update and force manual verification...

IshKebab 8y ago
That would be easy to workaround.

fxfan 8y ago

See my other comment, rust already has a decent system (unofficial)

viraptor 8y ago

> Require two-step verification for all commits for packages with over 1000 downloads. Just require 2fa everywhere. We're way past the point where password-only is enough if you distribute software. Package distribution platforms could also ensure that your package comes from a tagged/signed commit in a selected repo. Reproducible builds could help here too. Finally package signing itself and possibly WoT for authors. (Where either the author has to trust the dependencies or you have to explicitly trust all of them)

ChrisCinelli 8y ago
This is going to practically stop remote attacks with stolen credential. But there are a few other attacks that it will not stop. For example the NPM event-stream kind of attacks.

zzzcpan 8y ago

There are some things to do, but not if you want to trust random third parties. In which case nothing will save you, because not only you have to trust them, you also have to trust they are not compromised and have impenetrable security. So, not trusting third parties is the only way and with that language level capability based security can help. Although this is pretty far from common programming languages where any function can access any file, environment variable, global variable, constant, socket, etc.

ChrisCinelli 8y ago

The only way to completely prevent to get malicious code in your products is writing all code by yourself including the interpreter/compiler and maybe the operating system too. Machines should be off the Internet of course and security should be there 24/7. And of course you need to have a very high bar and a paranoid process to select the people you hire. But now you will move at the pace of a bacteria on a snail. But since the company will probably introduce vulnerability bugs in the code they write, it is probably not a very good idea. We are using open source and free packages to save time and move fast. Security and speed/usability have tradeoffs. The paranoid will do everything in their knowledge to protect themselves but they will move slowly. The naive will not do anything and sooner than later will get burnt. Assuming you trust your current modules and you can trust that they will not be altered, you can lock down the dependencies. But now monitor and updating a library when a bug or vulnerability is found in a modules become another burden. Those actors that have the budget, patience and the motivation will always find a way in. I sometimes feel quite sad about the real situation but there is not such a thing as perfect security. That said some of the suggestions I see here will dramatically improve the situation with a relatively small effort. 2FA and reproducible builds for example.

intertextuality 8y ago

2fa should always be on. 1) Signing should always happen. Publish time + hashes of external resources should be taken into account to prevent meddling with things after the fact. 2- a much harder solution) Don't use so many libraries. In this way I think it's borderline impossible. Of course this vuln targeted `bootstrap-sass`.. it's basically a default gem for new rails installations since bootstrap is so commonly used. Actually, rails 6.0 not only installs dependencies for you upon generating a new app, but it downloads yarn and installs 602 dependencies too. It's a miracle we do anything as programmers.

chaz6 8y ago

I really like the idea of a cryptographically provable signature for a set of coordinates in space and time, but not sure how such a thing is possible.