8 ms·

One of the things that the recent Python cryptography debate has highlighted to me is how much we depend on this distribution of libraries like this. In that ca

by macksd 6y ago

One of the things that the recent Python cryptography debate has highlighted to me is how much we depend on this distribution of libraries like this. In that case, it's normal to just automatically get updates from a product and then one day, a whole bunch of software projects suddenly notice and, luckily, break (lucky, as opposed to being compromised). In this case, it's normal to just type a project name in install software, with very little vetting done by many people. I want to work on someone's web app, npm suddenly downloads the world. Who's actually audited all that? I know I haven't. Not sure how we could fix it without slowing way down and doing a lot more work.

ryanmarsh 6y ago
Who's actually audited all that? I'm sure it's insufficient but you asked who audits all that (re: npm) well, apparently somebody. https://docs.npmjs.com/cli/v7/commands/npm-audit https://docs.npmjs.com/cli/v7/commands/npm-audit The audit command submits a description of the dependencies configured in your project to your default registry and asks for a report of known vulnerabilities. If any vulnerabilities are found, then the impact and appropriate remediation will be calculated. If the fix argument is provided, then remediations will be applied to the package tree.
- jhardy54 6y ago
  This provides a list of well-known vulnerabilities, it doesn't actually audit the source code to find unknown vulnerabilities.
tux1968 6y ago
The only practical way is to have a centralized vetting organization that can give a trusted stamp of approval to signed packages. That would still slow things down, but at least wouldn't require duplication of work by everyone on a massive scale. You could outsource your vetting work to a distribution you trust.
- artful-hacker 6y ago
  I was trying to find a solution to this problem for my organization and came across https://tidelift.com/ https://tidelift.com/. I like some of their ideas and think it's a solid path forward at chipping away on this issue. https://libraries.io/ https://libraries.io/ is a project of theirs I use quite often when vetting third party dependencies for our organization.
- JulianMorrison 6y ago
  The issue that runs into is a mirror of the one raised by the pyca/cryptography business. For which CPU and hardware architecture should they be verified? On which OS? With which set of dependencies, including dependencies on non-Python compiled libraries? Presumably all of those will need to be verified as well, and the problem recurses.
  - freeone3000 6y ago
    Every different CPU architecture and OS architecture is a different package. Software distributions already know this. The individual packages need to be verified.
    - vel0city 6y ago
      At least with Python, this is true only if you're installing pre-compiled wheels which is quickly become more and more the norm. For a while though several packages were compile on install so you wouldn't necessarily have a different package per architecture/OS combination.
      - freeone3000 6y ago
        You don't have a different package name, but you for sure have a different package. Take, for instance, the keyword "long". 64 bit on amd64, 32-bit on armhf. So even source distributions to be compiled locally differ based on target, even if they have the same coordinates in pypy.
ChrisMarshallNY 6y ago
> Not sure how we could fix it without slowing way down and doing a lot more work. Unfortunately, this is pretty much the only way, besides the "walled garden" approach. I am often derided as being a "grumpy old control freak." They def have a point, but I am very, very leery of bringing code into my projects that I don't understand, and stepping through source takes almost as long as writing from scratch. Most dependencies cover a wider range of functionality than the narrow application that we seek in our individual projects, so, to truly understand the dependency, we need to examine all of its faces. I recently had to yank a dependency out of a project that I'm developing, because it had a regex bug that caused memory leaks and random crashes in worker threads. It would have been way too difficult for me to figure out what was happening, so I removed it, and wrote my own functionality that is not as good, but doesn't crash my application. This is the kind of behavior that gets me scorn. I'm supposed to keep the crash, and shrug. So, yeah. I guess I am a grumpy old control freak...
- da39a3ee 6y ago
  I don't think that understanding your dependencies is a reasonable criterion (the intellectual scope of your dependencies should be beyond your intellect for all but very simple projects). Furthermore I think that your approach essentially rejects the good engineering/design principles of modularity/separation of concerns (it is _good_ to black box things).
  - xapata 6y ago
    Depends on how much understanding you're talking about. I don't need to know how to build an x86 chip, but as an experienced and educated professional programmer I should probably have a decent sense of which kinds of instructions are time consuming and which aren't, how the cache works, etc. Abstractions are leaky. You should strive to understand where your black box is transparent.
  - mikem170 6y ago
    >I don't think that understanding your dependencies is a reasonable criterion I agree, I've never gone as far as reviewing/understanding source code for all the libraries I use. Might be nice to have the time ;-) >it is _good_ to black box things I trust that you are talking about using other people's black boxes, as opposed to reinventing your own? I would say that there is some element of risk in trusting a library that someone else made, particularly over time as systems need to be upgraded. Will there be problems during or after an upgrade? Is the whole dependency chain upgradable? Will the upgrades work on all operating systems in use? Are there relevant corporate policies and/or whitelists that might interfere with or slow down upgrades? Personally I pay close attention to the dependencies I'm using in a project, especially long-lived projects. I find it super aggravating when something I've tested and deployed breaks a couple years later after routine upgrades due to some kind of drift problem to do with a dependency. How much benefit am I getting by using the library? Best is using base library that comes with the language. Next best would be a well maintained third-party library with a good reputation in the community, one that doesn't have too many dependencies itself. Am I happy with the api of the library, and its documentation? Is is reasonably easy to approach given its functionality? Is the library available as a package in the operating systems I'm working with? How long has it existed? How stable has it been over time? Is the library a good fit for what I am doing, or do I have to bend my code to use it as intended? Do I have multiple libraries to choose from? Can I use the library in such a way that I can change my mind later? I think there is value in trying to minimize the use of external libraries, up to a point.
JMTQp8lwXL 6y ago
The business case for accepting the risk of downloading untrusted software has proven time and time again to be worth it. Without NPM modules, time to deliver would be much, much longer. If it was that bad of a problem, we would've collectively moved away. Many companies privately invest in their own registries that proxy to NPM, with an intermediary security layer. The maintainers of NPM have shown they'll cleanup the left-pad's and major security issues when they occur.
- goatinaboat 6y ago
  Without NPM modules, time to deliver would be much, much longer. This is a circular argument. “We chose a language that has millions of tiny dependencies now we need a tool to manage millions of tiny dependencies”
  - JMTQp8lwXL 6y ago
    If you must ship JavaScript for the web, you don't have much of a choice in languages (you might choose a superset like TypeScript, which is still distributed via the NPM ecosystem). For server-side code, agreed. You don't have to choose Node.js or JavaScript or NPM there.
hanniabu 6y ago
> Not sure how we could fix it without slowing way down and doing a lot more work. Can create a market for library audits that the library's team would pay for and provide "vetted" tags for libraries that have audits on minor updates. Things can continue as they are for smaller libraries, but any library that's used in more critical areas should have no issue finding audit sponsors. This would also create in incentive for somebody to develop automated auditing tools.
- BiteCode_dev 6y ago
  That's what anaconda is.
ryukafalz 6y ago
> Not sure how we could fix it without slowing way down and doing a lot more work. You restrict what your dependencies can do in the first place, so that if they're malicious (or just buggy) the scope of what they can do is limited. This doesn't eliminate the risk entirely - after all, it's possible for a library to introduce vulnerabilities just by doing its job incorrectly - but it massively limits the scope of what you'd need to audit. Right now, any dependency can do anything, so you'd need to audit all of them. See POLA Would Have Prevented the Event-Stream Incident[0] for more explanation and LavaMoat[1] for an example of tooling that's trying to tackle this problem. Now, this might count as "doing a lot more work" since it's admittedly not quite as simple as just typing in a project name. It's much less work than rewriting everything from scratch though. :) [0] https://medium.com/agoric/pola-would-have-prevented-the-event-stream-incident-45653ecbda99 https://medium.com/agoric/pola-would-have-prevented-the-even... [1] https://github.com/LavaMoat/lavamoat https://github.com/LavaMoat/lavamoat
matheusmoreira 6y ago
> Who's actually audited all that? That's the point of software distribution maintainers. They are actual humans who must essentially sponsor a project before it becomes a package in a repository. They take responsibility for the packages they maintain. Users trust these humans since they usually don't make mistakes like letting literal malware into the software repositories. Most modern languages are deliberately designed to be incompatible with this model. They optimize for developer ease of use and developers don't like getting approval from other people in order to publish their code. So they make their own little isolated worlds where everyone can upload anything they want. Honestly it's amazing it took this long for people's trust to be exploited.
- alisonkisk 6y ago
  Languages aren't incompatible at all. People choose to use excessively liberal package-mangement systems.
alfiedotwtf 6y ago
This is why I absolutely hate not pinning down versions/hashes. For example, with `cargo`, let’s say a library I require passes a security audit. However, because a library it requires, or one of its dependencies requires doesn’t pin down a specific version but instead just requires “latest” or “2.1.”, my security audit is for naught given that malware can slip in any time for 2.1.. That goes for testing too. I’ve tested my software etc, but sometime down the line a transitive dependency updates and adds a bug. Now my software that i haven’t changed, is broken. Software requirements without pinning is a code smell.
- gpm 6y ago
  With Cargo, the final binary (or artifact if you're building a .so/.a/.dylib/...) has a `Cargo.lock` file that pins every recursive and non-recursive dependency. If you audit the packages in that, you've audited every dependency that is used in your project, and they will not change unless you yourself explicitly tell them to.
  - alfiedotwtf 6y ago
    Oh! TIL, thanks!
- remram 6y ago
  If your direct dependencies pin their dependencies, then you will definitely end with conflicting versions for your transitive dependencies. This might work in environments like Node, where each library having their own private versions of their dependencies is acceptable, and Rust/Cargo in some situations, but doesn't work in environments where only one version of a package can be present (e.g. Python) or you care about the total size of the dependencies. Also if you're auditing, wouldn't you audit all dependencies? What does it matter whether they are pinned by your direct dependencies, wouldn't you yourself pin down the transitive dependencies after auditing them?
  - alfiedotwtf 6y ago
    > Also if you're auditing, wouldn't you audit all dependencies? What’s the point of a transitive dependency can pull the rug out from under your audio with a single update
ohiovr 6y ago
Would anyone pay for a curated service?
jjav 6y ago
Well you can't, slowing down is a must. Blindly taking packages of executable code to be embedded in your code from anyone and everyone in the world who can publish a dependency is very flawed. It only mostly works because most people are nice most of the time but that's not a process one can trust. I understand the desire for these free-for-all package distributions where every tool has it's own library downloader and anyone can publish anything... but the model is inherently incompatible with security. On the provider side, there needs to be a strict process of validation and code reviews to get any code changes into a public library. On the consumer side, there needs to be the will to take personal responsibility to curate everything that is imported and minimize dependencies. There's no shortcut to secure code.
TheCabin 6y ago
Many people here say that slowing down is a must -- and I agree it's probably the best solution -- but surely there are more approaches we could think of: * Not allowing packages with similar names to popular ones * Not allowing packages creation to be anonymous (in the extreme case you would require to validate your passport or similar) * Automatic detection of malicious code * Central auditing organization ... This is just on top of my head, there must be many more ideas.

by macksd 6y ago

ryanmarsh 6y ago

Who's actually audited all that? I'm sure it's insufficient but you asked who audits all that (re: npm) well, apparently somebody. https://docs.npmjs.com/cli/v7/commands/npm-audit https://docs.npmjs.com/cli/v7/commands/npm-audit The audit command submits a description of the dependencies configured in your project to your default registry and asks for a report of known vulnerabilities. If any vulnerabilities are found, then the impact and appropriate remediation will be calculated. If the fix argument is provided, then remediations will be applied to the package tree.

jhardy54 6y ago
This provides a list of well-known vulnerabilities, it doesn't actually audit the source code to find unknown vulnerabilities.

tux1968 6y ago

The only practical way is to have a centralized vetting organization that can give a trusted stamp of approval to signed packages. That would still slow things down, but at least wouldn't require duplication of work by everyone on a massive scale. You could outsource your vetting work to a distribution you trust.

artful-hacker 6y ago
I was trying to find a solution to this problem for my organization and came across https://tidelift.com/ https://tidelift.com/. I like some of their ideas and think it's a solid path forward at chipping away on this issue. https://libraries.io/ https://libraries.io/ is a project of theirs I use quite often when vetting third party dependencies for our organization.
JulianMorrison 6y ago
The issue that runs into is a mirror of the one raised by the pyca/cryptography business. For which CPU and hardware architecture should they be verified? On which OS? With which set of dependencies, including dependencies on non-Python compiled libraries? Presumably all of those will need to be verified as well, and the problem recurses.
- freeone3000 6y ago
  Every different CPU architecture and OS architecture is a different package. Software distributions already know this. The individual packages need to be verified.
  - vel0city 6y ago
    At least with Python, this is true only if you're installing pre-compiled wheels which is quickly become more and more the norm. For a while though several packages were compile on install so you wouldn't necessarily have a different package per architecture/OS combination.
    - freeone3000 6y ago
      You don't have a different package name, but you for sure have a different package. Take, for instance, the keyword "long". 64 bit on amd64, 32-bit on armhf. So even source distributions to be compiled locally differ based on target, even if they have the same coordinates in pypy.

ChrisMarshallNY 6y ago

> Not sure how we could fix it without slowing way down and doing a lot more work. Unfortunately, this is pretty much the only way, besides the "walled garden" approach. I am often derided as being a "grumpy old control freak." They def have a point, but I am very, very leery of bringing code into my projects that I don't understand, and stepping through source takes almost as long as writing from scratch. Most dependencies cover a wider range of functionality than the narrow application that we seek in our individual projects, so, to truly understand the dependency, we need to examine all of its faces. I recently had to yank a dependency out of a project that I'm developing, because it had a regex bug that caused memory leaks and random crashes in worker threads. It would have been way too difficult for me to figure out what was happening, so I removed it, and wrote my own functionality that is not as good, but doesn't crash my application. This is the kind of behavior that gets me scorn. I'm supposed to keep the crash, and shrug. So, yeah. I guess I am a grumpy old control freak...

da39a3ee 6y ago
I don't think that understanding your dependencies is a reasonable criterion (the intellectual scope of your dependencies should be beyond your intellect for all but very simple projects). Furthermore I think that your approach essentially rejects the good engineering/design principles of modularity/separation of concerns (it is _good_ to black box things).
- xapata 6y ago
  Depends on how much understanding you're talking about. I don't need to know how to build an x86 chip, but as an experienced and educated professional programmer I should probably have a decent sense of which kinds of instructions are time consuming and which aren't, how the cache works, etc. Abstractions are leaky. You should strive to understand where your black box is transparent.
- mikem170 6y ago
  >I don't think that understanding your dependencies is a reasonable criterion I agree, I've never gone as far as reviewing/understanding source code for all the libraries I use. Might be nice to have the time ;-) >it is _good_ to black box things I trust that you are talking about using other people's black boxes, as opposed to reinventing your own? I would say that there is some element of risk in trusting a library that someone else made, particularly over time as systems need to be upgraded. Will there be problems during or after an upgrade? Is the whole dependency chain upgradable? Will the upgrades work on all operating systems in use? Are there relevant corporate policies and/or whitelists that might interfere with or slow down upgrades? Personally I pay close attention to the dependencies I'm using in a project, especially long-lived projects. I find it super aggravating when something I've tested and deployed breaks a couple years later after routine upgrades due to some kind of drift problem to do with a dependency. How much benefit am I getting by using the library? Best is using base library that comes with the language. Next best would be a well maintained third-party library with a good reputation in the community, one that doesn't have too many dependencies itself. Am I happy with the api of the library, and its documentation? Is is reasonably easy to approach given its functionality? Is the library available as a package in the operating systems I'm working with? How long has it existed? How stable has it been over time? Is the library a good fit for what I am doing, or do I have to bend my code to use it as intended? Do I have multiple libraries to choose from? Can I use the library in such a way that I can change my mind later? I think there is value in trying to minimize the use of external libraries, up to a point.

JMTQp8lwXL 6y ago

The business case for accepting the risk of downloading untrusted software has proven time and time again to be worth it. Without NPM modules, time to deliver would be much, much longer. If it was that bad of a problem, we would've collectively moved away. Many companies privately invest in their own registries that proxy to NPM, with an intermediary security layer. The maintainers of NPM have shown they'll cleanup the left-pad's and major security issues when they occur.

goatinaboat 6y ago
Without NPM modules, time to deliver would be much, much longer. This is a circular argument. “We chose a language that has millions of tiny dependencies now we need a tool to manage millions of tiny dependencies”
- JMTQp8lwXL 6y ago
  If you must ship JavaScript for the web, you don't have much of a choice in languages (you might choose a superset like TypeScript, which is still distributed via the NPM ecosystem). For server-side code, agreed. You don't have to choose Node.js or JavaScript or NPM there.

hanniabu 6y ago

> Not sure how we could fix it without slowing way down and doing a lot more work. Can create a market for library audits that the library's team would pay for and provide "vetted" tags for libraries that have audits on minor updates. Things can continue as they are for smaller libraries, but any library that's used in more critical areas should have no issue finding audit sponsors. This would also create in incentive for somebody to develop automated auditing tools.

BiteCode_dev 6y ago
That's what anaconda is.

ryukafalz 6y ago

> Not sure how we could fix it without slowing way down and doing a lot more work. You restrict what your dependencies can do in the first place, so that if they're malicious (or just buggy) the scope of what they can do is limited. This doesn't eliminate the risk entirely - after all, it's possible for a library to introduce vulnerabilities just by doing its job incorrectly - but it massively limits the scope of what you'd need to audit. Right now, any dependency can do anything, so you'd need to audit all of them. See POLA Would Have Prevented the Event-Stream Incident[0] for more explanation and LavaMoat[1] for an example of tooling that's trying to tackle this problem. Now, this might count as "doing a lot more work" since it's admittedly not quite as simple as just typing in a project name. It's much less work than rewriting everything from scratch though. :) [0] https://medium.com/agoric/pola-would-have-prevented-the-event-stream-incident-45653ecbda99 https://medium.com/agoric/pola-would-have-prevented-the-even... [1] https://github.com/LavaMoat/lavamoat https://github.com/LavaMoat/lavamoat

matheusmoreira 6y ago

> Who's actually audited all that? That's the point of software distribution maintainers. They are actual humans who must essentially sponsor a project before it becomes a package in a repository. They take responsibility for the packages they maintain. Users trust these humans since they usually don't make mistakes like letting literal malware into the software repositories. Most modern languages are deliberately designed to be incompatible with this model. They optimize for developer ease of use and developers don't like getting approval from other people in order to publish their code. So they make their own little isolated worlds where everyone can upload anything they want. Honestly it's amazing it took this long for people's trust to be exploited.

alisonkisk 6y ago
Languages aren't incompatible at all. People choose to use excessively liberal package-mangement systems.

alfiedotwtf 6y ago

This is why I absolutely hate not pinning down versions/hashes. For example, with `cargo`, let’s say a library I require passes a security audit. However, because a library it requires, or one of its dependencies requires doesn’t pin down a specific version but instead just requires “latest” or “2.1.”, my security audit is for naught given that malware can slip in any time for 2.1.. That goes for testing too. I’ve tested my software etc, but sometime down the line a transitive dependency updates and adds a bug. Now my software that i haven’t changed, is broken. Software requirements without pinning is a code smell.

gpm 6y ago
With Cargo, the final binary (or artifact if you're building a .so/.a/.dylib/...) has a `Cargo.lock` file that pins every recursive and non-recursive dependency. If you audit the packages in that, you've audited every dependency that is used in your project, and they will not change unless you yourself explicitly tell them to.
- alfiedotwtf 6y ago
  Oh! TIL, thanks!
remram 6y ago
If your direct dependencies pin their dependencies, then you will definitely end with conflicting versions for your transitive dependencies. This might work in environments like Node, where each library having their own private versions of their dependencies is acceptable, and Rust/Cargo in some situations, but doesn't work in environments where only one version of a package can be present (e.g. Python) or you care about the total size of the dependencies. Also if you're auditing, wouldn't you audit all dependencies? What does it matter whether they are pinned by your direct dependencies, wouldn't you yourself pin down the transitive dependencies after auditing them?
- alfiedotwtf 6y ago
  > Also if you're auditing, wouldn't you audit all dependencies? What’s the point of a transitive dependency can pull the rug out from under your audio with a single update

ohiovr 6y ago

Would anyone pay for a curated service?

jjav 6y ago

Well you can't, slowing down is a must. Blindly taking packages of executable code to be embedded in your code from anyone and everyone in the world who can publish a dependency is very flawed. It only mostly works because most people are nice most of the time but that's not a process one can trust. I understand the desire for these free-for-all package distributions where every tool has it's own library downloader and anyone can publish anything... but the model is inherently incompatible with security. On the provider side, there needs to be a strict process of validation and code reviews to get any code changes into a public library. On the consumer side, there needs to be the will to take personal responsibility to curate everything that is imported and minimize dependencies. There's no shortcut to secure code.

TheCabin 6y ago

Many people here say that slowing down is a must -- and I agree it's probably the best solution -- but surely there are more approaches we could think of: * Not allowing packages with similar names to popular ones * Not allowing packages creation to be anonymous (in the extreme case you would require to validate your passport or similar) * Automatic detection of malicious code * Central auditing organization ... This is just on top of my head, there must be many more ideas.