5 ms·

Another solution would be changing the ecosystem to no longer be reliant on so many third party dependencies. For instance if I am using Java and I build my we

Another solution would be changing the ecosystem to no longer be reliant on so many third party dependencies. For instance if I am using Java and I build my web app with only Spring Framework, I can have a lot more confidence that one of my JARs hasn’t been backdoored than I can in an ecosystem where it’s regularly the practice to pull 100s of dependencies from different individual FOSS developers, where it’s difficult to audit the process that each library author is using to secure their package manager upload credentials. I am not sure signatures are that useful since without a centralized authority to issue the certificates and securely verify author identities, we are just back to a trust-on-first-use policy for the signatures, and people will just end up setting their CI servers to always trust new signatures since they won’t want to deal with what happens when authors change their certificate from version to version (which will surely happen).

msbarnett 7y ago
Sure, obviously reinventing as many wheels as possible will minimize your exposure to third-party malfeasance. As in all forms of engineering there are, of course, no absolutes, only trade-offs to be made. The more wheels you reinvent, the slower your velocity for solving the core business problems that pay your bills. Moving too slowly can be fatal to the business. It’s a tricky balance. Signing isn’t perfect, but it can improve some aspects of some balances people strike.
- bdcravens 7y ago
  Reinventing the wheel is one end of the spectrum, but so is putting a bunch of magic words in a file and thinking you needn't worry. (To be fair, too many tutorials hand wave away what really happens in Gemfile/package.json/etc and barely pay lip service, if at all, to the responsibilities that come with having those dependencies) I don't miss the "old" days of searching for libs, extracting a zip, and trying to figure out the integration steps, but there is something to be said for there being a bit more of a sense of ownership.
  - msbarnett 7y ago
    > Reinventing the wheel is one end of the spectrum, but so is putting a bunch of magic words in a file and thinking you needn't worry. Yeah. I didn’t claim otherwise? Like I said, there are only trade-offs here. You can gain a ton of velocity if you abstract it all out to a file of 50,000 “magic words”; but obviously then your exposure to these issues is enormous. Trade-offs are like that. > I don't miss the "old" days of searching for libs, extracting a zip, and trying to figure out the integration steps, but there is something to be said for there being a bit more of a sense of ownership. Eh, to the extent that I get any such feeling, I think it’s mostly a completely false sense of security. I did some stuff in a C++ codebase that was fully developed that way, and did the ol’ “hunt, unzip, and compile” for some boost libs. I didn’t audit the source. It being boost C++ god knows if I’d even have been able to recognize a heavily template-metaprogrammed exploit. If boost’s account had been compromised I’d have been every bit as fucked as people who use this gem were. Supply chain attacks are dangerous regardless. All you can do is try to balance your exposure vs your time spent on solving problems that don’t pay the bills vs those that do.
    - bdcravens 7y ago
      > I think it’s mostly a completely false sense of security I wasn't so much thinking about avoiding vulns due to added scrutiny as much as issues with updated libs, since most build processes pull gems, etc when deploying or rebuilding a container; I don't think many vendor the gems. Typically you'd ship the actual files you unzipped as opposed to letting the package manager grab the most recent version within version spec.
      - msbarnett 7y ago
        Typically your deployment process would pull gems based on hashes recorded in a lock file you committed, not pull arbitrary new versions automagically. So while I’m not shipping the actual file, something is verifying that I’m shipping files that have the hash I expect. Barring some very alarming developments in hash insecurity, it’s mostly a distinction without a difference. Automating a “bundle update” to pull latest versions within spec and update the lockfile would be odd to my experience. You’d typically do that manually, (hopefully, if you’re competent) look at what changed, and retest (semver is great as long as everyone perfectly anticipates & categorizes every change’s impact. In the real world, however, ....) rather than blindly just letting a deployment run whatever. Bad devs can do the stupidest thing imaginable in any system, though, so I don’t doubt this is out there.
- jmiserez 7y ago
  I interpreted that differently: it's fine to have many dependencies, even third-party ones. What's a problem is having many third-parties. The maintenance and trust verification overhead for a micro "5 lines of code" dependency is usually way higher than just rewriting those 5 lines yourself. As an aside: just because someone made their code available doesn't mean it's good or that it solves all your edge cases. Getting those fixed also takes up time.
  - msbarnett 7y ago
    > The maintenance and trust verification overhead for a micro "5 lines of code" dependency is usually way higher than just rewriting those 5 lines yourself. Sure. I was in no way advocating “JS-style, 5 line microlibs Uber Alles”, merely pointing out that there’s a clear trade-off between dependencies and velocity that there’s no silver bullet for. There’s absolutely nothing wrong with OP saying “we can afford to make Spring our only dependency”, but there’s also nothing wrong with saying “we need actor-based concurrency and the business will be dead before we roll are own, let’s bring in Akka”, “we need to deal with time and spending 40 hours a month keeping up with every legislature on the planet’s timezone-related lawmaking doesn’t pay the bills, bring in JodaTime”, etc. It’s engineering; these trade-offs should of course be carefully considered (is_even should fail most any sane consideration), but it’s a bit silly to suggest they can just be avoided entirely by businesses that have to make money to pay the bills. Again, signing helps when you need to make those trade-offs. There are no absolutes here.
    - javagram 7y ago
      I think those are great examples of what I was talking about actually in the Java ecosystem. JodaTime is now deprecated in favor of Java 8 time (JSR-310). Akka is an official library from Typesafe/Lightbend. In a JVM ecosystem, you can get this kind of stuff directly from a supported corporate vendor. You can even easily pay them for support if you want. And a lot of times stuff even gets standardized through the JSR process. Now, if you’re in a pre-Java 8 world and you need JodaTime, sure it makes sense to bring it in, not just use only Spring. But eventually that software library gets recognized as necessary to the ecosystem and standardized, and you no longer have to rely on yet another 3rd party for it. Whereas in another ecosystem, JodaTime might just keep existing, maybe even alongside the bad default language library, and everyone has to always be told to go get this third party dependency if you want to do things right.
- PeterisP 7y ago
  It's not about the amount of functionality shifted to dependencies, but about the fragmentation of these dependencies. Packaging and distribution of libraries takes effort to do it properly, so they're only done properly if it's sufficiently centralized. If you have to import fifty third-party wheels, then it's unavoidable that some or most of these wheels can't be managed properly, but it's quite feasible to have a single (or three) well-managed third-party package that provides a hundred wheels so that you don't have to reinvent them. If the strong_password gem was integrated in (for example) Rails and managed/released by the same team with the same processes, then this risk would be avoided. If instead of a dozen separate gems with every functionality separate you'd have a single bundle of varied functionality like in Java there's Guava or Appache Commons, then that bundle can handle release management in a way that each separate gem developer can not. If you want to have reliable dependencies then you eithery have to choose only dependencies with buraucratic and pedantic release governance, or manage/audit each dependency yourself (as the author of the original article seems to have done). In ecosystems where it's reasonable to have serious projects that have 0-3 distinct (but large) external dependencies this works easily; in ecosystems where you have dozens or even a hundred dependencies, that overhead is impractical for most projects.
  - zbentley 7y ago
    > If you want to have reliable dependencies then you eithery have to choose only dependencies with buraucratic and pedantic release governance, or manage/audit each dependency yourself That's a false dichotomy. There are middle grounds which can and do work at scale: Upgrade knowingly and deliberately (don't just spray greenkeeper everywhere). Carefully monitor changed application/network behavior after upgrades. Devote a manageable, non-zero amount of time to reading/finding security bulletins or security incidents on your most-heavily-used dependencies. Pay attention to issue reports and prioritize any with possible security implications. ...and (at a slightly larger scale) hire, empower, and compensate people to do those kinds of things in a systematic, regular way. Seriously, security engineering isn't served well by "ZOMG NPM is garbage we must switch to $megaframework and pray that their release engineers get everything right" hysteria and absolutism. There are effective, moderate strategies that help with these issues every day.
- zbentley 7y ago
  > The more wheels you reinvent, the slower your velocity for solving the core business problems that pay your bills. And the more mistakes you make, pissing off your users (or worse, compromising their data because you thought you could roll-your-own of some security critical dependency you ditched).
- javagram 7y ago
  What I’d say though is that in an ecosystem like Java, it’s not necessarily “reinventing the wheel”. It’s more that you can import Spring, Guava, and Apache Commons and you have a collection of 1000s of wheels of different shapes and sizes ready to go when you need them. Whereas in some other ecosystems, you have to go get each wheel individually from a different person. There are certainly reasons why each ecosystem evolved the way it did, but I don’t think it’s impossible that this sort of stuff could centralize more in the future, especially as it becomes more clear what should be “batteries included” or what things are actually needed and used by the community.
  - msbarnett 7y ago
    > What I’d say though is that in an ecosystem like Java, it’s not necessarily “reinventing the wheel”. It’s more that you can import Spring, Guava, and Apache Commons and you have a collection of 1000s of wheels of different shapes and sizes ready to go when you need them. Sure, but even here we see you modulating your position from “only Spring” to “Spring, Guava, and Apache Commons”, tripling the number of dependencies you’re willing to admit. Really what it boils down to is, you’re saying what I’m saying. Declaring absolutely “this dependency, and no others” is silly — rather, it’s a question of trade-offs, and you feel that in your usecases trading off say, 3 or 4 large dependencies is worth the velocity gain. Nobody is arguing with that. Fewer, larger, carefully considered dependencies is a relational set of trade-offs to make.
matthewmacleod 7y ago
I doubt that any solutions which start off as “let’s change the whole universe” are going to get very far. Yes, the situation sucks. I just looked at the frontend of a relatively small app we use for administration, and it depends on almost 5000 node modules versions. But this problem needs to be dealt with as soon as possible, and I don’t think that a fundamental change in development culture—making everything harder for developers in the process—is going to help.
- nneonneo 7y ago
  5000? Wow. That's gotta be a pretty sizable fraction of the entire Node library ecosystem. Does it count duplicates? With a dependency forest that large, it's small wonder that not more JS projects aren't compromised by bad dependencies...
  - acdha 7y ago
    Create React App is pretty popular and it has 36k dependencies. That’s not saying everyone using it ships every one of those but that’s definitely a LOT of people who could potentially introduce malware, often with a fair chance of it being deniable.
your-nanny 7y ago
I know there has been some recent discussion within the python community that Python's batteries included philosophy is failing the community, with some arguing that the python standard library should be streamlined. But I see third party dependencies as often a liability both for security and a kind of technical debt (I maintain infrequently used but essential legacy python applications)

msbarnett 7y ago

Sure, obviously reinventing as many wheels as possible will minimize your exposure to third-party malfeasance. As in all forms of engineering there are, of course, no absolutes, only trade-offs to be made. The more wheels you reinvent, the slower your velocity for solving the core business problems that pay your bills. Moving too slowly can be fatal to the business. It’s a tricky balance. Signing isn’t perfect, but it can improve some aspects of some balances people strike.

bdcravens 7y ago
Reinventing the wheel is one end of the spectrum, but so is putting a bunch of magic words in a file and thinking you needn't worry. (To be fair, too many tutorials hand wave away what really happens in Gemfile/package.json/etc and barely pay lip service, if at all, to the responsibilities that come with having those dependencies) I don't miss the "old" days of searching for libs, extracting a zip, and trying to figure out the integration steps, but there is something to be said for there being a bit more of a sense of ownership.
- msbarnett 7y ago
  > Reinventing the wheel is one end of the spectrum, but so is putting a bunch of magic words in a file and thinking you needn't worry. Yeah. I didn’t claim otherwise? Like I said, there are only trade-offs here. You can gain a ton of velocity if you abstract it all out to a file of 50,000 “magic words”; but obviously then your exposure to these issues is enormous. Trade-offs are like that. > I don't miss the "old" days of searching for libs, extracting a zip, and trying to figure out the integration steps, but there is something to be said for there being a bit more of a sense of ownership. Eh, to the extent that I get any such feeling, I think it’s mostly a completely false sense of security. I did some stuff in a C++ codebase that was fully developed that way, and did the ol’ “hunt, unzip, and compile” for some boost libs. I didn’t audit the source. It being boost C++ god knows if I’d even have been able to recognize a heavily template-metaprogrammed exploit. If boost’s account had been compromised I’d have been every bit as fucked as people who use this gem were. Supply chain attacks are dangerous regardless. All you can do is try to balance your exposure vs your time spent on solving problems that don’t pay the bills vs those that do.
  - bdcravens 7y ago
    > I think it’s mostly a completely false sense of security I wasn't so much thinking about avoiding vulns due to added scrutiny as much as issues with updated libs, since most build processes pull gems, etc when deploying or rebuilding a container; I don't think many vendor the gems. Typically you'd ship the actual files you unzipped as opposed to letting the package manager grab the most recent version within version spec.
    - msbarnett 7y ago
      Typically your deployment process would pull gems based on hashes recorded in a lock file you committed, not pull arbitrary new versions automagically. So while I’m not shipping the actual file, something is verifying that I’m shipping files that have the hash I expect. Barring some very alarming developments in hash insecurity, it’s mostly a distinction without a difference. Automating a “bundle update” to pull latest versions within spec and update the lockfile would be odd to my experience. You’d typically do that manually, (hopefully, if you’re competent) look at what changed, and retest (semver is great as long as everyone perfectly anticipates & categorizes every change’s impact. In the real world, however, ....) rather than blindly just letting a deployment run whatever. Bad devs can do the stupidest thing imaginable in any system, though, so I don’t doubt this is out there.
jmiserez 7y ago
I interpreted that differently: it's fine to have many dependencies, even third-party ones. What's a problem is having many third-parties. The maintenance and trust verification overhead for a micro "5 lines of code" dependency is usually way higher than just rewriting those 5 lines yourself. As an aside: just because someone made their code available doesn't mean it's good or that it solves all your edge cases. Getting those fixed also takes up time.
- msbarnett 7y ago
  > The maintenance and trust verification overhead for a micro "5 lines of code" dependency is usually way higher than just rewriting those 5 lines yourself. Sure. I was in no way advocating “JS-style, 5 line microlibs Uber Alles”, merely pointing out that there’s a clear trade-off between dependencies and velocity that there’s no silver bullet for. There’s absolutely nothing wrong with OP saying “we can afford to make Spring our only dependency”, but there’s also nothing wrong with saying “we need actor-based concurrency and the business will be dead before we roll are own, let’s bring in Akka”, “we need to deal with time and spending 40 hours a month keeping up with every legislature on the planet’s timezone-related lawmaking doesn’t pay the bills, bring in JodaTime”, etc. It’s engineering; these trade-offs should of course be carefully considered (is_even should fail most any sane consideration), but it’s a bit silly to suggest they can just be avoided entirely by businesses that have to make money to pay the bills. Again, signing helps when you need to make those trade-offs. There are no absolutes here.
  - javagram 7y ago
    I think those are great examples of what I was talking about actually in the Java ecosystem. JodaTime is now deprecated in favor of Java 8 time (JSR-310). Akka is an official library from Typesafe/Lightbend. In a JVM ecosystem, you can get this kind of stuff directly from a supported corporate vendor. You can even easily pay them for support if you want. And a lot of times stuff even gets standardized through the JSR process. Now, if you’re in a pre-Java 8 world and you need JodaTime, sure it makes sense to bring it in, not just use only Spring. But eventually that software library gets recognized as necessary to the ecosystem and standardized, and you no longer have to rely on yet another 3rd party for it. Whereas in another ecosystem, JodaTime might just keep existing, maybe even alongside the bad default language library, and everyone has to always be told to go get this third party dependency if you want to do things right.
PeterisP 7y ago
It's not about the amount of functionality shifted to dependencies, but about the fragmentation of these dependencies. Packaging and distribution of libraries takes effort to do it properly, so they're only done properly if it's sufficiently centralized. If you have to import fifty third-party wheels, then it's unavoidable that some or most of these wheels can't be managed properly, but it's quite feasible to have a single (or three) well-managed third-party package that provides a hundred wheels so that you don't have to reinvent them. If the strong_password gem was integrated in (for example) Rails and managed/released by the same team with the same processes, then this risk would be avoided. If instead of a dozen separate gems with every functionality separate you'd have a single bundle of varied functionality like in Java there's Guava or Appache Commons, then that bundle can handle release management in a way that each separate gem developer can not. If you want to have reliable dependencies then you eithery have to choose only dependencies with buraucratic and pedantic release governance, or manage/audit each dependency yourself (as the author of the original article seems to have done). In ecosystems where it's reasonable to have serious projects that have 0-3 distinct (but large) external dependencies this works easily; in ecosystems where you have dozens or even a hundred dependencies, that overhead is impractical for most projects.
- zbentley 7y ago
  > If you want to have reliable dependencies then you eithery have to choose only dependencies with buraucratic and pedantic release governance, or manage/audit each dependency yourself That's a false dichotomy. There are middle grounds which can and do work at scale: Upgrade knowingly and deliberately (don't just spray greenkeeper everywhere). Carefully monitor changed application/network behavior after upgrades. Devote a manageable, non-zero amount of time to reading/finding security bulletins or security incidents on your most-heavily-used dependencies. Pay attention to issue reports and prioritize any with possible security implications. ...and (at a slightly larger scale) hire, empower, and compensate people to do those kinds of things in a systematic, regular way. Seriously, security engineering isn't served well by "ZOMG NPM is garbage we must switch to $megaframework and pray that their release engineers get everything right" hysteria and absolutism. There are effective, moderate strategies that help with these issues every day.
zbentley 7y ago
> The more wheels you reinvent, the slower your velocity for solving the core business problems that pay your bills. And the more mistakes you make, pissing off your users (or worse, compromising their data because you thought you could roll-your-own of some security critical dependency you ditched).
javagram 7y ago
What I’d say though is that in an ecosystem like Java, it’s not necessarily “reinventing the wheel”. It’s more that you can import Spring, Guava, and Apache Commons and you have a collection of 1000s of wheels of different shapes and sizes ready to go when you need them. Whereas in some other ecosystems, you have to go get each wheel individually from a different person. There are certainly reasons why each ecosystem evolved the way it did, but I don’t think it’s impossible that this sort of stuff could centralize more in the future, especially as it becomes more clear what should be “batteries included” or what things are actually needed and used by the community.
- msbarnett 7y ago
  > What I’d say though is that in an ecosystem like Java, it’s not necessarily “reinventing the wheel”. It’s more that you can import Spring, Guava, and Apache Commons and you have a collection of 1000s of wheels of different shapes and sizes ready to go when you need them. Sure, but even here we see you modulating your position from “only Spring” to “Spring, Guava, and Apache Commons”, tripling the number of dependencies you’re willing to admit. Really what it boils down to is, you’re saying what I’m saying. Declaring absolutely “this dependency, and no others” is silly — rather, it’s a question of trade-offs, and you feel that in your usecases trading off say, 3 or 4 large dependencies is worth the velocity gain. Nobody is arguing with that. Fewer, larger, carefully considered dependencies is a relational set of trade-offs to make.

matthewmacleod 7y ago

I doubt that any solutions which start off as “let’s change the whole universe” are going to get very far. Yes, the situation sucks. I just looked at the frontend of a relatively small app we use for administration, and it depends on almost 5000 node modules versions. But this problem needs to be dealt with as soon as possible, and I don’t think that a fundamental change in development culture—making everything harder for developers in the process—is going to help.

nneonneo 7y ago
5000? Wow. That's gotta be a pretty sizable fraction of the entire Node library ecosystem. Does it count duplicates? With a dependency forest that large, it's small wonder that not more JS projects aren't compromised by bad dependencies...
- acdha 7y ago
  Create React App is pretty popular and it has 36k dependencies. That’s not saying everyone using it ships every one of those but that’s definitely a LOT of people who could potentially introduce malware, often with a fair chance of it being deniable.

your-nanny 7y ago

I know there has been some recent discussion within the python community that Python's batteries included philosophy is failing the community, with some arguing that the python standard library should be streamlined. But I see third party dependencies as often a liability both for security and a kind of technical debt (I maintain infrequently used but essential legacy python applications)