5 ms·

Yes, in depth. Not just the packages but their dependencies as well.

gitgud 6y ago
Really? What about all the dependencies from those dependencies?... For example, our company is working on an app that has 82 npm dependencies and over 17,000 resolved npm packages... It's absolutely ridiculous to investigate all of them... but it's also necessary if you want to be sure...
- carapace 6y ago
  > Really? What about all the dependencies from those dependencies?... Yep, all the way to the end. I got the idea from a book called "Hollywood Secrets of Project Management Success" by James R. Persse. It's two books interleaved really, one is just a standard pitch for Agile methods (IMO), but the other is a presentation of the process that large film studios use to make movies. The movie industry is ~100 years old and mostly very good at bringing in projects on time and under budget. Somewhere in there he talks about how they'll track their dependencies in a kind of "portfolio", I forget the details, but it translates in IT to a "dependency portfolio" and you would (if you're large enough) have an actual "Deps Dept." and a Deps Manager whose sole job is tracking dependencies and their updates and patches, etc. > working on an app that has 82 npm dependencies Ach! Well, see, there's your problem right there. :-) Seriously though, one of the benefits of a dependency portfolio is to help you know when your system has gotten out of hand. The problems are still there even if you don't look at them, eh? > It's absolutely ridiculous to investigate all of them... but it's also necessary if you want to be sure... Ya feel. ;-)
  - gitgud 6y ago
    Thanks for the response, that's an interesting way to deal with it. How do you verify a dependency? Do you literally examine the source code? Make sure the build is reproduced? or just the meta data? (downloads, stars) has the portfolio actually prevented any vulnerabilities? It's pretty common for JS projects to have thousands of transitive dependencies, I'm not sure keeping a private portfolio is much use. The entire open-source ecosystem is built on the foundation of trust, if I use a package that's being used by 500 other packages, I can have a high degree of certainty that the package is safe, and by locking the dependencies with yarn.lock I can prevent sneaky updates from entering the system. Anyway maybe I'll look into the dependency portfolio, see how it goes.
    - carapace 6y ago
      Cheers! > How do you verify a dependency? Do you literally examine the source code? Yeah. It's part of the overhead of using the software. You also look at the history of bugs and how they were handled. > It's pretty common for JS projects to have thousands of transitive dependencies Yeah, I know, and it's bonkers IMO. > The entire open-source ecosystem is built on the foundation of trust In practice, yes, but in theory, no. The whole idea is that you get to see the code you're running, because the guys who wrote it are clowns. Free Software started when RMS wanted to fix his printer and Xerox said, "No." > if I use a package that's being used by 500 other packages, I can have a high degree of certainty that the package is safe I think history has shown that that reasoning is at best probabilistic, eh? You're gambling. Now, of course, there are limits. Some things get a pass. Do we audit the source of the bash shell? No, despite the fact that it's maintained by a single volunteer. > Anyway maybe I'll look into the dependency portfolio, see how it goes. Check out that "Hollywood Secrects" book I mentioned.
      - amoitnga 6y ago
        Thanks for sharing. It would be very interesting to know some of the examples when it payed off. Could you please share?
jfoster 6y ago
On what cadence do you do this? Every release?
- carapace 6y ago
  The "deps portfolio" gets updated whenever the deps change. In practice the flow goes like this: 0. A dev wants to use a new dependency, likely after experimenting with it a little bit. 1. Preliminary evaluation, which includes a transitive dependency scan. ("Too many dependencies" is a valid fail condition all on it's own.) 2. If everything looks good we bring it and it's deps into our internal repo. This includes the plumbing to add it to our dev|test|production envs. (Using Docker or whatever.) 3. Now the devs can use it in code destined for prod. There's a nice page in the company wiki that lists the exact version(s) with links to the docs, bug trackers, mailing lists, etc. and also the internal company lore for that package. It's tight. - - - - This might seem like a lot of work up front, but think about all the work it saves down the line.
  - jfoster 6y ago
    When one of the transitive dependencies fixes a security issue, is it then re-evaluated prior to being updated in the internal repo? I'm guessing you work at a pretty large tech company. It seems wasteful that so many companies might be replicating this work. I wonder if there might be the opportunity for a body to review & approve packages on behalf of many companies. Perhaps npm will eventually move in this direction.
    - carapace 6y ago
      > When one of the transitive dependencies fixes a security issue, is it then re-evaluated prior to being updated in the internal repo? Yes, but this is typically pretty low overhead. And when it's not, it usually means there is some issue that has to be addressed anyway. > I'm guessing you work at a pretty large tech company. I did once, but right now "we" is a tiny startup (we're using Elm and Erlang.) > I wonder if there might be the opportunity for a body to review & approve packages on behalf of many companies. Ideally, that's what Free/Open Source Software would be, eh? In the old days there were "sysadmins", System Administrators, who handled a lot of this sort of thing.
seanwilson 6y ago
How long does it take you to do this for e.g. all the dependencies for a bare bones Angular, Vue or React app?
- carapace 6y ago
  I couldn't begin to estimate. (We don't use those.)
  - seanwilson 6y ago
    I just don't see how anyone could realistically look at all the lines of code that any nontrivial JavaScript app relies on in any depth. I'm sure most people don't review the code for their operating system, drivers, web server, compiler, browser etc. but they do assess if the entities that write + support them are worth trusting. This is likely the only realistic approach for complex JavaScript apps also.
    - carapace 6y ago
      > I just don't see how anyone could realistically look at all the lines of code that any nontrivial JavaScript app relies on in any depth. Right. And that's really bad. > I'm sure most people don't review the code for their operating system, drivers, web server, compiler, browser etc. Right, but some people do. Hire one of them. (And if your "props dept." can't keep up with the changes to all the things that's also really bad.) > they do assess if the entities that write + support them are worth trusting. No one is a magic code elf. (Some people come close. Fabrice Bellard might count. But even that worthy commits bugs.) Like I said in a sib comment, yeah, some things get a pass. Bash shell for example. Then again, remember e.g. "heartbleed"?
      - deleted 6y ago
        [deleted]
      - seanwilson 6y ago
        > > I just don't see how anyone could realistically look at all the lines of code that any nontrivial JavaScript app relies on in any depth. > Right. And that's really bad. > > I'm sure most people don't review the code for their operating system, drivers, web server, compiler, browser etc. > Right, but some people do. Hire one of them. The interesting question isn't if you can do it, it's when should you, to what extent, and how much it will cost. "Always do it, do it in-depth, the time consumed isn't important and the budget isn't important" is a bad approach for example and isn't helpful to the OP. Successful software development is all about making appropriate tradeoffs - you're not going to get very far by conducting your own OpenSSL audit when all you want to do is write a todo web app.

Really? What about all the dependencies from those dependencies?... For example, our company is working on an app that has 82 npm dependencies and over 17,000 resolved npm packages... It's absolutely ridiculous to investigate all of them... but it's also necessary if you want to be sure...

carapace 6y ago
> Really? What about all the dependencies from those dependencies?... Yep, all the way to the end. I got the idea from a book called "Hollywood Secrets of Project Management Success" by James R. Persse. It's two books interleaved really, one is just a standard pitch for Agile methods (IMO), but the other is a presentation of the process that large film studios use to make movies. The movie industry is ~100 years old and mostly very good at bringing in projects on time and under budget. Somewhere in there he talks about how they'll track their dependencies in a kind of "portfolio", I forget the details, but it translates in IT to a "dependency portfolio" and you would (if you're large enough) have an actual "Deps Dept." and a Deps Manager whose sole job is tracking dependencies and their updates and patches, etc. > working on an app that has 82 npm dependencies Ach! Well, see, there's your problem right there. :-) Seriously though, one of the benefits of a dependency portfolio is to help you know when your system has gotten out of hand. The problems are still there even if you don't look at them, eh? > It's absolutely ridiculous to investigate all of them... but it's also necessary if you want to be sure... Ya feel. ;-)
- gitgud 6y ago
  Thanks for the response, that's an interesting way to deal with it. How do you verify a dependency? Do you literally examine the source code? Make sure the build is reproduced? or just the meta data? (downloads, stars) has the portfolio actually prevented any vulnerabilities? It's pretty common for JS projects to have thousands of transitive dependencies, I'm not sure keeping a private portfolio is much use. The entire open-source ecosystem is built on the foundation of trust, if I use a package that's being used by 500 other packages, I can have a high degree of certainty that the package is safe, and by locking the dependencies with yarn.lock I can prevent sneaky updates from entering the system. Anyway maybe I'll look into the dependency portfolio, see how it goes.
  - carapace 6y ago
    Cheers! > How do you verify a dependency? Do you literally examine the source code? Yeah. It's part of the overhead of using the software. You also look at the history of bugs and how they were handled. > It's pretty common for JS projects to have thousands of transitive dependencies Yeah, I know, and it's bonkers IMO. > The entire open-source ecosystem is built on the foundation of trust In practice, yes, but in theory, no. The whole idea is that you get to see the code you're running, because the guys who wrote it are clowns. Free Software started when RMS wanted to fix his printer and Xerox said, "No." > if I use a package that's being used by 500 other packages, I can have a high degree of certainty that the package is safe I think history has shown that that reasoning is at best probabilistic, eh? You're gambling. Now, of course, there are limits. Some things get a pass. Do we audit the source of the bash shell? No, despite the fact that it's maintained by a single volunteer. > Anyway maybe I'll look into the dependency portfolio, see how it goes. Check out that "Hollywood Secrects" book I mentioned.
    - amoitnga 6y ago
      Thanks for sharing. It would be very interesting to know some of the examples when it payed off. Could you please share?

jfoster 6y ago

On what cadence do you do this? Every release?

carapace 6y ago
The "deps portfolio" gets updated whenever the deps change. In practice the flow goes like this: 0. A dev wants to use a new dependency, likely after experimenting with it a little bit. 1. Preliminary evaluation, which includes a transitive dependency scan. ("Too many dependencies" is a valid fail condition all on it's own.) 2. If everything looks good we bring it and it's deps into our internal repo. This includes the plumbing to add it to our dev|test|production envs. (Using Docker or whatever.) 3. Now the devs can use it in code destined for prod. There's a nice page in the company wiki that lists the exact version(s) with links to the docs, bug trackers, mailing lists, etc. and also the internal company lore for that package. It's tight. - - - - This might seem like a lot of work up front, but think about all the work it saves down the line.
- jfoster 6y ago
  When one of the transitive dependencies fixes a security issue, is it then re-evaluated prior to being updated in the internal repo? I'm guessing you work at a pretty large tech company. It seems wasteful that so many companies might be replicating this work. I wonder if there might be the opportunity for a body to review & approve packages on behalf of many companies. Perhaps npm will eventually move in this direction.
  - carapace 6y ago
    > When one of the transitive dependencies fixes a security issue, is it then re-evaluated prior to being updated in the internal repo? Yes, but this is typically pretty low overhead. And when it's not, it usually means there is some issue that has to be addressed anyway. > I'm guessing you work at a pretty large tech company. I did once, but right now "we" is a tiny startup (we're using Elm and Erlang.) > I wonder if there might be the opportunity for a body to review & approve packages on behalf of many companies. Ideally, that's what Free/Open Source Software would be, eh? In the old days there were "sysadmins", System Administrators, who handled a lot of this sort of thing.

seanwilson 6y ago

How long does it take you to do this for e.g. all the dependencies for a bare bones Angular, Vue or React app?

carapace 6y ago
I couldn't begin to estimate. (We don't use those.)
- seanwilson 6y ago
  I just don't see how anyone could realistically look at all the lines of code that any nontrivial JavaScript app relies on in any depth. I'm sure most people don't review the code for their operating system, drivers, web server, compiler, browser etc. but they do assess if the entities that write + support them are worth trusting. This is likely the only realistic approach for complex JavaScript apps also.
  - carapace 6y ago
    > I just don't see how anyone could realistically look at all the lines of code that any nontrivial JavaScript app relies on in any depth. Right. And that's really bad. > I'm sure most people don't review the code for their operating system, drivers, web server, compiler, browser etc. Right, but some people do. Hire one of them. (And if your "props dept." can't keep up with the changes to all the things that's also really bad.) > they do assess if the entities that write + support them are worth trusting. No one is a magic code elf. (Some people come close. Fabrice Bellard might count. But even that worthy commits bugs.) Like I said in a sib comment, yeah, some things get a pass. Bash shell for example. Then again, remember e.g. "heartbleed"?
    - deleted 6y ago
      [deleted]
    - seanwilson 6y ago
      > > I just don't see how anyone could realistically look at all the lines of code that any nontrivial JavaScript app relies on in any depth. > Right. And that's really bad. > > I'm sure most people don't review the code for their operating system, drivers, web server, compiler, browser etc. > Right, but some people do. Hire one of them. The interesting question isn't if you can do it, it's when should you, to what extent, and how much it will cost. "Always do it, do it in-depth, the time consumed isn't important and the budget isn't important" is a bad approach for example and isn't helpful to the OP. Successful software development is all about making appropriate tradeoffs - you're not going to get very far by conducting your own OpenSSL audit when all you want to do is write a todo web app.