7 ms·

What I found frustrating is how fast a front-end project is getting old. Taking a NPM+Framework project of 9 months old I often have a low change of install-ru

What I found frustrating is how fast a front-end project is getting old. Taking a NPM+Framework project of 9 months old I often have a low change of install-run at the first time. Always there's an update, an incompatibility, or something that brakes the build process and I need a couple of hours to fix it again. Is not possible to work like this in the long run. Front-end projects are investments of time and money and I want them to be durable and maintainable for a long period. P.S. I started digging into clojurescript/webassembly for this reason

danShumway 6y ago
There's a dirty secret that's fallen out of favor in the Node community, but that helps dramatically with this problem: it is usually a good idea to vendor dependencies for front-end projects (basically any project you're making that isn't being published to NPM). The trick with this is you have to avoid dependencies that compile C code or native binaries as part of their installation process, since those binaries might not be portable. I try when possible to avoid anything that's using node-gyp. But if you are vendoring dependencies and they're compatible with the system you're compiling on, then you will never need to worry about libraries breaking or changing underneath your nose. You will still need to worry about security vulnerabilities, but that's just the case with all software. At the very least, 10 years from now your project will still compile.
- WiseWeasel 6y ago
  By “vendoring” dependencies, I’m guessing you are talking about pulling them out of node_modules (into a js/vendor dir), and checking them into your repo, repeating the process every time you want to update. I will make an offering to Cthulhu if it’ll ensure we won't be using webpack and node 10 years from now in order to build those old snapshots without considerable effort.
  - filipo 6y ago
    Not a node dev myself but why not simply reference dependencies with fixed version? According to semver (that is probably what most node projects use anyway), you are free to use latest minor and patch revisions and leave the major fixed. Shouldn‘t this prevent most compatibility problems?
    - danShumway 6y ago
      > why not simply reference dependencies with fixed version? This is the second best strategy, but: - node makes it kind of annoying to do this, because unless you do a clean build every time (which is much slower) `package-lock.json` will get largely ignored. - I have seen multiple projects break with semver. Just because people are supposed to avoid breaking changes in minor releases doesn't mean they actually do. - Even if you're downloading the same version, there are scenarios where natively compiled binaries can bite you. I've run into issues where node-gyp just wouldn't work unless I downloaded a completely separate compile toolchain onto Windows and restarted the computer. If you can avoid that and have all of your dependencies be pure Javascript, you will eventually save yourself a lot of horrible dev-ops work when the "quick" install on a new VM turns into an hour long process of watching Visual Studio tools download (tools that to GP's point may not be available or compatible 10 years from now). - And finally, sometimes you want to do work across multiple branches without an internet connection, and vendoring your dependencies allows you to check out a branch and know you have the correct dependencies even if you don't have a connection to run `npm install` with.
      - filipo 6y ago
        Ok I see; thank your for the explanation. So out of interest, I did some research and found that approx. 30% of packages depend on native implementations further down... so this seems to be a huge problem... Additionally I looked at the sqlite3 package and not too surprisingly it uses node-gyp as well. So even checking in node_modules might cause the problems you described. Out of interest: which backend language do you recommend to reduce such problems...?
        danShumway 6y ago
  - danShumway 6y ago
    > pulling them out of node_modules No, even further. Just commit your node_modules folder. There's this idea that vendoring has to be really complicated, but committing node_modules was actually official advice early on in Node's history. And what that means is that if you clone your repo, you just run `build`, you don't have to change or configure anything at all. The only dependency you have in your project becomes a node binary -- and honestly, you can vendor that someplace as well, node is reasonably portable. There are a lot of advantages to this, not the least being that if you're demoing or working on a project someplace without an internet connection, having an up-to-date Git repo is all you need, you just have your entire project. If you switch branches, your project is ready to go. Of course, if you have something like 1G of dependencies, checking them into Git starts becoming kind of cumbersome. But if possible, avoid that, for the same reasons you would avoid JS dependencies that compile native binaries. Having gigabytes of dependencies is going to be a nightmare for debugging and security, no matter what language you're in. But I have built some pretty complicated projects where vendoring is fine. It doesn't balloon the size of my git repo, it doesn't make the project less portable, and it regularly saves my butt when I forget to check out the right branch before getting on a train without an Internet connection. And all you need to do is not add node_modules to your gitignore.
infensus 6y ago
Are you using packgage-lock.json/yarn.lock?

danShumway 6y ago

There's a dirty secret that's fallen out of favor in the Node community, but that helps dramatically with this problem: it is usually a good idea to vendor dependencies for front-end projects (basically any project you're making that isn't being published to NPM). The trick with this is you have to avoid dependencies that compile C code or native binaries as part of their installation process, since those binaries might not be portable. I try when possible to avoid anything that's using node-gyp. But if you are vendoring dependencies and they're compatible with the system you're compiling on, then you will never need to worry about libraries breaking or changing underneath your nose. You will still need to worry about security vulnerabilities, but that's just the case with all software. At the very least, 10 years from now your project will still compile.

WiseWeasel 6y ago
By “vendoring” dependencies, I’m guessing you are talking about pulling them out of node_modules (into a js/vendor dir), and checking them into your repo, repeating the process every time you want to update. I will make an offering to Cthulhu if it’ll ensure we won't be using webpack and node 10 years from now in order to build those old snapshots without considerable effort.
- filipo 6y ago
  Not a node dev myself but why not simply reference dependencies with fixed version? According to semver (that is probably what most node projects use anyway), you are free to use latest minor and patch revisions and leave the major fixed. Shouldn‘t this prevent most compatibility problems?
  - danShumway 6y ago
    > why not simply reference dependencies with fixed version? This is the second best strategy, but: - node makes it kind of annoying to do this, because unless you do a clean build every time (which is much slower) `package-lock.json` will get largely ignored. - I have seen multiple projects break with semver. Just because people are supposed to avoid breaking changes in minor releases doesn't mean they actually do. - Even if you're downloading the same version, there are scenarios where natively compiled binaries can bite you. I've run into issues where node-gyp just wouldn't work unless I downloaded a completely separate compile toolchain onto Windows and restarted the computer. If you can avoid that and have all of your dependencies be pure Javascript, you will eventually save yourself a lot of horrible dev-ops work when the "quick" install on a new VM turns into an hour long process of watching Visual Studio tools download (tools that to GP's point may not be available or compatible 10 years from now). - And finally, sometimes you want to do work across multiple branches without an internet connection, and vendoring your dependencies allows you to check out a branch and know you have the correct dependencies even if you don't have a connection to run `npm install` with.
    - filipo 6y ago
      Ok I see; thank your for the explanation. So out of interest, I did some research and found that approx. 30% of packages depend on native implementations further down... so this seems to be a huge problem... Additionally I looked at the sqlite3 package and not too surprisingly it uses node-gyp as well. So even checking in node_modules might cause the problems you described. Out of interest: which backend language do you recommend to reduce such problems...?
      - danShumway 6y ago
- danShumway 6y ago
  > pulling them out of node_modules No, even further. Just commit your node_modules folder. There's this idea that vendoring has to be really complicated, but committing node_modules was actually official advice early on in Node's history. And what that means is that if you clone your repo, you just run `build`, you don't have to change or configure anything at all. The only dependency you have in your project becomes a node binary -- and honestly, you can vendor that someplace as well, node is reasonably portable. There are a lot of advantages to this, not the least being that if you're demoing or working on a project someplace without an internet connection, having an up-to-date Git repo is all you need, you just have your entire project. If you switch branches, your project is ready to go. Of course, if you have something like 1G of dependencies, checking them into Git starts becoming kind of cumbersome. But if possible, avoid that, for the same reasons you would avoid JS dependencies that compile native binaries. Having gigabytes of dependencies is going to be a nightmare for debugging and security, no matter what language you're in. But I have built some pretty complicated projects where vendoring is fine. It doesn't balloon the size of my git repo, it doesn't make the project less portable, and it regularly saves my butt when I forget to check out the right branch before getting on a train without an Internet connection. And all you need to do is not add node_modules to your gitignore.

infensus 6y ago

Are you using packgage-lock.json/yarn.lock?