5 ms·

We built our app on node and typescript, and I would never choose it again at this point because of the package ecosystem. We do a lot to validate integrity of

We built our app on node and typescript, and I would never choose it again at this point because of the package ecosystem. We do a lot to validate integrity of packages (including checking in vetted archives to our repo), but it’s hard. Our images are ballooned to like 500-600MB (we’ve hit past the GB mark because of certain packages messing up dependencies before) based on a pretty conservative list of dependencies because of node_modules. I’m constantly fighting a battle against image size increases. The sheer amount of files in node_modules ensures that io is always a problem for image size and build speed on CI. Solutions like yarn berry hardly help: zipfs and patched tsservers is annoying in many editors still. Often packages break because package maintainers include implicit dependencies or the packages their packages depend on do so. Arc has frozen emacs for me several times when jumping to definition in a zip. I’m just so over the package situation for node.

martpie 6y ago
500-600mb sounds like you are shipping dev dependencies in your images. You should use npm/yarn's production flags when installing your dependencies for your images, so you only ship runtime dependencies. Your images will shrink to 100-150mb.
- citrons 6y ago
  Yeah, for building files in docker try using multi-stage builds. They on the "final" image you can only use prod dependencies.
  - SirensOfTitan 6y ago
    So we’re using multi-stage builds (totally awesome feature!), only including production dependencies, and I’m —squash ing the final image. The issue is that packages often balloon in size from an accidental transitive inclusion of Babel or webpack. Because JS packages depend on so much, the fanout virtually guarantees someone in your dependency tree will accidentally a dependency every now and then. It’s compounded by the fact that I have several different versions of the same package installed because folks don’t use peerDependencies when they should (and I’m afraid to pin most packages because I don’t want to introduce subtle bugs). I’d imagine I could solve this for us if I had enough time, but startup life leaves little time for battling with the package manager. And it creeps up on you slowly. (Also: we use next-transpire-modules @martpie, thanks for a great library)
    - ketzo 6y ago
      Yeah, sounds like the kind of problems you’d hand to your devops team, but oftentimes that’s just you :D I wonder how much “build maintenance” is/will become a factor in choosing between ecosystems.
    - skinner927 6y ago
      All I’m hearing is that now and then a teammate makes a mistake and installs a dev dependency as a production dependency. How is this NPM’s fault?
      - SirensOfTitan 6y ago
        This is not what I said.
- takeda 6y ago
  100-150mb of just a JavaScript code doesn't look that great either.
  - kchr 6y ago
    I think the problem is that you normally use 5% of the functions from your deps but still have to pull it all in. Minimizing dependencies and writing as much as possible on your own would probably cut a 500 MB nodejs project to 5 MB in plain JS. Note: Not a node dev but share the grief...
kitten_mittens_ 6y ago
Yarn 2’s Plug and Play has actually been a dream dep size wise. Once you get over the initial hurdles of setup, you can do things like vend your reps for zero installs. core-js alone (two major versions because @rjsf felt compelled to polyfill in a library) is 3K files worth of node_modules.
- SirensOfTitan 6y ago
  Maybe I’ll drive yarn v2 migration over the finish line then. Have you run into any hiccups post setup?
  - kitten_mittens_ 6y ago
    Once you get through unspecified dependencies of dependencies (and pnpifying tools), my last big hiccup with yarn 2 is that zipfs doesn’t work with vs code and typescript’s language server. So you can’t go to code or type definitions within zips.
    - acemarke 6y ago
      What specific issues are you seeing? I was just trying out Yarn v2 for the first time over the last couple days. I ran the `yarn dlx @yarnpkg/pnpify --sdk vscode` command suggested on their migration guide, and that generated a config file. VS Code then suggested that I add a ZipFS-related extension, and once I did that, I could view typedef files from libraries.
notanotherycomb 6y ago
Are you writing an OS? I'm pretty sure the linux kernel compiled is < 350mb. Debian net install is less than this

martpie 6y ago

500-600mb sounds like you are shipping dev dependencies in your images. You should use npm/yarn's production flags when installing your dependencies for your images, so you only ship runtime dependencies. Your images will shrink to 100-150mb.

citrons 6y ago
Yeah, for building files in docker try using multi-stage builds. They on the "final" image you can only use prod dependencies.
- SirensOfTitan 6y ago
  So we’re using multi-stage builds (totally awesome feature!), only including production dependencies, and I’m —squash ing the final image. The issue is that packages often balloon in size from an accidental transitive inclusion of Babel or webpack. Because JS packages depend on so much, the fanout virtually guarantees someone in your dependency tree will accidentally a dependency every now and then. It’s compounded by the fact that I have several different versions of the same package installed because folks don’t use peerDependencies when they should (and I’m afraid to pin most packages because I don’t want to introduce subtle bugs). I’d imagine I could solve this for us if I had enough time, but startup life leaves little time for battling with the package manager. And it creeps up on you slowly. (Also: we use next-transpire-modules @martpie, thanks for a great library)
  - ketzo 6y ago
    Yeah, sounds like the kind of problems you’d hand to your devops team, but oftentimes that’s just you :D I wonder how much “build maintenance” is/will become a factor in choosing between ecosystems.
  - skinner927 6y ago
    All I’m hearing is that now and then a teammate makes a mistake and installs a dev dependency as a production dependency. How is this NPM’s fault?
    - SirensOfTitan 6y ago
      This is not what I said.
takeda 6y ago
100-150mb of just a JavaScript code doesn't look that great either.
- kchr 6y ago
  I think the problem is that you normally use 5% of the functions from your deps but still have to pull it all in. Minimizing dependencies and writing as much as possible on your own would probably cut a 500 MB nodejs project to 5 MB in plain JS. Note: Not a node dev but share the grief...

kitten_mittens_ 6y ago

Yarn 2’s Plug and Play has actually been a dream dep size wise. Once you get over the initial hurdles of setup, you can do things like vend your reps for zero installs. core-js alone (two major versions because @rjsf felt compelled to polyfill in a library) is 3K files worth of node_modules.

SirensOfTitan 6y ago
Maybe I’ll drive yarn v2 migration over the finish line then. Have you run into any hiccups post setup?
- kitten_mittens_ 6y ago
  Once you get through unspecified dependencies of dependencies (and pnpifying tools), my last big hiccup with yarn 2 is that zipfs doesn’t work with vs code and typescript’s language server. So you can’t go to code or type definitions within zips.
  - acemarke 6y ago
    What specific issues are you seeing? I was just trying out Yarn v2 for the first time over the last couple days. I ran the `yarn dlx @yarnpkg/pnpify --sdk vscode` command suggested on their migration guide, and that generated a config file. VS Code then suggested that I add a ZipFS-related extension, and once I did that, I could view typedef files from libraries.

notanotherycomb 6y ago

Are you writing an OS? I'm pretty sure the linux kernel compiled is < 350mb. Debian net install is less than this