3 ms·

There’s something I’ve never quite understood and I wondered if someone who knows this history might be able to explain it. I’m used to Java and Maven. My pack

by MBCook 4y ago

There’s something I’ve never quite understood and I wondered if someone who knows this history might be able to explain it. I’m used to Java and Maven. My package manager (mvn) downloads 3rd party packages into my local repo, and all my Java projects have access to it. So I only have to download a specific version of a specific package once. Node and NPM came along, and I think were written by a former Java developer. But NPM puts a full (unzipped!) copy of every package in each of my projects. This leads to pointless duplication of files, and in Windows is slow as dirt due to how NTFS works. Yarn’s PNP mode sort of solves this, and it seems like recent versions of NPM may be changing things. I’m not sure. But why did it start that way? Does anyone know?

webkike 4y ago
The reason is that JavaScript is not compiled and needs the dependency’s source files to run, and those files need to be there when packaged.
- wging 4y ago
  That's not an obstacle to a central cache - you can solve that with symlinks. A build tool should make node_modules/foo a symlink to ~/.npm/cache/foo-1.23 if foo 1.23 is what you need. A packaging tool should be able to follow symlinks easily.
  - MBCook 4y ago
    Right. And Node/Webpack/whatever could easily be taught to read inside zip files instead of needing things unpacked on disk. That’s actually already been done for Yarn PNP support in Webpack.
jonahbenton 4y ago
The reason for a centralized local cache in Java is just because Java and jar files date back to the mid 1990s. Maven to the early 2000s. Storage was expensive. I wrote my first Java programs on a machine that had 16mb of RAM and a couple hundred mb of disk space. Centralized caches of compiled code present all sorts of logistical issues when it comes to deployment. Java has suffered through several generations of uberjar-like designs. Python, also dating to the 90s, as a source based language tried to solve for both local and centralized trees and the result is a horrendous soup. In contrast, PHP- yet another 1990s language- had a model of just dumping files into the directory and serving them, which was comparatively simple and uniform. Node, invented in the late 2000s by a C++ and Ruby programmer, was intended to simplify all sorts of aspects of writing webservers, leveraging in particular the deployment model of PHP (all needed files in source code in the local file tree) with the superior runtime performance and scalability, including support for async, of the Javascript V8 runtime, and with the benefit of having the same language both client and server side, something that was first tried with both Java and Javascript in the late 1990s but for various reasons failed. The other major envisioned benefit to Node was that it was its own webserver, and this has led to many other innovations. That Windows still after 25 years of the NT architecture has a file system that doesn't scale is not something most tooling in the web ecosystem doesn't concern itself with. Node, like PHP, was linux first and foremost. HTH.
- wging 4y ago
  It's not the storage used but the repeated downloading, if I understand the complaint correctly. Avoiding the network traffic if you have an exact match should be much faster on a clean build, even if you don't care about disk space.
  - MBCook 4y ago
    Right. If I have three projects using React 17.0.0, I have three copies. If I start another project with React 17.0.0 it will download another copy (and of the hundreds/thousands of dependencies) even though I may already have three copies of each. And what if I try to make a project while offline? Maven can do it if I’ve previously downloaded that package. My employer is big on Windows. All those copies of tens of thousands of files has a serious time cost. Even if dependencies were kept zipped up but not shared that would make more sense to me. Easier to deploy than individual files, smaller too.
  - dec0dedab0de 4y ago
    Npm has a cache
- User23 4y ago
  > That Windows still after 25 years of the NT architecture has a file system that doesn't scale is not something most tooling in the web ecosystem doesn't concern itself with. This isn't exactly true. It's not so much that NTFS is slow, it's that it's not optimized for the usage patterns that Linux's FS is. That being said, it's certainly possible to have high IO performance on Windows if you know what you're doing.
  - MBCook 4y ago
    Right. Creating and deleting thousands of tiny files and directories isn’t something it was designed to be fast at. But that’s exactly what NPM does.
  - adgjlsfhk1 4y ago
    ntfs is straight up bad. the way you get high performance io on windows is by inventing your own file system and dumping all the data into the same file because ntfs hasn't figured out how to make your ssd slower (yet).

by MBCook 4y ago

webkike 4y ago

The reason is that JavaScript is not compiled and needs the dependency’s source files to run, and those files need to be there when packaged.

wging 4y ago
That's not an obstacle to a central cache - you can solve that with symlinks. A build tool should make node_modules/foo a symlink to ~/.npm/cache/foo-1.23 if foo 1.23 is what you need. A packaging tool should be able to follow symlinks easily.
- MBCook 4y ago
  Right. And Node/Webpack/whatever could easily be taught to read inside zip files instead of needing things unpacked on disk. That’s actually already been done for Yarn PNP support in Webpack.

jonahbenton 4y ago

The reason for a centralized local cache in Java is just because Java and jar files date back to the mid 1990s. Maven to the early 2000s. Storage was expensive. I wrote my first Java programs on a machine that had 16mb of RAM and a couple hundred mb of disk space. Centralized caches of compiled code present all sorts of logistical issues when it comes to deployment. Java has suffered through several generations of uberjar-like designs. Python, also dating to the 90s, as a source based language tried to solve for both local and centralized trees and the result is a horrendous soup. In contrast, PHP- yet another 1990s language- had a model of just dumping files into the directory and serving them, which was comparatively simple and uniform. Node, invented in the late 2000s by a C++ and Ruby programmer, was intended to simplify all sorts of aspects of writing webservers, leveraging in particular the deployment model of PHP (all needed files in source code in the local file tree) with the superior runtime performance and scalability, including support for async, of the Javascript V8 runtime, and with the benefit of having the same language both client and server side, something that was first tried with both Java and Javascript in the late 1990s but for various reasons failed. The other major envisioned benefit to Node was that it was its own webserver, and this has led to many other innovations. That Windows still after 25 years of the NT architecture has a file system that doesn't scale is not something most tooling in the web ecosystem doesn't concern itself with. Node, like PHP, was linux first and foremost. HTH.

wging 4y ago
It's not the storage used but the repeated downloading, if I understand the complaint correctly. Avoiding the network traffic if you have an exact match should be much faster on a clean build, even if you don't care about disk space.
- MBCook 4y ago
  Right. If I have three projects using React 17.0.0, I have three copies. If I start another project with React 17.0.0 it will download another copy (and of the hundreds/thousands of dependencies) even though I may already have three copies of each. And what if I try to make a project while offline? Maven can do it if I’ve previously downloaded that package. My employer is big on Windows. All those copies of tens of thousands of files has a serious time cost. Even if dependencies were kept zipped up but not shared that would make more sense to me. Easier to deploy than individual files, smaller too.
- dec0dedab0de 4y ago
  Npm has a cache
User23 4y ago
> That Windows still after 25 years of the NT architecture has a file system that doesn't scale is not something most tooling in the web ecosystem doesn't concern itself with. This isn't exactly true. It's not so much that NTFS is slow, it's that it's not optimized for the usage patterns that Linux's FS is. That being said, it's certainly possible to have high IO performance on Windows if you know what you're doing.
- MBCook 4y ago
  Right. Creating and deleting thousands of tiny files and directories isn’t something it was designed to be fast at. But that’s exactly what NPM does.
- adgjlsfhk1 4y ago
  ntfs is straight up bad. the way you get high performance io on windows is by inventing your own file system and dumping all the data into the same file because ntfs hasn't figured out how to make your ssd slower (yet).