3 ms·

The block cloning feature in newer versions of Windows is enabled by copy-on-write filesystems. macOS ships with one by default - APFS. Linux also has BTRFS, an

The block cloning feature in newer versions of Windows is enabled by copy-on-write filesystems. macOS ships with one by default - APFS. Linux also has BTRFS, and before that there was the ZFS-on-Linux project. Microsoft is now shipping a CoW filesystem for Windows that appears to be a ReFS derivative. This is .NET specific insamuch as this is getting MSBuild to take advantage of ReFS features; but there's no particular reason why other build systems couldn't take advantage of ReFS in the same way MSBuild can take advantage of APFS. The main question is if the build system needs to make lots of copies of files that may not ever be updated. I imagine anything that does dependency fetching (especially Node/NPM) would benefit.

bhouston 2y ago
I know that npm now has a per-user cache in ~/.npm: https://docs.npmjs.com/cli/v7/commands/npm-cache https://docs.npmjs.com/cli/v7/commands/npm-cache I am not sure if it uses CoW to bring those packages into each project. If it did, that would be efficient and speed up "npm install" if the cache was warm.
- derefr 2y ago
  Language package managers don't need copy-on-write, because there's no "write" — the files that make up dependencies are immutable from the perspective of the projects that they get installed into. There's no advantage to using CoW to "deploy" such files into work trees, over using plain-old hard links to do so. (And hard-linking these files is indeed what all the Node package managers — other than NPM — already do.)
- styfle 2y ago
  Not npm but it’s used by pnpm for awhile. https://github.com/pnpm/pnpm/issues/1505 https://github.com/pnpm/pnpm/issues/1505
nullindividual 2y ago
Volume Shadow Copies (Win Svr 2003) on NTFS were the first implementation by Microsoft of CoW. But it was limited to VSS snapshots, so not useful for day-to-day storage usage. DevDrive is not a derivative of ReFS, it is ReFS with some file system filter bits turned off among a couple of other things. DevDrive is a collection of features centered around ReFS for the purposes of speeding up file read/writes (think node modules).
adrian_b 2y ago
Linux XFS has also added copy-on-write a few years ago. Initially I was not aware of this and I was surprised when I have copied a directory with a total size greater than 50 GB and the copy was instantaneous. At first I believed that I had given some wrong command, but then I searched the XFS documentation and I saw that this was a new feature at that time.

bhouston 2y ago

I know that npm now has a per-user cache in ~/.npm: https://docs.npmjs.com/cli/v7/commands/npm-cache https://docs.npmjs.com/cli/v7/commands/npm-cache I am not sure if it uses CoW to bring those packages into each project. If it did, that would be efficient and speed up "npm install" if the cache was warm.

derefr 2y ago
Language package managers don't need copy-on-write, because there's no "write" — the files that make up dependencies are immutable from the perspective of the projects that they get installed into. There's no advantage to using CoW to "deploy" such files into work trees, over using plain-old hard links to do so. (And hard-linking these files is indeed what all the Node package managers — other than NPM — already do.)
styfle 2y ago
Not npm but it’s used by pnpm for awhile. https://github.com/pnpm/pnpm/issues/1505 https://github.com/pnpm/pnpm/issues/1505

nullindividual 2y ago

Volume Shadow Copies (Win Svr 2003) on NTFS were the first implementation by Microsoft of CoW. But it was limited to VSS snapshots, so not useful for day-to-day storage usage. DevDrive is not a derivative of ReFS, it is ReFS with some file system filter bits turned off among a couple of other things. DevDrive is a collection of features centered around ReFS for the purposes of speeding up file read/writes (think node modules).

adrian_b 2y ago

Linux XFS has also added copy-on-write a few years ago. Initially I was not aware of this and I was surprised when I have copied a directory with a total size greater than 50 GB and the copy was instantaneous. At first I believed that I had given some wrong command, but then I searched the XFS documentation and I saw that this was a new feature at that time.