3 ms·

I had to read an early blog to figure out what it was: https://devblogs.microsoft.com/engineering-at-microsoft/dev-drive-and-copy-on-write-for-developer-perfor

I had to read an early blog to figure out what it was: https://devblogs.microsoft.com/engineering-at-microsoft/dev-drive-and-copy-on-write-for-developer-performance/ https://devblogs.microsoft.com/engineering-at-microsoft/dev-... "Copy-on-write (CoW) linking, also known as block cloning in the Windows API documentation, avoids fully copying a file by creating a metadata reference to the original data on-disk. CoW links are like hardlinks but are safe to write to, as the filesystem lazily copies the original data into the link as needed when opened for append or random-access write. With a CoW link you save disk space and time since the link consists of a small amount of metadata and they write fast." It seems there is a MacOS implementation: https://github.com/dotnet/runtime/pull/79243 https://github.com/dotnet/runtime/pull/79243 But it seems that this is .Net specific and not something that would speed up other build systems? It is confusing if this can apply to other build technologies other than .NET. Can it speed up TypeScript/JavaScript builds? Can it speed up Rust builds? Also what are the speed ups on these other platforms like MacOS and Linux? Is this something that all build systems and all OSes would benefit from? I guess this blog post for me raises more questions than it answers.

supriyo-biswas 2y ago
Certain filesystems like XFS do support CoW copying, and ZFS also does chunk-based deduplication. You'd typically use it through `cp --reflink` and similar.
kmeisthax 2y ago
The block cloning feature in newer versions of Windows is enabled by copy-on-write filesystems. macOS ships with one by default - APFS. Linux also has BTRFS, and before that there was the ZFS-on-Linux project. Microsoft is now shipping a CoW filesystem for Windows that appears to be a ReFS derivative. This is .NET specific insamuch as this is getting MSBuild to take advantage of ReFS features; but there's no particular reason why other build systems couldn't take advantage of ReFS in the same way MSBuild can take advantage of APFS. The main question is if the build system needs to make lots of copies of files that may not ever be updated. I imagine anything that does dependency fetching (especially Node/NPM) would benefit.
- bhouston 2y ago
  I know that npm now has a per-user cache in ~/.npm: https://docs.npmjs.com/cli/v7/commands/npm-cache https://docs.npmjs.com/cli/v7/commands/npm-cache I am not sure if it uses CoW to bring those packages into each project. If it did, that would be efficient and speed up "npm install" if the cache was warm.
  - derefr 2y ago
    Language package managers don't need copy-on-write, because there's no "write" — the files that make up dependencies are immutable from the perspective of the projects that they get installed into. There's no advantage to using CoW to "deploy" such files into work trees, over using plain-old hard links to do so. (And hard-linking these files is indeed what all the Node package managers — other than NPM — already do.)
  - styfle 2y ago
    Not npm but it’s used by pnpm for awhile. https://github.com/pnpm/pnpm/issues/1505 https://github.com/pnpm/pnpm/issues/1505
- nullindividual 2y ago
  Volume Shadow Copies (Win Svr 2003) on NTFS were the first implementation by Microsoft of CoW. But it was limited to VSS snapshots, so not useful for day-to-day storage usage. DevDrive is not a derivative of ReFS, it is ReFS with some file system filter bits turned off among a couple of other things. DevDrive is a collection of features centered around ReFS for the purposes of speeding up file read/writes (think node modules).
- adrian_b 2y ago
  Linux XFS has also added copy-on-write a few years ago. Initially I was not aware of this and I was surprised when I have copied a directory with a total size greater than 50 GB and the copy was instantaneous. At first I believed that I had given some wrong command, but then I searched the XFS documentation and I saw that this was a new feature at that time.
ack_complete 2y ago
It could speed up other build systems, but the .NET build system (MSBuild) has a particular design issue where by default it will copy dependencies local to each project that's using them (Copy Local). This leads to assemblies being copied multiple times throughout the filesystem according to the build process.
neonsunset 2y ago
The article talks about CoW as feature of ReFS, while the linked PR in dotnet/runtime is about adjusting the way File API issues calls on macOS so they take advantage of APFS's CoW instead.
- bhouston 2y ago
  Sorry, I was confused trying to decipher exactly what was going on. I apologize.

supriyo-biswas 2y ago

Certain filesystems like XFS do support CoW copying, and ZFS also does chunk-based deduplication. You'd typically use it through `cp --reflink` and similar.

kmeisthax 2y ago

The block cloning feature in newer versions of Windows is enabled by copy-on-write filesystems. macOS ships with one by default - APFS. Linux also has BTRFS, and before that there was the ZFS-on-Linux project. Microsoft is now shipping a CoW filesystem for Windows that appears to be a ReFS derivative. This is .NET specific insamuch as this is getting MSBuild to take advantage of ReFS features; but there's no particular reason why other build systems couldn't take advantage of ReFS in the same way MSBuild can take advantage of APFS. The main question is if the build system needs to make lots of copies of files that may not ever be updated. I imagine anything that does dependency fetching (especially Node/NPM) would benefit.

bhouston 2y ago
I know that npm now has a per-user cache in ~/.npm: https://docs.npmjs.com/cli/v7/commands/npm-cache https://docs.npmjs.com/cli/v7/commands/npm-cache I am not sure if it uses CoW to bring those packages into each project. If it did, that would be efficient and speed up "npm install" if the cache was warm.
- derefr 2y ago
  Language package managers don't need copy-on-write, because there's no "write" — the files that make up dependencies are immutable from the perspective of the projects that they get installed into. There's no advantage to using CoW to "deploy" such files into work trees, over using plain-old hard links to do so. (And hard-linking these files is indeed what all the Node package managers — other than NPM — already do.)
- styfle 2y ago
  Not npm but it’s used by pnpm for awhile. https://github.com/pnpm/pnpm/issues/1505 https://github.com/pnpm/pnpm/issues/1505
nullindividual 2y ago
Volume Shadow Copies (Win Svr 2003) on NTFS were the first implementation by Microsoft of CoW. But it was limited to VSS snapshots, so not useful for day-to-day storage usage. DevDrive is not a derivative of ReFS, it is ReFS with some file system filter bits turned off among a couple of other things. DevDrive is a collection of features centered around ReFS for the purposes of speeding up file read/writes (think node modules).
adrian_b 2y ago
Linux XFS has also added copy-on-write a few years ago. Initially I was not aware of this and I was surprised when I have copied a directory with a total size greater than 50 GB and the copy was instantaneous. At first I believed that I had given some wrong command, but then I searched the XFS documentation and I saw that this was a new feature at that time.

ack_complete 2y ago

It could speed up other build systems, but the .NET build system (MSBuild) has a particular design issue where by default it will copy dependencies local to each project that's using them (Copy Local). This leads to assemblies being copied multiple times throughout the filesystem according to the build process.

neonsunset 2y ago

The article talks about CoW as feature of ReFS, while the linked PR in dotnet/runtime is about adjusting the way File API issues calls on macOS so they take advantage of APFS's CoW instead.

bhouston 2y ago
Sorry, I was confused trying to decipher exactly what was going on. I apologize.