4 ms·

Stress test for parallel disk i/o using git and pnpm

77 points by robin_reala 1y ago

systemz 1y ago
How do we know it's not something related more to how Node.js handles I/O? Repo uses only JS
- tuyiown 1y ago
  It's even worse, it's not excluded that it might be a pure pnpm issue
- udev 1y ago
  The clean test is just an invocation of git clean, and shows the same kind of variation across operating systems, file systems, etc.
semiquaver 1y ago
I think it’s much more likely that I/O syscalls on MacOS are the culprit of slowness than APFS. I don’t see any entries in the table against HFS+.
- c0balt 1y ago
  That would also match with the workaround of using a Linux VM (assuming that VM disk i/o is bypassing the relevant part of the darwin i/o subsystem)
  - llimllib 1y ago
    node/npm operations are vastly slower for me on docker with a linux vm, I have tested this before. APFS does suck though, don't get me wrong edit: I just tested, 36s for an `npm i` on a decently large project I have locally, 46s with an overlay fs
    - dawnerd 1y ago
      Can also confirm. We’ve had some client projects that had npm inside of containers and those were horribly slow on macOS.
    - Scaevolus 1y ago
      Not an overlay fs, a VM writing to its own virtual block device.
      - llimllib 1y ago
        I should have been more specific, I tested with: `rm -rf node_modules && docker run --rm -ti -v $PWD:/app node:20.19.0-alpine3.20 sh -c "cd /app && time npm i"` So an overlay FS (if I'm understanding you/the OP correctly), not the VM's disk
ZeroConcerns 1y ago
It's interesting that both NTFS and APFS have now apparently been declared 'broken' due to performance problems with two very specific applications (Git and NPM), and that the reverse statement, that is's perhaps those applications that are being unreasonable, isn't quite as fashionable. I mean, I don't use NPM, but I recently copied over some small-ish raw Git repositories between machines, and had several WTF moments when I looked at what was actually going over the wire. Instead of refactoring two of the planet's most widely-used 'consumer' file systems, perhaps https://www.sqlite.org/appfileformat.html https://www.sqlite.org/appfileformat.html should be seen as a more reasonable way forward?
- p_ing 1y ago
  "NTFS" is fine, the Windows File System Filters are the culprit of I/O performance problems [with many small files/requests] on Windows.
  - jandrese 1y ago
    It's shocking how much performance you gain by temporarily turning off Windows Defender. I had a local disk to disk copy that the Windows File Transfer dialog estimated was going to finish in 10 hours after it settled down. It wasn't even that much data, just a few hundred GB, but that consisted of a great number of small files. On a hunch I tried disabling Windows Defender and the estimate dropped to about 30 minutes, and in the end it was spot on.
    - pixl97 1y ago
      Anti-virus kills small file IO. I work with a windows product that can deal with huge amounts of code files. With large enterprise that demands AV is on in most places the performance loss is pretty staggering, were you can easily lose 50% of your servers capacity per measurement interval.
  - ynik 1y ago
    Using a second partition D: is already twice as fast at small-file-writes compared to the system partition C:. This was on Windows 10 with both partitions using NTFS on the same SSD, and everything else on default settings. This is because C: has additional file system filters for system integrity that are not present on other drives; and also because C: has 8.3 name compatibility enabled by default, but additional NTFS partitions have that disabled. (so if your filenames are longer than the 8.3 format, Windows has to write twice the number of directory entries)
tuyiown 1y ago
This got me curious and it seems that there is known APFS behaviors slowing things down on parallel I/O on same folders [1] I would suspect that this is related to to the fact that reading/writing dirs have no atomic semantics in POSIX fs, and same directories are easily read/written simultaneously by unrelated processes, but surely is cause to edge cases bugs, especially in a world with backup system based on fs snapshots. This is a sad state of affairs. https://gregoryszorc.com/blog/2018/10/29/global-kernel-locks-in-apfs/ https://gregoryszorc.com/blog/2018/10/29/global-kernel-locks...
whartung 1y ago
Doesn’t macOS have dtrace? Couldn’t these tests be instrumented to give a deeper dive into what may be happening here?
hnt2025 1y ago
There is very low chance this is "something wrong with APFS on macOS". But there is something wrong for example with "Time Machine" backups reliability and breaking changes. Not supporting perfectly good hardware just to force users to buy new ones is also not nice. For example you can't easily move TM backup between disks or between hfs+ and apfs because time machine on hfs+ use hard links and on apfs snapshots. One time I've lost whole TM backup, not because of drive or filesystem failure but because of TM being unable to manage edge cases which arrise during network backup.
- bartvk 1y ago
  Is it that bad to buy a new backup disk when you change file systems? It's been many years ago since HFS+ got retired.
  - hnt2025 1y ago
    You're right, in the grand scheme of things, buying a new disk every few years isn't a major hardship but obstacle. My point wasn't really about the cost of hardware or not being able to solve problem, but rather the brittleness of the software, documentation and usabillity. The difficult backup migration is just one symptom of a larger problem: Time Machine isn't robust. It has sharp edges and can fail in ways that are hard to recover from, like the network backup I lost. That lack of reliability and breaking changes not just in backup software but in whole ecosystem is the real issue for me. Apple "obsoletes" things in its own timeframe regardless of userbase. I've got enormous respect for Apple products and people who are building them and with them. I wish Apple would have "LTS" version of MacOS and better documentation - often tools exist but aren't easy accessible.
- fragmede 1y ago
  > There is very low chance this is "something wrong with APFS on macOS". Why do you consider the chance of that to be low? APFS is functional and I trust to not eat my data, but it isn't a battle tested high performance file system that's done the rounds in the arena like, say, XFS.
sitkack 1y ago
It is apfs snapshots for Time Machine. Exclude your git folders from time machine and the perf of everything should improve.
- joshstrange 1y ago
  APFS snapshots for Time Machine suck so much. They take up a ton of disk space and TM's GUI is horrible, it will just sit there syncing for forever with nonsensical time estimates. Then, once it finishes syncing, it will _eventually_ clean up those old snapshots. I've considered dropping TM completely because of this and sticking with just Backblaze and CarbonCopyCloner instead. If I exclude my git directories (one of the most important things on my computer) then I really don't see the point of using TM at all with the other backups I do regularly.
  - prerok 1y ago
    Genuinely curious... why do you see the need to back up git? Even if I don't want to push to a server, I use a different disk or USB as an additional remote in git and push to that. Although I do that only for financial data, so only done once a month so it's not too big a hassle.
  - standard_indian 1y ago
    I have had good experience with Arq backup 7
  - fragmede 1y ago
    Yeah TM has its shortcomings. I went with Arq (non-affiliated) instead.
  - greggsy 1y ago
    Despite its apparent opaqueness, TM is reasonably transparent for the everyday Mac user
- Brajeshwar 1y ago
  I’m very serious and not kidding but where are these settings? I always thought that Time Machine just eats my Computer whole and in chunks with no option for any customization.
joshstrange 1y ago
I'm not sure if it's related but recently (last 2 years-ish?) I've noticed some issue with watching files on macOS. I edit a file but the dev server doesn't pick it up for way longer than it should. It's sometimes faster for me to restart the dev server than wait for it to notice the files changed. I have not dug deeper into this aside from raising my open file limits and closing some apps I thought might be the culprit (but ended up not being the cause). It's annoying because it's not consistent. The dev server will work just fine for a while then randomly slow down. It sucks because sometimes I spend a decent amount of time trying to figure out why my change/fix/etc isn't working only to realize I haven't been testing updated code (I made a code change and thought it hot-reloaded but it didn't).
- Scaevolus 1y ago
  File watching is flaky enough that polling based implementations are a better choice. You can stat tens of thousands of files every second, and easily optimize it to check frequently changes files more often. https://esbuild.github.io/api/#watch https://esbuild.github.io/api/#watch
fudged71 1y ago
Maybe it’s a combination of two things (a perfect storm): 1) APFS's snapshot feature, driven by Time Machine, adds a heavy tax on the massive number of file writes and deletes performed by pnpm and git. 2) macOS's less efficient handling of parallel I/O negates the speed benefits of modern, concurrent tooling, creating a bottleneck that is particularly severe when combined with the overhead from snapshots

77 points by robin_reala 1y ago

systemz 1y ago

How do we know it's not something related more to how Node.js handles I/O? Repo uses only JS

tuyiown 1y ago
It's even worse, it's not excluded that it might be a pure pnpm issue
udev 1y ago
The clean test is just an invocation of git clean, and shows the same kind of variation across operating systems, file systems, etc.

semiquaver 1y ago

I think it’s much more likely that I/O syscalls on MacOS are the culprit of slowness than APFS. I don’t see any entries in the table against HFS+.

c0balt 1y ago
That would also match with the workaround of using a Linux VM (assuming that VM disk i/o is bypassing the relevant part of the darwin i/o subsystem)
- llimllib 1y ago
  node/npm operations are vastly slower for me on docker with a linux vm, I have tested this before. APFS does suck though, don't get me wrong edit: I just tested, 36s for an `npm i` on a decently large project I have locally, 46s with an overlay fs
  - dawnerd 1y ago
    Can also confirm. We’ve had some client projects that had npm inside of containers and those were horribly slow on macOS.
  - Scaevolus 1y ago
    Not an overlay fs, a VM writing to its own virtual block device.
    - llimllib 1y ago
      I should have been more specific, I tested with: `rm -rf node_modules && docker run --rm -ti -v $PWD:/app node:20.19.0-alpine3.20 sh -c "cd /app && time npm i"` So an overlay FS (if I'm understanding you/the OP correctly), not the VM's disk

ZeroConcerns 1y ago

It's interesting that both NTFS and APFS have now apparently been declared 'broken' due to performance problems with two very specific applications (Git and NPM), and that the reverse statement, that is's perhaps those applications that are being unreasonable, isn't quite as fashionable. I mean, I don't use NPM, but I recently copied over some small-ish raw Git repositories between machines, and had several WTF moments when I looked at what was actually going over the wire. Instead of refactoring two of the planet's most widely-used 'consumer' file systems, perhaps https://www.sqlite.org/appfileformat.html https://www.sqlite.org/appfileformat.html should be seen as a more reasonable way forward?

p_ing 1y ago
"NTFS" is fine, the Windows File System Filters are the culprit of I/O performance problems [with many small files/requests] on Windows.
- jandrese 1y ago
  It's shocking how much performance you gain by temporarily turning off Windows Defender. I had a local disk to disk copy that the Windows File Transfer dialog estimated was going to finish in 10 hours after it settled down. It wasn't even that much data, just a few hundred GB, but that consisted of a great number of small files. On a hunch I tried disabling Windows Defender and the estimate dropped to about 30 minutes, and in the end it was spot on.
  - pixl97 1y ago
    Anti-virus kills small file IO. I work with a windows product that can deal with huge amounts of code files. With large enterprise that demands AV is on in most places the performance loss is pretty staggering, were you can easily lose 50% of your servers capacity per measurement interval.
- ynik 1y ago
  Using a second partition D: is already twice as fast at small-file-writes compared to the system partition C:. This was on Windows 10 with both partitions using NTFS on the same SSD, and everything else on default settings. This is because C: has additional file system filters for system integrity that are not present on other drives; and also because C: has 8.3 name compatibility enabled by default, but additional NTFS partitions have that disabled. (so if your filenames are longer than the 8.3 format, Windows has to write twice the number of directory entries)

tuyiown 1y ago

This got me curious and it seems that there is known APFS behaviors slowing things down on parallel I/O on same folders [1] I would suspect that this is related to to the fact that reading/writing dirs have no atomic semantics in POSIX fs, and same directories are easily read/written simultaneously by unrelated processes, but surely is cause to edge cases bugs, especially in a world with backup system based on fs snapshots. This is a sad state of affairs. https://gregoryszorc.com/blog/2018/10/29/global-kernel-locks-in-apfs/ https://gregoryszorc.com/blog/2018/10/29/global-kernel-locks...

whartung 1y ago

Doesn’t macOS have dtrace? Couldn’t these tests be instrumented to give a deeper dive into what may be happening here?

hnt2025 1y ago

There is very low chance this is "something wrong with APFS on macOS". But there is something wrong for example with "Time Machine" backups reliability and breaking changes. Not supporting perfectly good hardware just to force users to buy new ones is also not nice. For example you can't easily move TM backup between disks or between hfs+ and apfs because time machine on hfs+ use hard links and on apfs snapshots. One time I've lost whole TM backup, not because of drive or filesystem failure but because of TM being unable to manage edge cases which arrise during network backup.

bartvk 1y ago
Is it that bad to buy a new backup disk when you change file systems? It's been many years ago since HFS+ got retired.
- hnt2025 1y ago
  You're right, in the grand scheme of things, buying a new disk every few years isn't a major hardship but obstacle. My point wasn't really about the cost of hardware or not being able to solve problem, but rather the brittleness of the software, documentation and usabillity. The difficult backup migration is just one symptom of a larger problem: Time Machine isn't robust. It has sharp edges and can fail in ways that are hard to recover from, like the network backup I lost. That lack of reliability and breaking changes not just in backup software but in whole ecosystem is the real issue for me. Apple "obsoletes" things in its own timeframe regardless of userbase. I've got enormous respect for Apple products and people who are building them and with them. I wish Apple would have "LTS" version of MacOS and better documentation - often tools exist but aren't easy accessible.
fragmede 1y ago
> There is very low chance this is "something wrong with APFS on macOS". Why do you consider the chance of that to be low? APFS is functional and I trust to not eat my data, but it isn't a battle tested high performance file system that's done the rounds in the arena like, say, XFS.

sitkack 1y ago

It is apfs snapshots for Time Machine. Exclude your git folders from time machine and the perf of everything should improve.

joshstrange 1y ago
APFS snapshots for Time Machine suck so much. They take up a ton of disk space and TM's GUI is horrible, it will just sit there syncing for forever with nonsensical time estimates. Then, once it finishes syncing, it will _eventually_ clean up those old snapshots. I've considered dropping TM completely because of this and sticking with just Backblaze and CarbonCopyCloner instead. If I exclude my git directories (one of the most important things on my computer) then I really don't see the point of using TM at all with the other backups I do regularly.
- prerok 1y ago
  Genuinely curious... why do you see the need to back up git? Even if I don't want to push to a server, I use a different disk or USB as an additional remote in git and push to that. Although I do that only for financial data, so only done once a month so it's not too big a hassle.
- standard_indian 1y ago
  I have had good experience with Arq backup 7
- fragmede 1y ago
  Yeah TM has its shortcomings. I went with Arq (non-affiliated) instead.
- greggsy 1y ago
  Despite its apparent opaqueness, TM is reasonably transparent for the everyday Mac user
Brajeshwar 1y ago
I’m very serious and not kidding but where are these settings? I always thought that Time Machine just eats my Computer whole and in chunks with no option for any customization.

joshstrange 1y ago

I'm not sure if it's related but recently (last 2 years-ish?) I've noticed some issue with watching files on macOS. I edit a file but the dev server doesn't pick it up for way longer than it should. It's sometimes faster for me to restart the dev server than wait for it to notice the files changed. I have not dug deeper into this aside from raising my open file limits and closing some apps I thought might be the culprit (but ended up not being the cause). It's annoying because it's not consistent. The dev server will work just fine for a while then randomly slow down. It sucks because sometimes I spend a decent amount of time trying to figure out why my change/fix/etc isn't working only to realize I haven't been testing updated code (I made a code change and thought it hot-reloaded but it didn't).

Scaevolus 1y ago
File watching is flaky enough that polling based implementations are a better choice. You can stat tens of thousands of files every second, and easily optimize it to check frequently changes files more often. https://esbuild.github.io/api/#watch https://esbuild.github.io/api/#watch

fudged71 1y ago

Maybe it’s a combination of two things (a perfect storm): 1) APFS's snapshot feature, driven by Time Machine, adds a heavy tax on the massive number of file writes and deletes performed by pnpm and git. 2) macOS's less efficient handling of parallel I/O negates the speed benefits of modern, concurrent tooling, creating a bottleneck that is particularly severe when combined with the overhead from snapshots