8 ms·

That's a very rigorously written article. Let's also note the 4x speed increase on windows 10, once again underlining just how slow windows filesystem calls ar

by lc64 2y ago

That's a very rigorously written article. Let's also note the 4x speed increase on windows 10, once again underlining just how slow windows filesystem calls are, when compared to direct access, and other (kernel, filesystem) combinations.

wolfi1 2y ago
maybe the malware detection program adds to the performance as well
- cjblomqvist 2y ago
  NTFS is really horrible handling many small files. When compiling/watching node modules (easily 10-100k files), we've seen a 10x size difference internally (same hardware, just different OSes). At some point that meant a compile time difference of 10-30 sec vs 6-10 min. Not fun.
  - 01HNNWZ0MV43FF 2y ago
    Must be why windows 11 added that dev drive feature
    - TeMPOraL 2y ago
      That, and continuous scanning of all file IO corporations love so much can 2x-10x that time still. Dev drive is excluded from Defender scrutiny by default because of that.
      - naikrovek 2y ago
        Not excluded, but the scanning in a dev drive is asynchronous rather than synchronous.
  - unchar1 2y ago
    That may be due to a combination of Malware detection + most unix programs not really written to take advantage of the features NTFS has to offer This is a great talk on the topic https://youtu.be/qbKGw8MQ0i8?si=rh6WJ3DV0jDZLddn https://youtu.be/qbKGw8MQ0i8?si=rh6WJ3DV0jDZLddn
    - account42 2y ago
      It's more a case of Linux programs not being written to work around the performance issues of Windows filesystems + layers above them. NTFS doesn't offer magical featurs that fix the performance.
  - chipdart 2y ago
    > NTFS is really horrible handling many small files. To pile onto NTFS, it's performance is so notoriously bad that there are developer teams working on Windows projects that configure their build farm to do cross builds from Linux to Windows just to avoid the performance penalty.
    - ta23948234 2y ago
      As an anecdote, we had a really long build time for our pipeline (going back prob 15 years). I argued for a Linux laptop, and the boss said, "OK, prove it. Here's two equivalent laptops, time it.". Turns out there was zero difference, or negligible (Windows won), between compilation times. That has always annoyed me.
      - rapind 2y ago
        Love that your boss was open to the idea but also wanted data to back it up. (imagine a slow clap)
        ta9345908345098 2y ago
        Yep, was a good experience. I'm sure with fine tuning it (Linux) could've been better, but I ate that humble pie.
        goodpoint 2y ago
        A single datapoint from an experiment done by only one person you call it data? What's not to love...
        tssva 2y ago
        It was the only data point which mattered for the decision that needed to be made.
      - chipdart 2y ago
        > Turns out there was zero difference, or negligible (Windows won), between compilation times. I think there was something seriously flawed in your test. If you Google for a minute, you find multiple posts on how moving the same builds to Linux led to performance improvements in the range of 40% drops in build times. Some anecdotes even compare doing the same builds in Ubunto with NTFS to see double-digit gains. NTFS is notoriously awful in scenarios involving reading/writing many small projects. This is the bottleneck in Windows builds. There is a myriad of benchmarks documenting this problem. Nowadays there are plenty of cross-platform projects to serve as benchmarks. Checking this can be as easy as checking out a project, start a full rebuild, and check how long it takes.
  - butz 2y ago
    So we should put node_modules into SQLite database?
    - Cupprum 2y ago
      Honestly, maybe? Would be a fun project to look at at least.
  - snazz 2y ago
    It’s somewhat more complex than “NTFS is slow”. Here’s a good explanation: https://github.com/Microsoft/WSL/issues/873#issuecomment-425272829 https://github.com/Microsoft/WSL/issues/873#issuecomment-425... I’ve benchmarked deleting files (around ~65,000 small node_modules sort of files) and it takes 40 seconds through Explorer, 20 seconds with rd, and roughly a second inside WSL2 (cloned to the VM’s ext4 virtual hard drive).
  - polski-g 2y ago
    Why not use https://github.com/bobranten/Ext4Fsd https://github.com/bobranten/Ext4Fsd ??
  - nullindividual 2y ago
    NTFS is perfectly fine at handling small files and performs on-par with other modern file systems. The issue is Defender in sync mode/other AV/other file system filters. DevDrive as noted by default uses an async scanning technique as well as ReFS. ReFS will suffer the exact same performance issues with Defender (or other AV/other file system filters) doing its thing when running in sync mode, which it does by default for ReFS-formatted drives in Windows Server. https://gregoryszorc.com/blog/2021/04/06/surprisingly-slow/ https://gregoryszorc.com/blog/2021/04/06/surprisingly-slow/ https://news.ycombinator.com/item?id=26737521 https://news.ycombinator.com/item?id=26737521 > Except for CloseHandle(). These calls were often taking 1-10+ milliseconds to complete. > While I didn't realize it at the time, the cause for this was/is Windows Defender. Windows Defender (and other anti-virus / scanning software) typically work on Windows by installing what's called a filesystem filter driver. This doesn't take away from your point that _it is slow_, but the reasons are not due to the file system in use.
    - Aerroon 2y ago
      >The issue is Defender in sync mode/other AV/other file system filters. I've had folders take a full minute to open on an SSD. It got to the point where I went to open the folder, it started loading. I needed the file quickly, so I searched for it online, found it, and opened it before windows finished loading that folder for me. After exempting that folder from Windows Defender the folder loads instantly. For the life of me I cannot understand why Defender blocks Explorer.
      - nullindividual 2y ago
        > For the life of me I cannot understand why Defender blocks Explorer. I suppose if you wanted to find out, you could use dtrace/ETW. Explorer has other things going on, though, including other apps that hook into it (shell extensions, like Adobe Reader, TortiseGit/SVN, and so on) which can certainly cause performance issues.
      - layer8 2y ago
        Probably because Explorer hosts shell hooks which can potentially execute arbitrary code. Just one example: File icons or thumbnails can be dynamically generated by shell extensions based on the file contents. A maliciously crafted file could potentially exploit a vulnerability in such a shell extension.
- deleted 2y ago
  [deleted]
- chefandy 2y ago
  Their "windows dev drive" setup addresses this. I haven't tested it myself but I saw a couple of inexpertly executed tests showing significant performance gains. I honestly have no idea if my compile times are quicker.
  - nullindividual 2y ago
    You can get benches from the horses mouth. https://devblogs.microsoft.com/visualstudio/devdrive/ https://devblogs.microsoft.com/visualstudio/devdrive/ But the primary improvement comes from async A/V, not the file system. ReFS offers other improvements such as CoW and instant file initialization which can benefit developers. From this standpoint, it is the correct choice over NTFS. https://devblogs.microsoft.com/engineering-at-microsoft/dev-drive-and-copy-on-write-for-developer-performance/ https://devblogs.microsoft.com/engineering-at-microsoft/dev-...
nextaccountic 2y ago
But is it faster than accessing the filesystem with io_uring as well? I feel like this article should be updated
- nullindividual 2y ago
  io_uring/IOCP won't change how long it takes to access a file.
  - nextaccountic 2y ago
    So what's exactly the performance benefits of using io_uring to access files?
    - nullindividual 2y ago
      It allows you to do something else while waiting for the request to complete. Linux has historically bad hacks to enable this. Windows NT has had IOCP since it's inception, but not only for storage devices, as io_uring is limited to, but networking, TCP sockets, mail slots, named pipes, etc. "Are you done yet?" vs "Get back to me when you're ready, gonna go do something else". Aka blocking i/o vs nonblocking i/o. (I always wonder why Windows NT doesn't get more high performance implementations for networking et. al. with this feature; licensing? trade off in perf elsewhere? can't tweak kernel params to your heart's desire?) io_uring implemented an additional feature of a ring buffer, but Microsoft has followed this feature; I think it was first introduced in a later build of Windows 10 and should be in 11 & 2022. Microsoft has a graphical example of IOCP: https://learn.microsoft.com/en-us/windows/win32/fileio/synchronous-and-asynchronous-i-o https://learn.microsoft.com/en-us/windows/win32/fileio/synch... More info on a comparison of ring implementations in Windows/Linux: https://windows-internals.com/ioring-vs-io_uring-a-comparison-of-windows-and-linux-implementations/ https://windows-internals.com/ioring-vs-io_uring-a-compariso...
alexhornby 2y ago
https://news.ycombinator.com/item?id=18783525 https://news.ycombinator.com/item?id=18783525 has previous discussion on why windows filesystem is slow
OttoCoddo 2y ago
With the right code, NTFS is not much slower than ext4, for example. Nearly 3% for tar.gz and more than 30% for a heavily multi-threaded use like Pack. https://forum.lazarus.freepascal.org/index.php/topic,66281.msg507111.html#msg507111 https://forum.lazarus.freepascal.org/index.php/topic,66281.m...
deleted 2y ago
[deleted]

by lc64 2y ago

wolfi1 2y ago

maybe the malware detection program adds to the performance as well

cjblomqvist 2y ago
NTFS is really horrible handling many small files. When compiling/watching node modules (easily 10-100k files), we've seen a 10x size difference internally (same hardware, just different OSes). At some point that meant a compile time difference of 10-30 sec vs 6-10 min. Not fun.
- 01HNNWZ0MV43FF 2y ago
  Must be why windows 11 added that dev drive feature
  - TeMPOraL 2y ago
    That, and continuous scanning of all file IO corporations love so much can 2x-10x that time still. Dev drive is excluded from Defender scrutiny by default because of that.
    - naikrovek 2y ago
      Not excluded, but the scanning in a dev drive is asynchronous rather than synchronous.
- unchar1 2y ago
  That may be due to a combination of Malware detection + most unix programs not really written to take advantage of the features NTFS has to offer This is a great talk on the topic https://youtu.be/qbKGw8MQ0i8?si=rh6WJ3DV0jDZLddn https://youtu.be/qbKGw8MQ0i8?si=rh6WJ3DV0jDZLddn
  - account42 2y ago
    It's more a case of Linux programs not being written to work around the performance issues of Windows filesystems + layers above them. NTFS doesn't offer magical featurs that fix the performance.
- chipdart 2y ago
  > NTFS is really horrible handling many small files. To pile onto NTFS, it's performance is so notoriously bad that there are developer teams working on Windows projects that configure their build farm to do cross builds from Linux to Windows just to avoid the performance penalty.
  - ta23948234 2y ago
    As an anecdote, we had a really long build time for our pipeline (going back prob 15 years). I argued for a Linux laptop, and the boss said, "OK, prove it. Here's two equivalent laptops, time it.". Turns out there was zero difference, or negligible (Windows won), between compilation times. That has always annoyed me.
    - rapind 2y ago
      Love that your boss was open to the idea but also wanted data to back it up. (imagine a slow clap)
      - ta9345908345098 2y ago
        Yep, was a good experience. I'm sure with fine tuning it (Linux) could've been better, but I ate that humble pie.
      - goodpoint 2y ago
        A single datapoint from an experiment done by only one person you call it data? What's not to love...
        tssva 2y ago
        It was the only data point which mattered for the decision that needed to be made.
    - chipdart 2y ago
      > Turns out there was zero difference, or negligible (Windows won), between compilation times. I think there was something seriously flawed in your test. If you Google for a minute, you find multiple posts on how moving the same builds to Linux led to performance improvements in the range of 40% drops in build times. Some anecdotes even compare doing the same builds in Ubunto with NTFS to see double-digit gains. NTFS is notoriously awful in scenarios involving reading/writing many small projects. This is the bottleneck in Windows builds. There is a myriad of benchmarks documenting this problem. Nowadays there are plenty of cross-platform projects to serve as benchmarks. Checking this can be as easy as checking out a project, start a full rebuild, and check how long it takes.
- butz 2y ago
  So we should put node_modules into SQLite database?
  - Cupprum 2y ago
    Honestly, maybe? Would be a fun project to look at at least.
- snazz 2y ago
  It’s somewhat more complex than “NTFS is slow”. Here’s a good explanation: https://github.com/Microsoft/WSL/issues/873#issuecomment-425272829 https://github.com/Microsoft/WSL/issues/873#issuecomment-425... I’ve benchmarked deleting files (around ~65,000 small node_modules sort of files) and it takes 40 seconds through Explorer, 20 seconds with rd, and roughly a second inside WSL2 (cloned to the VM’s ext4 virtual hard drive).
- polski-g 2y ago
  Why not use https://github.com/bobranten/Ext4Fsd https://github.com/bobranten/Ext4Fsd ??
- nullindividual 2y ago
  NTFS is perfectly fine at handling small files and performs on-par with other modern file systems. The issue is Defender in sync mode/other AV/other file system filters. DevDrive as noted by default uses an async scanning technique as well as ReFS. ReFS will suffer the exact same performance issues with Defender (or other AV/other file system filters) doing its thing when running in sync mode, which it does by default for ReFS-formatted drives in Windows Server. https://gregoryszorc.com/blog/2021/04/06/surprisingly-slow/ https://gregoryszorc.com/blog/2021/04/06/surprisingly-slow/ https://news.ycombinator.com/item?id=26737521 https://news.ycombinator.com/item?id=26737521 > Except for CloseHandle(). These calls were often taking 1-10+ milliseconds to complete. > While I didn't realize it at the time, the cause for this was/is Windows Defender. Windows Defender (and other anti-virus / scanning software) typically work on Windows by installing what's called a filesystem filter driver. This doesn't take away from your point that _it is slow_, but the reasons are not due to the file system in use.
  - Aerroon 2y ago
    >The issue is Defender in sync mode/other AV/other file system filters. I've had folders take a full minute to open on an SSD. It got to the point where I went to open the folder, it started loading. I needed the file quickly, so I searched for it online, found it, and opened it before windows finished loading that folder for me. After exempting that folder from Windows Defender the folder loads instantly. For the life of me I cannot understand why Defender blocks Explorer.
    - nullindividual 2y ago
      > For the life of me I cannot understand why Defender blocks Explorer. I suppose if you wanted to find out, you could use dtrace/ETW. Explorer has other things going on, though, including other apps that hook into it (shell extensions, like Adobe Reader, TortiseGit/SVN, and so on) which can certainly cause performance issues.
    - layer8 2y ago
      Probably because Explorer hosts shell hooks which can potentially execute arbitrary code. Just one example: File icons or thumbnails can be dynamically generated by shell extensions based on the file contents. A maliciously crafted file could potentially exploit a vulnerability in such a shell extension.
deleted 2y ago
[deleted]
chefandy 2y ago
Their "windows dev drive" setup addresses this. I haven't tested it myself but I saw a couple of inexpertly executed tests showing significant performance gains. I honestly have no idea if my compile times are quicker.
- nullindividual 2y ago
  You can get benches from the horses mouth. https://devblogs.microsoft.com/visualstudio/devdrive/ https://devblogs.microsoft.com/visualstudio/devdrive/ But the primary improvement comes from async A/V, not the file system. ReFS offers other improvements such as CoW and instant file initialization which can benefit developers. From this standpoint, it is the correct choice over NTFS. https://devblogs.microsoft.com/engineering-at-microsoft/dev-drive-and-copy-on-write-for-developer-performance/ https://devblogs.microsoft.com/engineering-at-microsoft/dev-...

nextaccountic 2y ago

But is it faster than accessing the filesystem with io_uring as well? I feel like this article should be updated

nullindividual 2y ago
io_uring/IOCP won't change how long it takes to access a file.
- nextaccountic 2y ago
  So what's exactly the performance benefits of using io_uring to access files?
  - nullindividual 2y ago
    It allows you to do something else while waiting for the request to complete. Linux has historically bad hacks to enable this. Windows NT has had IOCP since it's inception, but not only for storage devices, as io_uring is limited to, but networking, TCP sockets, mail slots, named pipes, etc. "Are you done yet?" vs "Get back to me when you're ready, gonna go do something else". Aka blocking i/o vs nonblocking i/o. (I always wonder why Windows NT doesn't get more high performance implementations for networking et. al. with this feature; licensing? trade off in perf elsewhere? can't tweak kernel params to your heart's desire?) io_uring implemented an additional feature of a ring buffer, but Microsoft has followed this feature; I think it was first introduced in a later build of Windows 10 and should be in 11 & 2022. Microsoft has a graphical example of IOCP: https://learn.microsoft.com/en-us/windows/win32/fileio/synchronous-and-asynchronous-i-o https://learn.microsoft.com/en-us/windows/win32/fileio/synch... More info on a comparison of ring implementations in Windows/Linux: https://windows-internals.com/ioring-vs-io_uring-a-comparison-of-windows-and-linux-implementations/ https://windows-internals.com/ioring-vs-io_uring-a-compariso...

alexhornby 2y ago

https://news.ycombinator.com/item?id=18783525 https://news.ycombinator.com/item?id=18783525 has previous discussion on why windows filesystem is slow

OttoCoddo 2y ago

With the right code, NTFS is not much slower than ext4, for example. Nearly 3% for tar.gz and more than 30% for a heavily multi-threaded use like Pack. https://forum.lazarus.freepascal.org/index.php/topic,66281.msg507111.html#msg507111 https://forum.lazarus.freepascal.org/index.php/topic,66281.m...

deleted 2y ago

[deleted]