4 ms·

NTFS is perfectly fine at handling small files and performs on-par with other modern file systems. The issue is Defender in sync mode/other AV/other file syste

NTFS is perfectly fine at handling small files and performs on-par with other modern file systems. The issue is Defender in sync mode/other AV/other file system filters. DevDrive as noted by default uses an async scanning technique as well as ReFS. ReFS will suffer the exact same performance issues with Defender (or other AV/other file system filters) doing its thing when running in sync mode, which it does by default for ReFS-formatted drives in Windows Server. https://gregoryszorc.com/blog/2021/04/06/surprisingly-slow/ https://gregoryszorc.com/blog/2021/04/06/surprisingly-slow/ https://news.ycombinator.com/item?id=26737521 https://news.ycombinator.com/item?id=26737521 > Except for CloseHandle(). These calls were often taking 1-10+ milliseconds to complete. > While I didn't realize it at the time, the cause for this was/is Windows Defender. Windows Defender (and other anti-virus / scanning software) typically work on Windows by installing what's called a filesystem filter driver. This doesn't take away from your point that _it is slow_, but the reasons are not due to the file system in use.

Aerroon 2y ago
>The issue is Defender in sync mode/other AV/other file system filters. I've had folders take a full minute to open on an SSD. It got to the point where I went to open the folder, it started loading. I needed the file quickly, so I searched for it online, found it, and opened it before windows finished loading that folder for me. After exempting that folder from Windows Defender the folder loads instantly. For the life of me I cannot understand why Defender blocks Explorer.
- nullindividual 2y ago
  > For the life of me I cannot understand why Defender blocks Explorer. I suppose if you wanted to find out, you could use dtrace/ETW. Explorer has other things going on, though, including other apps that hook into it (shell extensions, like Adobe Reader, TortiseGit/SVN, and so on) which can certainly cause performance issues.
- layer8 2y ago
  Probably because Explorer hosts shell hooks which can potentially execute arbitrary code. Just one example: File icons or thumbnails can be dynamically generated by shell extensions based on the file contents. A maliciously crafted file could potentially exploit a vulnerability in such a shell extension.
ffsm8 2y ago
Technically, they're because of the filesystem in use: it's providing the APIs these garbage-ware utilize... which causes the performance issues ( ◠ ‿ ・ ) —
- nullindividual 2y ago
  File system filter drives apply to all (RW) file systems on Windows. It's not exclusive to NTFS or ReFS. Windows has an extensible model. It's a different approach from most (all?) other OSes. It offers a different set of features. Sure, AV could perhaps be done in a different manner that would be more effective/faster, I can't comment on that as I lack the insight required -- only MSFTies that work on kernel code could respond in any authoritative way.
  - ffsm8 2y ago
    I was of the understanding that these sync APIs are only available on Windows filesystems, so a fat32 formatted filesystem wouldn't suffer the same performance impact, which is why windows provides "virtual drives" for performance on their cloud instances that give you extra performance... Precisely by formatting them with a filesystem that doesn't support these sync/blocking Apis. But I'm not particularly knowledgeable either on this topic, just a (forced) consumer of the operating system with the occasional reading on the side
    - nullindividual 2y ago
      Azure Temporary Storage disks are stored on the host hypervisor machine. Non-temp storage is stored in Azure Storage. That's the only difference. FAT[32] does implement minifilters. https://learn.microsoft.com/en-us/windows-hardware/drivers/ifs/file-system-support-for-contexts https://learn.microsoft.com/en-us/windows-hardware/drivers/i... https://www.osr.com/nt-insider/2019-issue1/the-state-of-windows-file-system-filtering-in-2019/ https://www.osr.com/nt-insider/2019-issue1/the-state-of-wind...
      - ffsm8 2y ago
marwis 2y ago
Doesn't it (Windows VFS layer) also lack the equivalent of dentry cache, making all metadata lookups slow?
- nullindividual 2y ago
  The MFT describes resident (in-MFT) metadata about files, or non-resident (external to the MFT, but the MFT has a pointer) if the metadata does not fit within the MFT. The MFT itself is a database; Windows makes use of file system caching, be it in-memory or another caching mechanism, like on-device disk cache.
  - marwis 2y ago
    So why is it still pretty slow for file operations on DevDrive (no filters, async antivirus)? I found the performance improvement not that significant.
    - nullindividual 2y ago
      Doesn't match what I found.
michaelcampbell 2y ago
I don't know if async, AV, NTFS, or what, but some years ago we could demonstrate reliably that running a "big" maven compile on a lot of .java files was faster in Linux than Windows. The kicker... Linux was running inside a VirtualBox VM inside the very same Windows host. This could also be some variance in the `javac` command between OS's, granted.

Aerroon 2y ago

>The issue is Defender in sync mode/other AV/other file system filters. I've had folders take a full minute to open on an SSD. It got to the point where I went to open the folder, it started loading. I needed the file quickly, so I searched for it online, found it, and opened it before windows finished loading that folder for me. After exempting that folder from Windows Defender the folder loads instantly. For the life of me I cannot understand why Defender blocks Explorer.

nullindividual 2y ago
> For the life of me I cannot understand why Defender blocks Explorer. I suppose if you wanted to find out, you could use dtrace/ETW. Explorer has other things going on, though, including other apps that hook into it (shell extensions, like Adobe Reader, TortiseGit/SVN, and so on) which can certainly cause performance issues.
layer8 2y ago
Probably because Explorer hosts shell hooks which can potentially execute arbitrary code. Just one example: File icons or thumbnails can be dynamically generated by shell extensions based on the file contents. A maliciously crafted file could potentially exploit a vulnerability in such a shell extension.

ffsm8 2y ago

Technically, they're because of the filesystem in use: it's providing the APIs these garbage-ware utilize... which causes the performance issues ( ◠ ‿ ・ ) —

nullindividual 2y ago
File system filter drives apply to all (RW) file systems on Windows. It's not exclusive to NTFS or ReFS. Windows has an extensible model. It's a different approach from most (all?) other OSes. It offers a different set of features. Sure, AV could perhaps be done in a different manner that would be more effective/faster, I can't comment on that as I lack the insight required -- only MSFTies that work on kernel code could respond in any authoritative way.
- ffsm8 2y ago
  I was of the understanding that these sync APIs are only available on Windows filesystems, so a fat32 formatted filesystem wouldn't suffer the same performance impact, which is why windows provides "virtual drives" for performance on their cloud instances that give you extra performance... Precisely by formatting them with a filesystem that doesn't support these sync/blocking Apis. But I'm not particularly knowledgeable either on this topic, just a (forced) consumer of the operating system with the occasional reading on the side
  - nullindividual 2y ago
    Azure Temporary Storage disks are stored on the host hypervisor machine. Non-temp storage is stored in Azure Storage. That's the only difference. FAT[32] does implement minifilters. https://learn.microsoft.com/en-us/windows-hardware/drivers/ifs/file-system-support-for-contexts https://learn.microsoft.com/en-us/windows-hardware/drivers/i... https://www.osr.com/nt-insider/2019-issue1/the-state-of-windows-file-system-filtering-in-2019/ https://www.osr.com/nt-insider/2019-issue1/the-state-of-wind...
    - ffsm8 2y ago

marwis 2y ago

Doesn't it (Windows VFS layer) also lack the equivalent of dentry cache, making all metadata lookups slow?

nullindividual 2y ago
The MFT describes resident (in-MFT) metadata about files, or non-resident (external to the MFT, but the MFT has a pointer) if the metadata does not fit within the MFT. The MFT itself is a database; Windows makes use of file system caching, be it in-memory or another caching mechanism, like on-device disk cache.
- marwis 2y ago
  So why is it still pretty slow for file operations on DevDrive (no filters, async antivirus)? I found the performance improvement not that significant.
  - nullindividual 2y ago
    Doesn't match what I found.

michaelcampbell 2y ago

I don't know if async, AV, NTFS, or what, but some years ago we could demonstrate reliably that running a "big" maven compile on a lot of .java files was faster in Linux than Windows. The kicker... Linux was running inside a VirtualBox VM inside the very same Windows host. This could also be some variance in the `javac` command between OS's, granted.