5 ms·

Is it a known issue that the filesystem on Windows 10 is so slow? Being 5 times slower than macOS was roughly my experience but I thought there was just somethi

Is it a known issue that the filesystem on Windows 10 is so slow? Being 5 times slower than macOS was roughly my experience but I thought there was just something wrong with my Windows laptop. I can't find any benchmark or explanation about this.

deleted 5y ago
[deleted]
shawnz 5y ago
See here for some notes from the WSL team why certain filesystem operations that would be fast on Linux are slow on Windows: https://github.com/microsoft/WSL/issues/873#issuecomment-391810696 https://github.com/microsoft/WSL/issues/873#issuecomment-391... https://github.com/microsoft/WSL/issues/873#issuecomment-425272829 https://github.com/microsoft/WSL/issues/873#issuecomment-425...
- nivenhuh 5y ago
  Thanks for the links -- one pro-tip that stood out to me was to use the D: drive (because it's likely to have less filter drivers attached). "Windows's IO stack is extensible, allowing filter drivers to attach to volumes and intercept IO requests before the file system sees them. This is used for numerous things, including virus scanning, compression, encryption, file virtualization, things like OneDrive's files on demand feature, gathering pre-fetching data to speed up app startup, and much more. Even a clean install of Windows will have a number of filters present, particularly on the system volume (so if you have a D: drive or partition, I recommend using that instead, since it likely has fewer filters attached). Filters are involved in many IO operations, most notably creating/opening files."
- hughrr 5y ago
  I'm going to have to complain about this because in real world use these are far less of a problem than MFT contention on sub-900 byte files generated by typical unix environments. Particularly things that are lockfile heavy and VCSs etc are the most painful. The WSL thing reads like an excuse here. If you actually go and look at what's happening it's small files which are the damage multiplier. I think the real issue is that you shouldn't mix the two operating system paradigms and it's far better to just run Linux in a VM and benefit from the near native performance at the cost of a tiny bit of inconvenience. It's not a bad option when you consider the remote IDE capabilities that VScode gives you, which is the one product they're doing 100% right.
  - deleted 5y ago
    [deleted]
  - rossy 5y ago
    I'm sorry, but I find that hard to believe when the linked posts are from a Microsoft employee, who says repeatedly that the problem isn't a simple as "NTFS is slow," that they spent a release optimising NTFS, they've gotten rid of all the low hanging fruit and so on. I don't see why they'd be making excuses for NTFS when the underlying problems seem to be much more fundamental to Windows, which is much worse. You say you discovered the MFT contention issue when you were working on Subversion a decade ago, but these posts are from 2018, so I'm inclined to believe that the MFT contention issue has been mitigated and the real issues really are things like filter drivers, like they say. I'm also not sure there's two paradigms here, if by that you mean there are workloads more suited to Linux and workloads more suited to Windows. Are there any file system operations that are actually faster on Windows? I'm still inclined to believe, like they say, that "file operations in Windows are more expensive than in Linux," even if large files are less impacted than small files. I really hope Microsoft can improve this, rather than just throwing it under the rug and saying to use Linux VMs, even if they have to make drastic changes like deprecating filesystem filter drivers completely. I think a lot of us depend on the performance of software that was designed for fast filesystems and ported to Windows (not just Git, either.) I also think a lot of tasks fundamentally use a lot of small files. After all, what is source code?
    - hughrr 5y ago
      No it’s still MFT contention. Someone at my company independently discovered the same problem recently with git on WSL. MSFT don’t listen to or care about their customers findings from experience. That’s why canned our partner status. In this case they haven’t touched NTFS at all - that’s out right rubbish. I pointed out the paradigm mismatch already. Run unix loads on unix not NT.
jjoonathan 5y ago
It isn't just Windows 10, as far as I know every version of windows has always wanted to spend like a minute (or multiple minutes!) slowly counting the files before it proceeds to slowly delete them one by one... until it finds one that's open in a background process and nixes the whole operation in a half complete state. 5 times slower sounds conservative. It's awful. Hey, maybe they could put rounded corners on the file deletion window, I bet that would fix it!
- metalliqaz 5y ago
  it does that for the UI For power users, you're much better off using the command line tools
  - jjoonathan 5y ago
    I have a different power user workaround: I don't use Windows if I can at all avoid it. EDIT: Does the CLI override the "open in background" check? Or do you still need Mark Russinovich sysinternals to reliably perform elementary file system administration tasks in Windows?
- tyingq 5y ago
  I think that's warts in the file explorer more so than warts in the system itself.
  - logifail 5y ago
    > warts in the file explorer more so than warts in the system itself I'm struggling to understand why one shouldn't treat the behaviour of explorer.exe as simply part of the Windows system. Are there third-party explorer.exe alternatives available? If so, during Windows install, where do I click to _not_ install the MSFT explorer.exe and instead select one of the better alternatives?
    - edoceo 5y ago
      The last time I really used Windows (Win2k) there were alternative WinShell options. You'd set a regkey to your program. There used to be a 3.1 style Program Manager - I miss that one
Isthatablackgsd 5y ago
Likely it is Windows Defender. Someone recently post the issue tracker for this particular issue and some are pointing at Defender as the cause of the I/O performance. As a daily windows user, I'm not surprised if it is Defender. I have some files and folders in exclusion list because Defender was interfering with those files that the software is trying to use.
- naikrovek 5y ago
  It's not Windows Defender. Not entirely, anyway. NTFS is slow when reading or writing lots of small files. It just is, and it always has been, and it's because of how NTFS was designed, if I am remembering correctly.
- boardwaalk 5y ago
  That’s probably part of it, but NTFS has just been slow forever and ever. Anything involving lots of small files is just pain. Perhaps it seems “normal” for anyone used to Explorer progress dialogs, but it’s not. And Microsoft just doesn’t seem to care :-(.
  - naikrovek 5y ago
    They care, as much as a group of people without a mind of its own can care, but it's a use case that was not common among windows software when NTFS was designed. Other design considerations were more important than high performance while writing or reading 1 million small files per minute.
modeless 5y ago
If you're doing a bunch of I/O, it's often worth it to go into Windows Defender settings and turn off "real time protection" temporarily. It can make a big difference.
- dexterhaslem 5y ago
  absolutely, and for directories for your projects /places where lots of files thrash about, add those dirs as exceptions in the options. i mean i added c:\ as an exception because i do not need realtime av but i dont think it will actually work like that
Someone1234 5y ago
Windows' slow performance when dealing with millions of tiny files per hour (e.g. <50 KB) and inability to offer fine-grain IO throttling for processes was why we moved from Windows to Linux in a previous career. Linux handles this scenario in its stride, Windows chokes (and, no, "background" IO priority on Windows isn't a solution). Simply moving from files to a DBMS wasn't an option, since the whole point of this edge system was to allow third parties to FTP/SFTP/FTPS/etc in and out, to put/get files. We would have had to re-implement multiple transfer technologies/protocols to have a file-less database system (which didn't make cost-effective sense).
jacobwilliamroy 5y ago
My theory is that the OS likes to do a deep inspection of your files in order to feed microsoft's machine learning models.
- naikrovek 5y ago
  your theory is based on ignorance and suspicion instead of anything real.
  - jacobwilliamroy 5y ago
    And what is your's based on? It's proprietary software. We're not supposed to know how it works or what it's doing.
naikrovek 5y ago
Yes. NTFS filesystem performance when accessing many small files rapidly is slow. It's an NTFS limitation, not any specific version of Windows. I believe it is caused by some per-file overhead that NTFS incurs, though I'm not 100% sure.
- masklinn 5y ago
  > It's an NTFS limitation, not any specific version of Windows. I would guess their surprise is because the charts specifically show Win10 as being 100% slower than Win7 in the first chart, and still a good 30% slower in the best case (third chart). That's not NTFS v other, that's Win7 v Win10, both on NTFS.
  - naikrovek 5y ago
    NTFS comes in different versions, you know. I don't care what version of Windows you're on, or what version of NTFS you're on; NTFS is dog slow on small files. I am not sure that NTFS has versioned between Windows 7 and Windows 10, so let's just assume that it hasn't. If the limitations of the filesystem do not change between operating systems, then what must change is how the OS deals with the filesystem limitation, and/or attempts to work around them. Windows 7 does better than Windows 10. That doesn't mean NTFS isn't slow.
    - masklinn 5y ago
      > NTFS comes in different versions, you know. A completely pointless remark when versions of NTFS and Windows are correlated. And AFAIK there's been no updated to NTFS itself since the addition of symlinks in NTFS 3.1, released with Windows XP. All "filesystem features" since were added at the level OS, using existing NTFS features. > If the limitations of the filesystem do not change between operating systems, then what must change is how the OS deals with the filesystem limitation, and/or attempts to work around them. Yeees? Which would make the issue vary based on the specific version of Windows maybe? > Windows 7 does better than Windows 10. That doesn't mean NTFS isn't slow. No, but I never claimed such a thing. I'm just pointing out that your assertion that > It's […] not any specific version of Windows. is bullshit.
mqus 5y ago
I think this was known for a long time. E.g. Git takes forever, compared to the equivalent operations in Linux
hughrr 5y ago
This is actually a problem with the design of NTFS. Small files are stored in the MFT (Master File Table) which results in heavy lock contention and slowness if you're doing lots of small files which is what most people tend to do these days. NTFS really does quite well on large files but small Unixy things and source code it is hopeless on. I first learned about this trying to make Subversion fast about a decade ago. You can tune some of this out with fsutil but it gives marginal gains. It's depressing though when you have a VM on windows that is faster than the native OS filesystem :(
outworlder 5y ago
Known issue? Yes. One of the reasons many games have faster load times under Wine when compared with the actual OS. No clue on what the actual underlying issue is. And it's not like we can easily switch filesystems to compare.
passivate 5y ago
Personally, I have not seen that. Which operations did you think were slow? Small-block/large-block?, sequential/random I/O ? etc, etc. Or did you just mean general productivity?
- animatedb 5y ago
  I know that the open function on files is very slow. This makes opening many small files very slow.
jokoon 5y ago
One thing I know, is that mechanical disk drives have suffered a very big drop in performance in windows 10. I've asked this question several times, "Why would an OS need an SSD to work normally and be responsive when it was already fast without SSD some years ago", and I often get responses like I don't know how computers work, that win10 takes advantages of SSD and that it cannot work well on old techs. Even when you have a high amount of RAM and use a HDD, it's still slow. I'm still looking for a decent filesystem or kernel developer to answer that question with a good enough explanation, because to me, I'm either stupid, or there is negligence or some attempt to increase sales of SSD. Maybe windows now uses SSD as some sort of "secondary RAM", and thus it will never be fast with a HDD. I mean, linux desktops seem to still be fast enough with mechanical HDD.
qwertox 5y ago
Alone the fact that it is twice as slow as Windows 7 is puzzling. One would assume that it gets better with every upgrade, but a factor of 2x worse seems to indicate that they added some kind of logging or something. Like the logging/access system which Android has started to integrate with Android 9 where every access needs to go through the StorageManager.
phepranto 5y ago
At the end of the article it says: "SQLite is much faster than direct writes to disk on Windows when anti-virus protection is turned on. Since anti-virus software is and should be on by default in Windows, that means that SQLite is generally much faster than direct disk writes on Windows. "
- wscott 5y ago
  This is a key point. Windows is starting up the anti-virus program and scanning every tiny file as it is written. This is less about NTFS being slow and more about anti-virus being considered a required overhead on Windows.

deleted 5y ago

[deleted]

shawnz 5y ago

See here for some notes from the WSL team why certain filesystem operations that would be fast on Linux are slow on Windows: https://github.com/microsoft/WSL/issues/873#issuecomment-391810696 https://github.com/microsoft/WSL/issues/873#issuecomment-391... https://github.com/microsoft/WSL/issues/873#issuecomment-425272829 https://github.com/microsoft/WSL/issues/873#issuecomment-425...

nivenhuh 5y ago
Thanks for the links -- one pro-tip that stood out to me was to use the D: drive (because it's likely to have less filter drivers attached). "Windows's IO stack is extensible, allowing filter drivers to attach to volumes and intercept IO requests before the file system sees them. This is used for numerous things, including virus scanning, compression, encryption, file virtualization, things like OneDrive's files on demand feature, gathering pre-fetching data to speed up app startup, and much more. Even a clean install of Windows will have a number of filters present, particularly on the system volume (so if you have a D: drive or partition, I recommend using that instead, since it likely has fewer filters attached). Filters are involved in many IO operations, most notably creating/opening files."
hughrr 5y ago
I'm going to have to complain about this because in real world use these are far less of a problem than MFT contention on sub-900 byte files generated by typical unix environments. Particularly things that are lockfile heavy and VCSs etc are the most painful. The WSL thing reads like an excuse here. If you actually go and look at what's happening it's small files which are the damage multiplier. I think the real issue is that you shouldn't mix the two operating system paradigms and it's far better to just run Linux in a VM and benefit from the near native performance at the cost of a tiny bit of inconvenience. It's not a bad option when you consider the remote IDE capabilities that VScode gives you, which is the one product they're doing 100% right.
- deleted 5y ago
  [deleted]
- rossy 5y ago
  I'm sorry, but I find that hard to believe when the linked posts are from a Microsoft employee, who says repeatedly that the problem isn't a simple as "NTFS is slow," that they spent a release optimising NTFS, they've gotten rid of all the low hanging fruit and so on. I don't see why they'd be making excuses for NTFS when the underlying problems seem to be much more fundamental to Windows, which is much worse. You say you discovered the MFT contention issue when you were working on Subversion a decade ago, but these posts are from 2018, so I'm inclined to believe that the MFT contention issue has been mitigated and the real issues really are things like filter drivers, like they say. I'm also not sure there's two paradigms here, if by that you mean there are workloads more suited to Linux and workloads more suited to Windows. Are there any file system operations that are actually faster on Windows? I'm still inclined to believe, like they say, that "file operations in Windows are more expensive than in Linux," even if large files are less impacted than small files. I really hope Microsoft can improve this, rather than just throwing it under the rug and saying to use Linux VMs, even if they have to make drastic changes like deprecating filesystem filter drivers completely. I think a lot of us depend on the performance of software that was designed for fast filesystems and ported to Windows (not just Git, either.) I also think a lot of tasks fundamentally use a lot of small files. After all, what is source code?
  - hughrr 5y ago
    No it’s still MFT contention. Someone at my company independently discovered the same problem recently with git on WSL. MSFT don’t listen to or care about their customers findings from experience. That’s why canned our partner status. In this case they haven’t touched NTFS at all - that’s out right rubbish. I pointed out the paradigm mismatch already. Run unix loads on unix not NT.

jjoonathan 5y ago

It isn't just Windows 10, as far as I know every version of windows has always wanted to spend like a minute (or multiple minutes!) slowly counting the files before it proceeds to slowly delete them one by one... until it finds one that's open in a background process and nixes the whole operation in a half complete state. 5 times slower sounds conservative. It's awful. Hey, maybe they could put rounded corners on the file deletion window, I bet that would fix it!

metalliqaz 5y ago
it does that for the UI For power users, you're much better off using the command line tools
- jjoonathan 5y ago
  I have a different power user workaround: I don't use Windows if I can at all avoid it. EDIT: Does the CLI override the "open in background" check? Or do you still need Mark Russinovich sysinternals to reliably perform elementary file system administration tasks in Windows?
tyingq 5y ago
I think that's warts in the file explorer more so than warts in the system itself.
- logifail 5y ago
  > warts in the file explorer more so than warts in the system itself I'm struggling to understand why one shouldn't treat the behaviour of explorer.exe as simply part of the Windows system. Are there third-party explorer.exe alternatives available? If so, during Windows install, where do I click to _not_ install the MSFT explorer.exe and instead select one of the better alternatives?
  - edoceo 5y ago
    The last time I really used Windows (Win2k) there were alternative WinShell options. You'd set a regkey to your program. There used to be a 3.1 style Program Manager - I miss that one

Isthatablackgsd 5y ago

Likely it is Windows Defender. Someone recently post the issue tracker for this particular issue and some are pointing at Defender as the cause of the I/O performance. As a daily windows user, I'm not surprised if it is Defender. I have some files and folders in exclusion list because Defender was interfering with those files that the software is trying to use.

naikrovek 5y ago
It's not Windows Defender. Not entirely, anyway. NTFS is slow when reading or writing lots of small files. It just is, and it always has been, and it's because of how NTFS was designed, if I am remembering correctly.
boardwaalk 5y ago
That’s probably part of it, but NTFS has just been slow forever and ever. Anything involving lots of small files is just pain. Perhaps it seems “normal” for anyone used to Explorer progress dialogs, but it’s not. And Microsoft just doesn’t seem to care :-(.
- naikrovek 5y ago
  They care, as much as a group of people without a mind of its own can care, but it's a use case that was not common among windows software when NTFS was designed. Other design considerations were more important than high performance while writing or reading 1 million small files per minute.

modeless 5y ago

If you're doing a bunch of I/O, it's often worth it to go into Windows Defender settings and turn off "real time protection" temporarily. It can make a big difference.

dexterhaslem 5y ago
absolutely, and for directories for your projects /places where lots of files thrash about, add those dirs as exceptions in the options. i mean i added c:\ as an exception because i do not need realtime av but i dont think it will actually work like that

Someone1234 5y ago

Windows' slow performance when dealing with millions of tiny files per hour (e.g. <50 KB) and inability to offer fine-grain IO throttling for processes was why we moved from Windows to Linux in a previous career. Linux handles this scenario in its stride, Windows chokes (and, no, "background" IO priority on Windows isn't a solution). Simply moving from files to a DBMS wasn't an option, since the whole point of this edge system was to allow third parties to FTP/SFTP/FTPS/etc in and out, to put/get files. We would have had to re-implement multiple transfer technologies/protocols to have a file-less database system (which didn't make cost-effective sense).

jacobwilliamroy 5y ago

My theory is that the OS likes to do a deep inspection of your files in order to feed microsoft's machine learning models.

naikrovek 5y ago
your theory is based on ignorance and suspicion instead of anything real.
- jacobwilliamroy 5y ago
  And what is your's based on? It's proprietary software. We're not supposed to know how it works or what it's doing.

naikrovek 5y ago

Yes. NTFS filesystem performance when accessing many small files rapidly is slow. It's an NTFS limitation, not any specific version of Windows. I believe it is caused by some per-file overhead that NTFS incurs, though I'm not 100% sure.

masklinn 5y ago
> It's an NTFS limitation, not any specific version of Windows. I would guess their surprise is because the charts specifically show Win10 as being 100% slower than Win7 in the first chart, and still a good 30% slower in the best case (third chart). That's not NTFS v other, that's Win7 v Win10, both on NTFS.
- naikrovek 5y ago
  NTFS comes in different versions, you know. I don't care what version of Windows you're on, or what version of NTFS you're on; NTFS is dog slow on small files. I am not sure that NTFS has versioned between Windows 7 and Windows 10, so let's just assume that it hasn't. If the limitations of the filesystem do not change between operating systems, then what must change is how the OS deals with the filesystem limitation, and/or attempts to work around them. Windows 7 does better than Windows 10. That doesn't mean NTFS isn't slow.
  - masklinn 5y ago
    > NTFS comes in different versions, you know. A completely pointless remark when versions of NTFS and Windows are correlated. And AFAIK there's been no updated to NTFS itself since the addition of symlinks in NTFS 3.1, released with Windows XP. All "filesystem features" since were added at the level OS, using existing NTFS features. > If the limitations of the filesystem do not change between operating systems, then what must change is how the OS deals with the filesystem limitation, and/or attempts to work around them. Yeees? Which would make the issue vary based on the specific version of Windows maybe? > Windows 7 does better than Windows 10. That doesn't mean NTFS isn't slow. No, but I never claimed such a thing. I'm just pointing out that your assertion that > It's […] not any specific version of Windows. is bullshit.

mqus 5y ago

I think this was known for a long time. E.g. Git takes forever, compared to the equivalent operations in Linux

hughrr 5y ago

This is actually a problem with the design of NTFS. Small files are stored in the MFT (Master File Table) which results in heavy lock contention and slowness if you're doing lots of small files which is what most people tend to do these days. NTFS really does quite well on large files but small Unixy things and source code it is hopeless on. I first learned about this trying to make Subversion fast about a decade ago. You can tune some of this out with fsutil but it gives marginal gains. It's depressing though when you have a VM on windows that is faster than the native OS filesystem :(

outworlder 5y ago

Known issue? Yes. One of the reasons many games have faster load times under Wine when compared with the actual OS. No clue on what the actual underlying issue is. And it's not like we can easily switch filesystems to compare.

passivate 5y ago

Personally, I have not seen that. Which operations did you think were slow? Small-block/large-block?, sequential/random I/O ? etc, etc. Or did you just mean general productivity?

animatedb 5y ago
I know that the open function on files is very slow. This makes opening many small files very slow.

jokoon 5y ago

One thing I know, is that mechanical disk drives have suffered a very big drop in performance in windows 10. I've asked this question several times, "Why would an OS need an SSD to work normally and be responsive when it was already fast without SSD some years ago", and I often get responses like I don't know how computers work, that win10 takes advantages of SSD and that it cannot work well on old techs. Even when you have a high amount of RAM and use a HDD, it's still slow. I'm still looking for a decent filesystem or kernel developer to answer that question with a good enough explanation, because to me, I'm either stupid, or there is negligence or some attempt to increase sales of SSD. Maybe windows now uses SSD as some sort of "secondary RAM", and thus it will never be fast with a HDD. I mean, linux desktops seem to still be fast enough with mechanical HDD.

qwertox 5y ago

Alone the fact that it is twice as slow as Windows 7 is puzzling. One would assume that it gets better with every upgrade, but a factor of 2x worse seems to indicate that they added some kind of logging or something. Like the logging/access system which Android has started to integrate with Android 9 where every access needs to go through the StorageManager.

phepranto 5y ago

At the end of the article it says: "SQLite is much faster than direct writes to disk on Windows when anti-virus protection is turned on. Since anti-virus software is and should be on by default in Windows, that means that SQLite is generally much faster than direct disk writes on Windows. "

wscott 5y ago
This is a key point. Windows is starting up the anti-virus program and scanning every tiny file as it is written. This is less about NTFS being slow and more about anti-virus being considered a required overhead on Windows.