5 ms·

Really? Filesystem performance on Windows is absolutely horrible compared to Linux. It blows me away how, even using SSDs, the performance of git over large r

by e40 3y ago

Really? Filesystem performance on Windows is absolutely horrible compared to Linux. It blows me away how, even using SSDs, the performance of git over large repos is at least 10x slower than a Linux box with the same CPU. It is the bane of my existence. It's not just git. rsync of a large directory takes >10x as long. This is with Server 2016 (and 2012 r2). My 2012 r2 machine had spinning disks and when I tested on 2016 with SSDs I was shocked the situation did not improve.

adzm 3y ago
This is true. I'm not sure what the obstacle is, and I know a lot of smart people have worked on it, but opening a ton of files and traversing a bunch of them in directories is very slow. Once the file is open, IO is just as fast though.
- amlib 3y ago
  Unless you have antivirus software and god knows what analyzing the files, then IO is actually slower...
- martinald 3y ago
  My guess is ACLs, which are enabled by default on NTFS but not (afiak) on most linux distros using ext4.
  - e40 3y ago
    Extended attributes are definitely turned on in my ext4 filesystems and to my knowledge I didn't do anything to turn them on.
    - martinald 3y ago
      Hmm that's true but I don't think much makes use of them? As in they are not set with anything. On NTFS they are used for everything.
- WorldMaker 3y ago
  Filesystems are a database and have to deal with CAP Theorem trade-offs like everything else. Windows and NTFS both took a heavy focus on Consistency/(lack of) Partitions over Availability. Most POSIX operating systems and their filesystems took a heavy focus on Availability at the expense of Eventual Consistency and Sometimes Partitions. Neither approach is wrong, they are just very different approaches with very different performance trade-offs. Also Windows' filesystem supports an entire plugin stack including user-space plugins, to support things like anti-virus scanners and virtual filesystems and all sorts of other things. Not all of Windows' "slow" filesystem is first-party problems, a lot of it can be third-party drivers and tools that are installed.
  - blibble 3y ago
    a local filesystem that can only be mounted once and of which state is 100% controlled by one entity (the local kernel) is not a distributed system the CAP theorem simply does not apply Windows IO subsystem was simply designed for extensibility over performance
    - WorldMaker 3y ago
      A) It's a useful analogy whether or not you think it technically applies or is a perfect analogy. B) The Windows filesystem (and to an extent the POSIX) isn't just "local", it also includes transparent and semi-transparent network file storage. C) Windows and POSIX are both multi-user and multi-process. They operate over multiple cores and multiple I/O buses. Even if it just one system API centralized in charge of all that, it still needs to be built on top of complex distributed dance of mutexes/locks/semaphores/other distributed control structures. Because of the nature of I/O control there are complex caches involved and transaction semantics of when data is actually pulled from/flushed to low level data stores. The transaction model in turn is reflected in what the files look like to other users or processes running at the same time. Windows and NTFS combined have strong transaction guarantees that other users and processes must see a highly consistent view of the same files. It makes heavy uses of locks by default even for cached data. POSIX favors the "inode" approach that favors high availability and fewer locks at the cost of eventual consistency and the occasional partition (the "same" file can and will sometimes have multiple "inodes" between different processes/users, many common Linux tools rely heavily on that). Those two different transaction models are most easily explained in analogy to CAP Theorem. They are very different transaction models with different trade-offs. Whether or not you see a single "local machine" or you see a distributed system of cores, processes, users, diverse I/O buses is partly a matter of perspective and will reflect how well you personally think CAP Theorem is a "perfect" analogy for a filesystem.
pjmlp 3y ago
Sadly a consequence of how NTFS plugin architecture works. Using Windows 11 the answer is ReFS.
- pxc 3y ago
  Do you still have to pay for a Workstation license to get that on your local machine?
  - pjmlp 3y ago
    It is coming to regular Windows 11 as well. https://learn.microsoft.com/en-us/windows/dev-drive/ https://learn.microsoft.com/en-us/windows/dev-drive/
FreshStart 3y ago
Did they not try to fix that user attention gets resources Bug called a scheduler?
jiggawatts 3y ago
This is mostly caused by the anti-malware scanning. Exclude your sources folder from real-time scanning.
- sedatk 3y ago
  Windows 11 is soon getting a new feature called Dev Drive, which uses an entirely different FS (ReFS) and reduced malware scanning, specifically designed for source code.
  - jiggawatts 3y ago
    90% of the speed up is from not scanning the files. ReFS isn’t much faster than NTFS. The “drive” part of dev drive is a clever hack to bypass corporate anti malware policy settings.
    - sedatk 3y ago
      It's not "not scanning" completely, it has less aggressive scanning by default. You can turn it off fully if you want, but then, you can do that with folder based exclusions on NTFS folders too. So, I'm not sure the perf increase can just be attributed to malware scanning. ReFS has some features that NTFS doesn't have like copy-on-write which might help in read-only I/O perf.

by e40 3y ago

adzm 3y ago

This is true. I'm not sure what the obstacle is, and I know a lot of smart people have worked on it, but opening a ton of files and traversing a bunch of them in directories is very slow. Once the file is open, IO is just as fast though.

amlib 3y ago
Unless you have antivirus software and god knows what analyzing the files, then IO is actually slower...
martinald 3y ago
My guess is ACLs, which are enabled by default on NTFS but not (afiak) on most linux distros using ext4.
- e40 3y ago
  Extended attributes are definitely turned on in my ext4 filesystems and to my knowledge I didn't do anything to turn them on.
  - martinald 3y ago
    Hmm that's true but I don't think much makes use of them? As in they are not set with anything. On NTFS they are used for everything.
WorldMaker 3y ago
Filesystems are a database and have to deal with CAP Theorem trade-offs like everything else. Windows and NTFS both took a heavy focus on Consistency/(lack of) Partitions over Availability. Most POSIX operating systems and their filesystems took a heavy focus on Availability at the expense of Eventual Consistency and Sometimes Partitions. Neither approach is wrong, they are just very different approaches with very different performance trade-offs. Also Windows' filesystem supports an entire plugin stack including user-space plugins, to support things like anti-virus scanners and virtual filesystems and all sorts of other things. Not all of Windows' "slow" filesystem is first-party problems, a lot of it can be third-party drivers and tools that are installed.
- blibble 3y ago
  a local filesystem that can only be mounted once and of which state is 100% controlled by one entity (the local kernel) is not a distributed system the CAP theorem simply does not apply Windows IO subsystem was simply designed for extensibility over performance
  - WorldMaker 3y ago
    A) It's a useful analogy whether or not you think it technically applies or is a perfect analogy. B) The Windows filesystem (and to an extent the POSIX) isn't just "local", it also includes transparent and semi-transparent network file storage. C) Windows and POSIX are both multi-user and multi-process. They operate over multiple cores and multiple I/O buses. Even if it just one system API centralized in charge of all that, it still needs to be built on top of complex distributed dance of mutexes/locks/semaphores/other distributed control structures. Because of the nature of I/O control there are complex caches involved and transaction semantics of when data is actually pulled from/flushed to low level data stores. The transaction model in turn is reflected in what the files look like to other users or processes running at the same time. Windows and NTFS combined have strong transaction guarantees that other users and processes must see a highly consistent view of the same files. It makes heavy uses of locks by default even for cached data. POSIX favors the "inode" approach that favors high availability and fewer locks at the cost of eventual consistency and the occasional partition (the "same" file can and will sometimes have multiple "inodes" between different processes/users, many common Linux tools rely heavily on that). Those two different transaction models are most easily explained in analogy to CAP Theorem. They are very different transaction models with different trade-offs. Whether or not you see a single "local machine" or you see a distributed system of cores, processes, users, diverse I/O buses is partly a matter of perspective and will reflect how well you personally think CAP Theorem is a "perfect" analogy for a filesystem.

pjmlp 3y ago

Sadly a consequence of how NTFS plugin architecture works. Using Windows 11 the answer is ReFS.

pxc 3y ago
Do you still have to pay for a Workstation license to get that on your local machine?
- pjmlp 3y ago
  It is coming to regular Windows 11 as well. https://learn.microsoft.com/en-us/windows/dev-drive/ https://learn.microsoft.com/en-us/windows/dev-drive/

FreshStart 3y ago

Did they not try to fix that user attention gets resources Bug called a scheduler?

jiggawatts 3y ago

This is mostly caused by the anti-malware scanning. Exclude your sources folder from real-time scanning.

sedatk 3y ago
Windows 11 is soon getting a new feature called Dev Drive, which uses an entirely different FS (ReFS) and reduced malware scanning, specifically designed for source code.
- jiggawatts 3y ago
  90% of the speed up is from not scanning the files. ReFS isn’t much faster than NTFS. The “drive” part of dev drive is a clever hack to bypass corporate anti malware policy settings.
  - sedatk 3y ago
    It's not "not scanning" completely, it has less aggressive scanning by default. You can turn it off fully if you want, but then, you can do that with folder based exclusions on NTFS folders too. So, I'm not sure the perf increase can just be attributed to malware scanning. ReFS has some features that NTFS doesn't have like copy-on-write which might help in read-only I/O perf.