4 ms·

> Kernel and low level stuff are actually very stable and good. In their intended applications, which might or might not be the ones you need. The slowness of

> Kernel and low level stuff are actually very stable and good. In their intended applications, which might or might not be the ones you need. The slowness of the filesystem that necessitated a whole custom caching layer in Git for Windows, or the slowness of process creation that necessitated adding “picoprocesses” to the kernel so that WSL1 would perform acceptably and still wasn’t enough for it to survive, those are entirely due to the kernel’s archtecture. It’s not necessarily a huge deal that NT makes a bad substrate for Unix, even if POSIX support has been in the product requirements since before Win32 was conceived. I agree with the MSR paper[1] on fork(), for instance. But for a Unix-head, the “good” in your statement comes with important caveats. The filesystem is in particular so slow that Windows users will unironically claim that Ripgrep is slow and build their own NTFS parsers to sell as the fix[2]. [1] https://lwn.net/Articles/785430/ https://lwn.net/Articles/785430/ [2] https://nitter.net/CharlieMQV/status/1972647630653227054 https://nitter.net/CharlieMQV/status/1972647630653227054

jph00 9mo ago
This is not due to slowness of the file system. Native ntfs tools are much faster than Unix ones in some situations. The issue is that running Unix software on windows will naturally have a performance impact. You see the same thing in reverse using Wine on Linux. Windows uses a different design for IO so requires software to be written with that design in mind.
- noumenon1111 9mo ago
  [flagged]
- MadnessASAP 9mo ago
  It's pretty typical these days for software, particularly games of the DX9-11 eras to perform better on Wine/Proton then they do under native Windows on the same hardware.
  - jph00 9mo ago
    They rarely are IO constrained.
- m132 9mo ago
  > Native ntfs tools are much faster than Unix ones in some situations. The issue is that running Unix software on windows will naturally have a performance impact. You see the same thing in reverse using Wine on Linux. Not true. There are increasingly more cases where Windows software, written with Windows in mind and only tested on Windows, performs better atop Wine. Sure, there are interface incompatibilities that naturally create performance penalties, but a lot of stuff maps 1:1, and Windows was historically designed to support multiple user-space ABIs; Win32 calls are broken down into native kernel calls by kernel32, advapi32, etc., for example, similar to how libc works on Unix-like operating systems.
dgxyz 9mo ago
This is on the mark. But there's another issue which is what cripples windows for dev! NTFS has a terrible design flaw which is the fact that small files, under 640 bytes, are stored in the MFT. The MFT ends up having serious lock contention so lots of small file changes are slow. This screws up anything Unixy and git horribly. WSL1 was built on top of that problem which was one of the many reasons it was slow as molasses. Also why ReFS and "dev drive" exist...
- mananaysiempre 9mo ago
  > NTFS has a terrible design flaw which is the fact that small files, under 640 bytes, are stored in the MFT. Ext4 also stores small (~150B) files inside the inode[1], and so do a number of other filesystems[2]? NTFS was unusually early to the party, but if you’re right that it’s problematic there then something else must also be wrong (perhaps with the locking?) to make it so. [1] https://www.kernel.org/doc/html/latest/filesystems/ext4/inlinedata.html https://www.kernel.org/doc/html/latest/filesystems/ext4/inli... [2] https://en.wikipedia.org/wiki/Comparison_of_file_systems#Allocation_and_layout_policies https://en.wikipedia.org/wiki/Comparison_of_file_systems#All..., the “Inline data” column.
p_ing 9mo ago
The file system isn't slow. The slowness will be present in any file system due to the file system filters that all file system calls pass though.
- mananaysiempre 9mo ago
  Right, by “file system” here I mean all of the layers between the application talking in terms of named files and whatever first starts talking in terms of block addresses. Also, as far as my (very limited) understanding goes, there are more architectural performance problems than just filters (and, to me, filters don’t necessarily sound like performance bankruptcy, provided the filter in question isn’t mandatory, un-removable Microsoft Defender). I seem to remember that path parsing is accomplished in NT by each handler chopping off the initial portion that it understands and passing the remaining suffix to the next one as an uninterpreted string (cf. COM monikers), unlike Unix where the slash-separated list is baked into the architecture, and the former design makes it much harder to have (what Unix calls) a “dentry cache” that would allow the kernel to look up meanings of popular names without going through the filesystem(s).
  - p_ing 9mo ago
    NTFS will perform directory B+-tree lookups (this is where it walks the path) until it finds the requested file. The Cache Manager caches these B+-trees. From there, it hits the MFT, finds the specific record for the file, loads the MFT record, and ultimately returns the FILE_OBJECT to the I/O Manager and it bubbles up the chain back to (presumably) Win32. The MFT is just a linear array of records, which include file and directories (directory records are just a record with directory = true, essentially). Obviously simplified. Windows Internals will be your friend, if you want to know more.
    - mananaysiempre 9mo ago
      Thanks for the explanation! Linux, meanwhile, will[1] in the normal case walk a sequence[2] of hash tables (representing incomplete but up-to-date views of directories) before hitting the filesystem’s vtable or the block I/O layer at all, and on the fast path[3] taking no locks other than the RCU read lock. [1] https://www.kernel.org/doc/html/latest/filesystems/path-lookup.html#more-than-just-a-cache https://www.kernel.org/doc/html/latest/filesystems/path-look... [2] I was under the impression that it could look up an entire path at once when I wrote my grandparent comment; it seems I was wrong, which on reflection makes sense given you can move directories. [3] https://www.kernel.org/doc/html/latest/filesystems/path-lookup.html#rcu-walk-faster-pathname-lookup-in-linux https://www.kernel.org/doc/html/latest/filesystems/path-look...
BrouteMinou 9mo ago
The Windows filesystem isn't slow per se, it's a slowness caused by "a thousand cuts" type of problem. https://github.com/Microsoft/WSL/issues/873#issuecomment-425272829 https://github.com/Microsoft/WSL/issues/873#issuecomment-425...

jph00 9mo ago

This is not due to slowness of the file system. Native ntfs tools are much faster than Unix ones in some situations. The issue is that running Unix software on windows will naturally have a performance impact. You see the same thing in reverse using Wine on Linux. Windows uses a different design for IO so requires software to be written with that design in mind.

noumenon1111 9mo ago
[flagged]
MadnessASAP 9mo ago
It's pretty typical these days for software, particularly games of the DX9-11 eras to perform better on Wine/Proton then they do under native Windows on the same hardware.
- jph00 9mo ago
  They rarely are IO constrained.
m132 9mo ago
> Native ntfs tools are much faster than Unix ones in some situations. The issue is that running Unix software on windows will naturally have a performance impact. You see the same thing in reverse using Wine on Linux. Not true. There are increasingly more cases where Windows software, written with Windows in mind and only tested on Windows, performs better atop Wine. Sure, there are interface incompatibilities that naturally create performance penalties, but a lot of stuff maps 1:1, and Windows was historically designed to support multiple user-space ABIs; Win32 calls are broken down into native kernel calls by kernel32, advapi32, etc., for example, similar to how libc works on Unix-like operating systems.

dgxyz 9mo ago

This is on the mark. But there's another issue which is what cripples windows for dev! NTFS has a terrible design flaw which is the fact that small files, under 640 bytes, are stored in the MFT. The MFT ends up having serious lock contention so lots of small file changes are slow. This screws up anything Unixy and git horribly. WSL1 was built on top of that problem which was one of the many reasons it was slow as molasses. Also why ReFS and "dev drive" exist...

mananaysiempre 9mo ago
> NTFS has a terrible design flaw which is the fact that small files, under 640 bytes, are stored in the MFT. Ext4 also stores small (~150B) files inside the inode[1], and so do a number of other filesystems[2]? NTFS was unusually early to the party, but if you’re right that it’s problematic there then something else must also be wrong (perhaps with the locking?) to make it so. [1] https://www.kernel.org/doc/html/latest/filesystems/ext4/inlinedata.html https://www.kernel.org/doc/html/latest/filesystems/ext4/inli... [2] https://en.wikipedia.org/wiki/Comparison_of_file_systems#Allocation_and_layout_policies https://en.wikipedia.org/wiki/Comparison_of_file_systems#All..., the “Inline data” column.

p_ing 9mo ago

The file system isn't slow. The slowness will be present in any file system due to the file system filters that all file system calls pass though.

mananaysiempre 9mo ago
Right, by “file system” here I mean all of the layers between the application talking in terms of named files and whatever first starts talking in terms of block addresses. Also, as far as my (very limited) understanding goes, there are more architectural performance problems than just filters (and, to me, filters don’t necessarily sound like performance bankruptcy, provided the filter in question isn’t mandatory, un-removable Microsoft Defender). I seem to remember that path parsing is accomplished in NT by each handler chopping off the initial portion that it understands and passing the remaining suffix to the next one as an uninterpreted string (cf. COM monikers), unlike Unix where the slash-separated list is baked into the architecture, and the former design makes it much harder to have (what Unix calls) a “dentry cache” that would allow the kernel to look up meanings of popular names without going through the filesystem(s).
- p_ing 9mo ago
  NTFS will perform directory B+-tree lookups (this is where it walks the path) until it finds the requested file. The Cache Manager caches these B+-trees. From there, it hits the MFT, finds the specific record for the file, loads the MFT record, and ultimately returns the FILE_OBJECT to the I/O Manager and it bubbles up the chain back to (presumably) Win32. The MFT is just a linear array of records, which include file and directories (directory records are just a record with directory = true, essentially). Obviously simplified. Windows Internals will be your friend, if you want to know more.
  - mananaysiempre 9mo ago
    Thanks for the explanation! Linux, meanwhile, will[1] in the normal case walk a sequence[2] of hash tables (representing incomplete but up-to-date views of directories) before hitting the filesystem’s vtable or the block I/O layer at all, and on the fast path[3] taking no locks other than the RCU read lock. [1] https://www.kernel.org/doc/html/latest/filesystems/path-lookup.html#more-than-just-a-cache https://www.kernel.org/doc/html/latest/filesystems/path-look... [2] I was under the impression that it could look up an entire path at once when I wrote my grandparent comment; it seems I was wrong, which on reflection makes sense given you can move directories. [3] https://www.kernel.org/doc/html/latest/filesystems/path-lookup.html#rcu-walk-faster-pathname-lookup-in-linux https://www.kernel.org/doc/html/latest/filesystems/path-look...

BrouteMinou 9mo ago

The Windows filesystem isn't slow per se, it's a slowness caused by "a thousand cuts" type of problem. https://github.com/Microsoft/WSL/issues/873#issuecomment-425272829 https://github.com/Microsoft/WSL/issues/873#issuecomment-425...