5 ms·
I've been writing to NTFS for at least 10 years. It was the FUSE driver but it worked fine and I never had any problem other than it seemed to write at half th
by lizknope 6y ago
I've been writing to NTFS for at least 10 years. It was the FUSE driver but it worked fine and I never had any problem other than it seemed to write at half the speed of ext4
- speedgoose 6y agoNTFS is super slow on windows too. You get better performances with ext4 on WSL2 (Windows Subsystem for Linux).
- risho 6y agohow is this possible? isn't wsl 2 literally a virtual machine image on top of your ntfs drive?
- speedgoose 6y agoYes but it's only one big file, the ext4 partition.
- Tuna-Fish 6y agoAnd to extend, Windows/NTFS is not particularly slow at accessing a large file (actually, until recently they have been ahead for async read/write into big files), the area where they really suck is anything that has anything to do with metadata.
- formerly_proven 6y agoI've been told this has to do with NTFS using a complicated metadata scheme and also all permissions in NTFS being backed by a complex ACL-security-descriptor scheme. Is that an accurate representation?
- Tuna-Fish 6y agoI don't believe it is. Windows metadata performance is not just bad, it's so bad it beggars belief. I don't believe the root cause is any specific technical decision, but rather that no-one cares about it. It has always been so bad that no-one can count on it, so no-one depends on it, so it's something that can safely be ignored. On Linux, and other unices, there have always been important workloads where it really matters, so that itch got scratched.
- loeg 6y agoI don't agree that it's because no one cares about Windows metadata performance; both userspace/3rd-party developers and the kernel engineers care. Rather, they are loath to break any published API, and their metadata performance suffers from the limitations imposed by those historical design decisions (filter drivers, etc).
- dboreham 6y agoThey didn't anticipate a workload where files would be opened and closed very frequently. E.g. running npm. Applications accessing large numbers of small records were expected to do so via a big file containing all the records a la SQLServer.
- formerly_proven 6y agoYet MSI and Windows Update love their gazillion of tiny files, probably one of the reasons why WU is so slow (aside from the "let's solve P=?NP" thing that Windows 7 does). Game devs have got it right for decades though, practically every game ships their own little take on random-access ZIP archives (they often used profiling to arrange the files inside the archives in the order the game would load them as well, though that is probably less important these days), because that's so much faster than using the host file system.
- bleepblorp 6y agoWindows filesystem metadata performance is also highly dependent on how applications access metadata. It's a lot slower than it needs to be, especially for software ported from Linux (or POSIX-like environments in general), because metadata access has to be done in a Windows way to be performant. Windows metadata performance is horrendously bad for code that needs to get metadata for a large number files in the same folder the way it would be done on Linux: by calling the Windows equivalent of `stat` (`GetFileAttributesEx`) for every file. Metadata performance is much better if the code uses `FindFirstFileEx`, which IIRC has no direct Linux equivalent, to grab metadata en masse for all files in a folder and then filtering the results down the files that the code needs to know about. Unfortunately, most code ported to Windows from Linux uses `GetFileAttributesEx`, under the assumption that it's as fast as `stat` is on Linux, but results in a poor user experience. Worse, many modern cross-platform languages (including Python) don't expose `FindFirstFileEx` without platform-specific add-ons, which means that devs have no easy way to do Windows metadata access right even if they know there's one specific way to do it.
- ufmace 6y agoI had heard the issue was with how Windows/NTFS let a bunch of extensions place hooks into reading and writing files. Antivirus apps would sometimes put a bunch of heavy-ish hooks in that would be terrible for performance with lots of tiny files, which many Unix-y workflows expect to be able to do quickly.
- opencl 6y agoThe "sometimes" includes the one Microsoft ships by default with Windows. It is ridiculous how much faster random file access becomes when disabling Defender's real time protection feature.
- ziml77 6y agoIt's one of the most impactful AVs when it comes to file r/w speed, especially on lots of small files. I have no idea why I still see people calling it lighter and faster than other AV options. It's not even like the performance loss is getting better protection. Products like Bitdefender and Kaspersky have better detection rates and don't murder performance. (Personally I just turn it off and don't install a replacement)
- giancarlostoro 6y agoThis is probably why JetBrains IDEs suggest to add your projects to antivirus exceptions for you. I rarely ever have performance issues while developing anymore after I let JetBrains do its thing.
- neurostimulant 6y agoI think it bypassed windows filesystem filters (antivirus, etc) and thus faster because there is nothing taxing your read/write operations.
- tfigment 6y agoIt least with WSL1 I know that McShield and cyvera/Traps are fully capable of massively interfering with linux performance under windows 10 just like they do normally. Whether they actually effective is not something I know but it seems like they understand ELF files.
- verst 6y agoThe WSL2 kernel at this time cannot access ext4 partitions natively. Only partitions available in Windows can be mounted in WSL2. So that means installing ext4 drivers... WSL2 itself uses a virtual file system.
- mehrdadn 6y agoThis is not an NTFS issue. You can verify it by trying a different file system.
- throwaway8941 6y agoHere's more or less official description of this issue. https://github.com/Microsoft/WSL/issues/873#issuecomment-425272829 https://github.com/Microsoft/WSL/issues/873#issuecomment-425...
- verst 6y agoWSL2 simulates a virtual ext4 file system. If you want to access ext4 natively from WSL2, wait for the next Windows Insiders release!
- microcolonel 6y agoI've seen numbers in some workloads where ntfs-3g is 40% faster than NTFS on Windows, despite crossing between userspace and the kernel three times for every filesystem call (recent work in the kernel will potentially cut this down to two context switches each way for FUSE).
- Macha 6y agoSame story here, I do not remember a time when I couldn't read/write NTFS from Linux since I started using it in with Ubuntu 8.04