5 ms·
NTFS is in some ways arguably still more advanced than ZFS: think reparse points, alternate streams, transactions, BitLocker integration, and shrinkable volumes
by otterley 8y ago
NTFS is in some ways arguably still more advanced than ZFS: think reparse points, alternate streams, transactions, BitLocker integration, and shrinkable volumes, to name a few.
Also, NTFS already has snapshots via Volume Shadow Copy.
ZFS is great but it’s not going to be a viable replacement as it exists today.
- setquk 8y agoNot a lot of this matters in the real world. The real shame is the shitty small file performance due to MFT contention on NTFS.
- otterley 8y agoIt absolutely matters: a fair amount of software, both enterprise and consumer, already uses these features. Malware detection, for example, makes heavy use of reparse points.
- blattimwind 8y agoWindows Update and Windows Installer use NTFS transactions.
- setquk 8y agoFrom Microsoft's documentation: Microsoft strongly recommends developers utilize alternative means to achieve your application s needs. Many scenarios that TxF was developed for can be achieved through simpler and more readily available techniques. Furthermore, TxF may not be available in future versions of Microsoft Windows. For more information, and alternatives to TxF, please see Alternatives to using Transactional NTFS. https://docs.microsoft.com/en-us/windows/desktop/fileio/about-transactional-ntfs https://docs.microsoft.com/en-us/windows/desktop/fileio/abou...
- mehrdadn 8y agoI'm pretty convinced that documentation is BS. I could be wrong I guess, but I honestly don't see them removing TxF anytime soon. They didn't exactly build their business on breaking backwards-compatibility of APIs.
- setquk 8y agoThey've burned a lot of APIs in the last few years.
- mehrdadn 8y agoExamples?
- setquk 8y agohttps://docs.microsoft.com/en-us/windows/desktop/apiindex/windows-api-list#deprecated-or-legacy-apis https://docs.microsoft.com/en-us/windows/desktop/apiindex/wi...
- mehrdadn 8y agoThey're not removed though? They're 'deprecated' as in they don't want you to depend on them, but they're still very much there and usable.
- cryptonector 8y agoZFS snapshots work well enough for that. The Sun Storage Appliance used ZFS snapshots for updates, and it worked quite well.
- otterley 8y agoDid that change recently? I haven't used Windows since XP, but back then, they used Volume Shadow Copies (VSS) for making backups used for reverting changes, not NTFS transactions.
- tjalfi 8y agoWindows Update switched to using NTFS transactions in Windows Vista.
- setquk 8y agoReparse points could be emulated at API level.
- ygra 8y agoIs the MFT even documented and available via APIs on Windows? Could NTFS (at the expense of losing backwards compatibility in the file system (which already only extends to features that have been there before)) just change how the MFT works transparently for well-behaved applications (i.e. those that do not grovel around in undocumented data structures)? Admittedly, there may still be a number of applications that decide to »optimize« a system by digging around in parts that they have reverse engineered, and those might simply corrupt the file system when running on a newer NTFS version, but I guess that problem may have existed previously as well.
- setquk 8y agoMFT is an internal structure. They could fix it TBH but the issue is that this would require an NTFS filesystem revision and for those of us who suffered through NT4's broken ass filesystems might start complaining :)
- Gladdyu 8y agoThough it might finally do away with the ever-persisting Windows error messages about "this file is already open" followed by a search through all of the processes that might have it open to terminate the offender.
- otterley 8y agoThat error is orthogonal to the filesystem in use. Every OS that provides exclusive locking, regardless of filesystem, will return an error if more than one process wants to acquire such a lock.
- reitanqild 8y ago> Every OS that provides exclusive locking, regardless of filesystem, will return an error if more than one process wants to acquire such a lock. Obviously. The super annoying thing is when the OS doesn't try to tell you which process(es) keeps it open and doesn't even ship with built in tooling to let you find out on your own. I think that is the point GP tries to make.
- Negitivefrags 8y agoOpen up "Resource Monitor" and go to the "Disk" tab. Under "Disk Activity" you can see all open file handles.
- coldtea 8y agoWhich are thousands...
- psykotic 8y agoYou need the Sysinternals utilities to make life tolerable: https://docs.microsoft.com/en-us/sysinternals/downloads/handle https://docs.microsoft.com/en-us/sysinternals/downloads/hand... from the command line or Ctrl-F in https://docs.microsoft.com/en-us/sysinternals/downloads/process-explorer https://docs.microsoft.com/en-us/sysinternals/downloads/proc... if you prefer a GUI. They do a brute-force search over the open handles in all processes, but it's effective.
- kuwze 8y agoDidn’t know any of that. Thanks!
- psykotic 8y ago> think reparse points, alternate streams, transactions, BitLocker integration, and shrinkable volumes, to name a few. NTFS's USN change journal (which is separate from the transaction journal) is also useful as a robust offline alternative to ReadDirectoryChangesW file monitoring. Toy example: https://gist.github.com/pervognsen/bcc610d6a5ae6cbc3b2b4f7aa00d8da9 https://gist.github.com/pervognsen/bcc610d6a5ae6cbc3b2b4f7aa.... Anyway, the problem I've always had when using non-standard file system metadata like alternate streams or sparse files is that they interoperate poorly in practice. There's lots of code and programs out there that wants to treat everything like a plain old file and hence won't preserve alternate streams or sparseness. From my recollections this is true even for several of Windows's built-in utilities. Same issue if you want to transfer files over a socket or pipe or a non-NTFS storage medium like a FAT-formatted USB drive or a NetApp filer or a source control system. As a result this kind of thing is only really useful when deployed in a bubble; I believe WSL uses alternate streams for Unix permission bits and other Unix-specific file properties. Even though those WSL guest files live in a directory tree on a normal NTFS volume, they strongly caution you against touching them directly outside of WSL, presumably for the aforementioned reasons. Aside from features, NTFS has some performance issues compared to other modern file systems, notably with lots of smaller files.
- mehrdadn 8y agoSome minor notes: - For permissions WSL uses EAs; for capabilities it uses ADS - I'm not sure if it's really NTFS that is slow with lots of small files or the I/O subsystem in general... I got the impression it's the latter but not sure.
- chungy 8y ago> think reparse points, alternate streams, transactions, BitLocker integration, and shrinkable volumes, to name a few. All of these except shrinkable volumes (somewhat) are already part of ZFS, also assuming that you replace "BitLocker integration" with more general "encryption integration" ZFS and NFSv4 were both designed to be able to serve the entire subset of NTFS features over the network, to interop with Windows computers.
- otterley 8y agoReparse points and ADS? Do you have a reference handy? I’ve never seen such things or anything like them in ZFS.
- chungy 8y agoReparse points are basically more primitive directory-only symlinks, aren't they? Even the symlinks introduced in Vista are better... As for ADS: basically xattrs. They can be arbitrarily named and each xattr on ZFS can be up to 16EiB large (same as the primary bin for file content). (As an aside, they aren't fully usable on Linux since Linux itself imposes a 64KiB limit on an xattr's content -- but that's not a ZFS limitation)
- ygra 8y agoReparse points are directory entries that are special and interpreted by the file system driver (or some other higher-level component that's not the file system itself). Directory junctions (which you mean here, I guess) are reparse points, as are symlinks and a bunch of other things.
- otterley 8y ago> Reparse points are basically more primitive directory-only symlinks, aren't they? Absolutely not! Reparse points are directories that are associated with drivers that extend the capability of the filesystem itself. They can be used to implement symlink-like behavior, but that's only a single use case that scratches the surface of their potential power. See https://docs.microsoft.com/en-us/windows/desktop/fileio/reparse-points https://docs.microsoft.com/en-us/windows/desktop/fileio/repa... for more information.
- usefulcat 8y agoIf a file is damaged by bit rot, NTFS will happily serve up garbage. So those are nice features, but viewed in that light they're little more than window dressing.
- Demiurge 8y agoYou say that as if bit rot is the black plague of data storage. In the past 20 years I have encountered 'bit rot' with NTFS... not once. I have encountered bad blocks and replaced drives once in a while by monitoring SMART status, which probably prevented the rot. It is definitely great to have protection against it, but to call other features window dressing when others have used them more often than they thought of checksumming feature is not objectively fair.
- usefulcat 8y agoIn the past 20 years I've lost multiple files to bit rot on NTFS. In my opinion, the primary purpose of a filesystem is to reliably store data. Checksumming is not particularly complex technically, does not impose significant run time costs, and allows the filesystem to at least detect when data has been damaged. Unless there's some other very compelling reason to not do checksumming, it's reasonable to consider it's omission to be a de facto bug.
- Demiurge 8y agoDoes this mean ext4 is bugged too? We many filesystems with this bug, and we lived with this bug for a while. Anyway, I don't disagree on the technical cost/benefit analysis, but still, unfair.