4 ms·
ExFAT Driver Boasts Much Faster "Dirsync" Performance with Linux 6.9
- sillysaurusx 3y agoAfter John sent me a 5.8TB .bzip2 file in the mail, I discovered that the default exFAT drivers in macOS simply keel over and die. https://x.com/theshawwn/status/1695563727326642377 https://x.com/theshawwn/status/1695563727326642377 I’m still not sure why. But after intensive debugging, the fix was to install a third party driver, which thankfully exists. It made me question why default drivers are often so bad.
- abofh 3y agoDefault drivers are designed to satisfy 95%-99% of customer use-cases, and are tested for those. Whoever was tasked with writing the default one probably didn't have "Test 5.8 TB file on a single exFAT volume" on their spec list.
- userbinator 3y agoThe first thing that came to mind is that newer versions of macOS have a filesystem throttling feature (yes, really): https://news.ycombinator.com/item?id=31788894 https://news.ycombinator.com/item?id=31788894 exFAT should be simple if it's a variant of traditional FAT, although I haven't looked at the spec in detail; hence why I doubt the FS driver itself is to blame. (FYI no need to keep the additional tracking parameters in your link.)
- sillysaurusx 3y agoI winced at the tracking parameters when I pasted the link. I gave up trying to keep my privacy private, but I forgot that another reason to delete them is out of respect for whoever is clicking, not just me. Thanks for calling that out; I’ve edited them out. And thanks for the data point about the mystery. It always bugged me that it wasn’t quite tracked down. Isn’t filesystem throttling a problem with the driver, though? Switching to a third party exFAT driver immediately fixed whatever was happening.
- jraph 3y agoThe new Firefox "Copy Link without Site Tracking" feature may help a lot for this :-)
- formerly_proven 3y agoGonna guess "find me the next block for this file offset" is accidentally O(n²) in their implementation. Exfat is probably still list-based, like FAT32, and not extent-based, like any other file system from the last 40 years?
- sillysaurusx 3y agoThis is the most plausible explanation I’ve come across. Thanks; the mystery was bugging me.
- Wowfunhappy 3y agoUsing exFAT under macOS over a period of a few years led to several of my drives becoming corrupted. I have not had issues since I switched away from exFAT, and I am now convinced that macOS's exFAT driver is unacceptably terrible. If you need a multiplatform filesystem for a thumb drive or small portable hard drive, I have had success with UDF. Surprisingly so, given that it's basically a hack. See: https://github.com/JElchison/format-udf https://github.com/JElchison/format-udf. This basically won't work on drives over 2TB, however.
- efrecon 3y agoI second that. At a past job we picked UDF for being the only format that would work across all platforms. That was before ExFAT got everywhere, but still seems like a good choice.
- goodburb 3y agoUseful comparison: https://www.sami-lehtinen.net/blog/exfat-extended-file-allocation-table-vs-udf-universal-disk-format-file-system-for-flash-drives https://www.sami-lehtinen.net/blog/exfat-extended-file-alloc...
- Wowfunhappy 3y agoThe note about the partition table makes me wonder how he was formatting the drive. https://github.com/JElchison/format-udf https://github.com/JElchison/format-udf specifically does some partition table related hacks to make UDF maximally compatible with different platforms.
- goodburb 3y agoLargest issue seems to be regarding corruptions, BSD platforms seems the only ones to include updated UDF file system check/repair tools.
- goodburb 3y agoSmall ExFAT chunk size has significant performance degradation on macOS and Linux's userspace ExFAT. It also took 5 minutes to mount an SSD with 500k photos. Bumping it to 4MB fixed it but used more storage. Windows and the Linux kernel module variant are unaffected.
- nicoty 3y agoI haven't kept up to date with ExFAT in a while. Is it still impossible to resize ExFAT partitions?