7 ms·

I think the basic problem is that FAT is generally "good enough," and in the increasingly common case where it isn't exFAT is close to universal and addresses t

I think the basic problem is that FAT is generally "good enough," and in the increasingly common case where it isn't exFAT is close to universal and addresses the only problem that consumers frequently run into (file size limit). While FAT/exFAT leave the possibility of a variety of different types of filesystem inconsistency, these seem to be fairly rare in actual practice, probably in good part due to Windows disabling writecaching on devices it thinks are portable. This kind of special handling of external devices is sort of distasteful, and leads to some real downsides on Windows (e.g. LDM handling USB devices weird), but using a newer file system doesn't really eliminate that problem - NTFS and Ext* external devices require special handling on mounting to avoid the problems that come from file permissions traveling from machine to machine, for example.

Wowfunhappy 6y ago
I don’t know if it’s just because I’m largely on Mac and Apple’s drivers suck, but I have run into lots of data corruption issues with exFAT on large drives (1TB+). More than enough for me to stop using it. (Now I use NTFS and Tuxera’s commercial Mac driver, because I don’t know how else to have a cross-platform filesystem without a stupidly-low file-size limit.)
- 0xcde4c3db 6y ago
  It's pretty widely accepted that exFAT should be avoided on Nintendo Switch because of data corruption issues. So I don't think it's just Apple.
- kasabali 6y ago
  > because I don’t know how else to have a cross-platform filesystem without a stupidly-low file-size limit UDF is natively supported on all major OS. I haven't used it in macOS but on paper macOS seems to be having even better support for it than Linux so you can give it a shot. Lastly, you can use https://github.com/JElchison/format-udf https://github.com/JElchison/format-udf for creating most compatible filesystem across different devices.
  - Wowfunhappy 6y ago
    Oh, the optical disk filesystem! That would be a useful hack. I won’t have problems using it on a standard hdd or ssd?
    - kasabali 6y ago
      It isn't a 1st class fs in any OS so it lacks some polish tooling-wise etc., but it should work fine for basic file transfer jobs. I'm using it on an external HDD for copying/watching video files between Linux and Windows boxes and haven't had any problems yet. On the other hand 7-8 years ago I tried to use an UDF partition for sharing a common Thunderbird profile between Windows and Linux and had done strange errors on Windows side after a while. I didn't dig further so it may have been a non-udf os or tooling issue, or it may have been solved in the meantime.
wheybags 6y ago
> in good part due to Windows disabling writecaching on devices it thinks are portable. This kind of special handling of external devices is sort of distasteful What's distasteful is that on my linux machine with a lot of ram, when I copy a multiple gigabyte file to a USB key it "completes" the copy almost immediately, when actually all it has done is copy the file to a ram buffer. Then when I try to disconnect the drive, it will hang for ages while it actually finishes the write. IMO windows does it better here (although I never realised what exactly they did, nice to know.
- sirffuzzylogik 6y ago
  You should try to use the "sync" command, this is exactly what it solves.
  - wheybags 6y ago
    I'm talking about dragging files with the GUI file manager, I shouldn't have to use any commands :(. What I end up doing is using a "watch" on some command I can't remember that shows overall dirty pages.
    - nitrogen 6y ago
      GUI file copy tools should be using O_DIRECT, or periodically calling f/sync(). An argument could also be made that the kernel write cache should have a size limit so that one-off write latency is masked, but very slow bulk I/O is not masked.
      - wtallis 6y ago
        O_DIRECT seems like overkill, and the lack of write buffering could be a real detriment in some circumstances. Syncing at the end of each operation (from the user's perspective) should be the best mix of throughput and safety, but it makes it hard to do an accurate progress bar. Before the whole batch operation is finished, it may be useful to periodically use madvise or posix_fadvise to encourage the OS to flush the right data from the page cache—but I don't know if Linux really makes good use of those hints at the moment. On really new kernels, it might work well to use io_uring to issue linked chains of read -> write -> fdatasync operations for everything the user wants to copy, and base the GUI's progress bar on the completion of those linked IO units. That will probably ensure the kernel has enough work enqueued to issue optimally large and aligned IOs to the underlying devices. (Also, any file management GUI really needs to be doing async IO to begin with, or at least on a separate thread. So adopting io_uring shouldn't be as big an issue as it would be for many other kinds of applications.)

Wowfunhappy 6y ago

I don’t know if it’s just because I’m largely on Mac and Apple’s drivers suck, but I have run into lots of data corruption issues with exFAT on large drives (1TB+). More than enough for me to stop using it. (Now I use NTFS and Tuxera’s commercial Mac driver, because I don’t know how else to have a cross-platform filesystem without a stupidly-low file-size limit.)

0xcde4c3db 6y ago
It's pretty widely accepted that exFAT should be avoided on Nintendo Switch because of data corruption issues. So I don't think it's just Apple.
kasabali 6y ago
> because I don’t know how else to have a cross-platform filesystem without a stupidly-low file-size limit UDF is natively supported on all major OS. I haven't used it in macOS but on paper macOS seems to be having even better support for it than Linux so you can give it a shot. Lastly, you can use https://github.com/JElchison/format-udf https://github.com/JElchison/format-udf for creating most compatible filesystem across different devices.
- Wowfunhappy 6y ago
  Oh, the optical disk filesystem! That would be a useful hack. I won’t have problems using it on a standard hdd or ssd?
  - kasabali 6y ago
    It isn't a 1st class fs in any OS so it lacks some polish tooling-wise etc., but it should work fine for basic file transfer jobs. I'm using it on an external HDD for copying/watching video files between Linux and Windows boxes and haven't had any problems yet. On the other hand 7-8 years ago I tried to use an UDF partition for sharing a common Thunderbird profile between Windows and Linux and had done strange errors on Windows side after a while. I didn't dig further so it may have been a non-udf os or tooling issue, or it may have been solved in the meantime.

wheybags 6y ago

> in good part due to Windows disabling writecaching on devices it thinks are portable. This kind of special handling of external devices is sort of distasteful What's distasteful is that on my linux machine with a lot of ram, when I copy a multiple gigabyte file to a USB key it "completes" the copy almost immediately, when actually all it has done is copy the file to a ram buffer. Then when I try to disconnect the drive, it will hang for ages while it actually finishes the write. IMO windows does it better here (although I never realised what exactly they did, nice to know.

sirffuzzylogik 6y ago
You should try to use the "sync" command, this is exactly what it solves.
- wheybags 6y ago
  I'm talking about dragging files with the GUI file manager, I shouldn't have to use any commands :(. What I end up doing is using a "watch" on some command I can't remember that shows overall dirty pages.
  - nitrogen 6y ago
    GUI file copy tools should be using O_DIRECT, or periodically calling f/sync(). An argument could also be made that the kernel write cache should have a size limit so that one-off write latency is masked, but very slow bulk I/O is not masked.
    - wtallis 6y ago
      O_DIRECT seems like overkill, and the lack of write buffering could be a real detriment in some circumstances. Syncing at the end of each operation (from the user's perspective) should be the best mix of throughput and safety, but it makes it hard to do an accurate progress bar. Before the whole batch operation is finished, it may be useful to periodically use madvise or posix_fadvise to encourage the OS to flush the right data from the page cache—but I don't know if Linux really makes good use of those hints at the moment. On really new kernels, it might work well to use io_uring to issue linked chains of read -> write -> fdatasync operations for everything the user wants to copy, and base the GUI's progress bar on the completion of those linked IO units. That will probably ensure the kernel has enough work enqueued to issue optimally large and aligned IOs to the underlying devices. (Also, any file management GUI really needs to be doing async IO to begin with, or at least on a separate thread. So adopting io_uring shouldn't be as big an issue as it would be for many other kinds of applications.)