6 ms·

TIL: In 2019, Microsoft released the exFAT [1] specs to a large group of vendors [2], including prominent Linux vendors. I.e. exFAT should now be preferred over

by m000 6y ago

TIL: In 2019, Microsoft released the exFAT [1] specs to a large group of vendors [2], including prominent Linux vendors. I.e. exFAT should now be preferred over FAT/NTFS for media used across different operating systems. [1] https://en.wikipedia.org/wiki/ExFAT https://en.wikipedia.org/wiki/ExFAT [2] https://en.wikipedia.org/wiki/Open_Invention_Network https://en.wikipedia.org/wiki/Open_Invention_Network

gruez 6y ago
>TIL: In 2019, Microsoft released the exFAT [1] specs [...] I.e. exFAT should now be preferred over FAT/NTFS for media used across different operating systems. You seem to be very optimistic about how fast these get implemented into embedded devices.
- netsharc 6y ago
  I have a 2011 car which has a USB port where I can plug a USB stick with MP3s (remember those, kids?) on them, I doubt it will ever be made ExFAT-compatible. OTOH, I do have the Windows logo on my steering wheel, and somewhere inside the car there's a Windows CE system installed (it doesn't have a multi-color LCD screen and interacts using speech recognition, so it's not really obvious).
- bilekas 6y ago
  I would imagine if embedded devices are being assembled and configured with filesize concerns, then exFAT would be the natural implementation they would opt for. Or do you mean updating/transitioning legacy embedded? The spec doesn't mention backwards compat, but does mention as a goal >Retain the simplicity of FAT-based file systems. I would like to test out if an input embed system is exFAT compat. And what might be required to update.
  - wongarsu 6y ago
    Embedded devices often have a lifespan measured in decades. And for short-lived devices like consumer all-in-one printers (~4 years) it's not like every new model has a new firmware written from scratch. They get whatever the last model had, with a few updates for new device features. That might include exFAT, or it might not.
  - numpad0 6y ago
    I believe the implication is SD card/USB drive support on microcontrollers without an OS. SD under 32GB uses FAT32 so you can’t drop supports, and FAT32 stack is often already pushing the maximum ROM size, so no room is left for a novel filesystem. Lots of developers aren’t themselves proficient at writing filesystem code as well, and there aren’t as many copy-pastable exFAT implementations as there are for FAT32. So unless they know what they’re doing and unless it’s critical to their product, the majority won’t easily move onto exFAT. As for the use within the device “firmware”, the user defined program is usually just a giant baremetal a.out in the middle of the ROM at a specified address, so no FS is involved there.
- riedel 6y ago
  Maybe there are even capable embedded implementations even before Microsoft implements all supported features like ACLs in their OS.
kop316 6y ago
Unfortunately, this only applies to systems that actually have this (i.e. they were updated after exFAT released or released after and included this update). There will be sadly many systems that will never receive this update. But for a fun fact, if you see many embedded systems that say they can only support an SD card up to 32 GB, it is a soft limitation due to the fact that media over 32 GB are pre-formatted to exFAT (and the system lacks support for exFAT). FAT32 can go up to 2 TB. So I have found that reformatting a larger media back to FAT32 makes it recognizable to that system with all of the storage space.
- est31 6y ago
  > But for a fun fact, if you see many embedded systems that say they can only support an SD card up to 32 GB, it is a soft limitation due to the fact that media over 32 GB are pre-formatted to exFAT (and the system lacks support for exFAT). There might also be the issue of the card reading hardware only supprting SDHC instead of SDXC (or SDUC).
  - kop316 6y ago
    That is true, though I have yet to run into that issue though for the devices I have tried it on (which admittedly is not a large sample size).
  - kasabali 6y ago
    Even then it is actually only a driver or firmware limitation most of the time. My old laptop has Ricoh SDHC card reader. Windows can't read 64GB SD cards (properly formatted with exFAT according to the standard) on it because it's using the proprietary Ricoh driver. On Linux it works just fine because it's using the generic SD card reader driver which supports SDXC.
- martinald 6y ago
  FAT32 only supports 4GB files though, so not sure how useful this is for a lot of media usecases.
  - 6y ago
tmd83 6y ago
I guess that's the best option for compatibility. But it's not journaled and single fat, very prone to corruption right and fixing errors could take a while for lager disk like external hdd. There's not really any good option in terms of reliability and compatibility I guess. I have had issues with ntfs getting corrupted and losing file both in mac and linux at times so there's that risk with implementation too.
grenoire 6y ago
Huh, I thought exFAT was much older, not 2019.
- MrDOS 6y ago
  It is – it dates back to 2006. But before 2019, the spec was proprietary and patent-encumbered. And the Linux kernel didn't include a good driver for it until 5.7, released in May of last year[0]. [0]: https://kernelnewbies.org/Linux_5.7#New_exFAT_file_system https://kernelnewbies.org/Linux_5.7#New_exFAT_file_system
  - kasabali 6y ago
    It is still patent encumbered. MS just promised to not give trouble for it being in the Linux kernel.
    - MrDOS 6y ago
      Right – that's what I meant, but I didn't communicate it clearly. Thanks for clarifying.
fuzzfactor 6y ago
The patents on FAT32 are now fully expired, including the final lingering LongFileName one. Therefore FAT32 will remain preferred until the patents expire on exFAT, even though the specification for exFAT has been released, things like that have proven to be not good enough until after the patents expire. The purpose of exFAT appears mainly to be an attempt to foist a new encumbered filesystem on developers now that the first mainstream Microsoft filesystem is in the public domain. We will know when Microsoft has reduced their anti-user stance on filesystems after they have released very detailed full NTFS specifications in the most useful way possible. Also I have the greatest respect for Dave, and like he mentioned he was only doing the UI not the bitmoving of the actual Format process. Those who were developing the NT version of FAT32 formatting failed in much more serious ways where Dave did not. The inconsistency compared to FAT32 formatted by W9x is actually intolerable. Best reliability still requires booting to the DOS from W98SE to properly format FAT32.
- adrian_b 6y ago
  The main use for FAT32 and exFAT is to transfer files between different operating systems or between a computer and some dedicated device, e.g. a photographic camera. They are not useful for anything else except memory cards, USB sticks and external SSDs. Unfortunately, nowadays it is frequently necessary to copy files larger than 4 GB. When using FAT32, you cannot do that without splitting the file and concatenating the parts at destination. That wastes time and it is annoying and sometimes it may be even impossible, when the destination lacks a concatenation program or an archiver that can deal with split archives. After Microsoft released it, now exFAT is supported in the Linux kernel. Because exFAT is also supported by all commercial operating systems, it is now the best way for file transfer. It does not matter any more if there are still valid patents for exFAT.
  - KronisLV 6y ago
    > It does not matter any more if there are still valid patents for exFAT. Would you care to elaborate for why that is? Would anyone else with knowledge about the patent system and any possible risks imposed by it also like to do so?
Abishek_Muthian 6y ago
Much before MS did that, macOS had builtin exFAT support presumably licensed from MS, Paragon had exFAT support for android for a one time fee which works with other file manager apps(also licensed from MS), On Linux Exfat-utils enabled support for exFAT(open-source implementation). Has there been any changes to these since MS has opened the license to exFAT?
- m000 6y ago
  Third-party exFAT support did exist before 2019, but vendors had to pay a licence fee to Microsoft. Apple apparently paid the fee. Regarding Linux, exfat-utils only provided limited functionality. The kernel mainline lacked a native exFAT driver. Mounting exFAT volumes was possible only through FUSE (with all its shortcomings).
- adrian_b 6y ago
  The change is that now there is an exFAT driver included in the Linux kernel. Therefore, if you have a recent Linux kernel, you no longer need to install a user-mode driver for exFAT and the kernel driver has better performance.

by m000 6y ago

gruez 6y ago

>TIL: In 2019, Microsoft released the exFAT [1] specs [...] I.e. exFAT should now be preferred over FAT/NTFS for media used across different operating systems. You seem to be very optimistic about how fast these get implemented into embedded devices.

netsharc 6y ago
I have a 2011 car which has a USB port where I can plug a USB stick with MP3s (remember those, kids?) on them, I doubt it will ever be made ExFAT-compatible. OTOH, I do have the Windows logo on my steering wheel, and somewhere inside the car there's a Windows CE system installed (it doesn't have a multi-color LCD screen and interacts using speech recognition, so it's not really obvious).
bilekas 6y ago
I would imagine if embedded devices are being assembled and configured with filesize concerns, then exFAT would be the natural implementation they would opt for. Or do you mean updating/transitioning legacy embedded? The spec doesn't mention backwards compat, but does mention as a goal >Retain the simplicity of FAT-based file systems. I would like to test out if an input embed system is exFAT compat. And what might be required to update.
- wongarsu 6y ago
  Embedded devices often have a lifespan measured in decades. And for short-lived devices like consumer all-in-one printers (~4 years) it's not like every new model has a new firmware written from scratch. They get whatever the last model had, with a few updates for new device features. That might include exFAT, or it might not.
- numpad0 6y ago
  I believe the implication is SD card/USB drive support on microcontrollers without an OS. SD under 32GB uses FAT32 so you can’t drop supports, and FAT32 stack is often already pushing the maximum ROM size, so no room is left for a novel filesystem. Lots of developers aren’t themselves proficient at writing filesystem code as well, and there aren’t as many copy-pastable exFAT implementations as there are for FAT32. So unless they know what they’re doing and unless it’s critical to their product, the majority won’t easily move onto exFAT. As for the use within the device “firmware”, the user defined program is usually just a giant baremetal a.out in the middle of the ROM at a specified address, so no FS is involved there.
riedel 6y ago
Maybe there are even capable embedded implementations even before Microsoft implements all supported features like ACLs in their OS.

kop316 6y ago

Unfortunately, this only applies to systems that actually have this (i.e. they were updated after exFAT released or released after and included this update). There will be sadly many systems that will never receive this update. But for a fun fact, if you see many embedded systems that say they can only support an SD card up to 32 GB, it is a soft limitation due to the fact that media over 32 GB are pre-formatted to exFAT (and the system lacks support for exFAT). FAT32 can go up to 2 TB. So I have found that reformatting a larger media back to FAT32 makes it recognizable to that system with all of the storage space.

est31 6y ago
> But for a fun fact, if you see many embedded systems that say they can only support an SD card up to 32 GB, it is a soft limitation due to the fact that media over 32 GB are pre-formatted to exFAT (and the system lacks support for exFAT). There might also be the issue of the card reading hardware only supprting SDHC instead of SDXC (or SDUC).
- kop316 6y ago
  That is true, though I have yet to run into that issue though for the devices I have tried it on (which admittedly is not a large sample size).
- kasabali 6y ago
  Even then it is actually only a driver or firmware limitation most of the time. My old laptop has Ricoh SDHC card reader. Windows can't read 64GB SD cards (properly formatted with exFAT according to the standard) on it because it's using the proprietary Ricoh driver. On Linux it works just fine because it's using the generic SD card reader driver which supports SDXC.
martinald 6y ago
FAT32 only supports 4GB files though, so not sure how useful this is for a lot of media usecases.
- 6y ago

tmd83 6y ago

I guess that's the best option for compatibility. But it's not journaled and single fat, very prone to corruption right and fixing errors could take a while for lager disk like external hdd. There's not really any good option in terms of reliability and compatibility I guess. I have had issues with ntfs getting corrupted and losing file both in mac and linux at times so there's that risk with implementation too.

grenoire 6y ago

Huh, I thought exFAT was much older, not 2019.

MrDOS 6y ago
It is – it dates back to 2006. But before 2019, the spec was proprietary and patent-encumbered. And the Linux kernel didn't include a good driver for it until 5.7, released in May of last year[0]. [0]: https://kernelnewbies.org/Linux_5.7#New_exFAT_file_system https://kernelnewbies.org/Linux_5.7#New_exFAT_file_system
- kasabali 6y ago
  It is still patent encumbered. MS just promised to not give trouble for it being in the Linux kernel.
  - MrDOS 6y ago
    Right – that's what I meant, but I didn't communicate it clearly. Thanks for clarifying.

fuzzfactor 6y ago

The patents on FAT32 are now fully expired, including the final lingering LongFileName one. Therefore FAT32 will remain preferred until the patents expire on exFAT, even though the specification for exFAT has been released, things like that have proven to be not good enough until after the patents expire. The purpose of exFAT appears mainly to be an attempt to foist a new encumbered filesystem on developers now that the first mainstream Microsoft filesystem is in the public domain. We will know when Microsoft has reduced their anti-user stance on filesystems after they have released very detailed full NTFS specifications in the most useful way possible. Also I have the greatest respect for Dave, and like he mentioned he was only doing the UI not the bitmoving of the actual Format process. Those who were developing the NT version of FAT32 formatting failed in much more serious ways where Dave did not. The inconsistency compared to FAT32 formatted by W9x is actually intolerable. Best reliability still requires booting to the DOS from W98SE to properly format FAT32.

adrian_b 6y ago
The main use for FAT32 and exFAT is to transfer files between different operating systems or between a computer and some dedicated device, e.g. a photographic camera. They are not useful for anything else except memory cards, USB sticks and external SSDs. Unfortunately, nowadays it is frequently necessary to copy files larger than 4 GB. When using FAT32, you cannot do that without splitting the file and concatenating the parts at destination. That wastes time and it is annoying and sometimes it may be even impossible, when the destination lacks a concatenation program or an archiver that can deal with split archives. After Microsoft released it, now exFAT is supported in the Linux kernel. Because exFAT is also supported by all commercial operating systems, it is now the best way for file transfer. It does not matter any more if there are still valid patents for exFAT.
- KronisLV 6y ago
  > It does not matter any more if there are still valid patents for exFAT. Would you care to elaborate for why that is? Would anyone else with knowledge about the patent system and any possible risks imposed by it also like to do so?

Abishek_Muthian 6y ago

Much before MS did that, macOS had builtin exFAT support presumably licensed from MS, Paragon had exFAT support for android for a one time fee which works with other file manager apps(also licensed from MS), On Linux Exfat-utils enabled support for exFAT(open-source implementation). Has there been any changes to these since MS has opened the license to exFAT?

m000 6y ago
Third-party exFAT support did exist before 2019, but vendors had to pay a licence fee to Microsoft. Apple apparently paid the fee. Regarding Linux, exfat-utils only provided limited functionality. The kernel mainline lacked a native exFAT driver. Mounting exFAT volumes was possible only through FUSE (with all its shortcomings).
adrian_b 6y ago
The change is that now there is an exFAT driver included in the Linux kernel. Therefore, if you have a recent Linux kernel, you no longer need to install a user-mode driver for exFAT and the kernel driver has better performance.