46 ms·
BigFAT – Backward compatible FAT extension for unlimited file size
- deleted 4y ago[deleted]
- tzahifadida 4y agoLooks good. Keep at it!
- mikece 4y ago> Why not exFAT... Microsoft owns several patents, and anyone who implements or uses exFAT technology needs Microsoft's permission, which typically also includes paying fees to Microsoft. While BigFAT not being encumbered by any patents is a good thing, the camera industry have pretty much standardized on exFAT for their removable file storage format. Something I'm curious about is how a 5GB video file (quite common and actually on the smaller size for 4K and 8K recording sessions) is written and accessed between the two file systems. BigFAT says that the file would be written in 4GB chunks; is there something similar happening with exFAT or is the file "one chunk?" (Apologies if I have the terms wrong -- I'm not a filesystem expert.) The author laments that the exFAT format has been adopted for SDXC cards but given who all is in this group and what their use cases are I can discount "because Microsoft strong-armed them" as a reason for them selecting it.
- lathiat 4y agoExFAT is not limited to a 4GB maximum file size. It just has more than 4GB in the file. I guess 4GB seemed like a reasonable limit when FAT32 was designed. Most likely FAT32 has a 32bit number for file size and ExFAT presumably has either a 64bit one or stores file size in some format other than bytes.
- creshal 4y ago> I guess 4GB seemed like a reasonable limit when FAT32 was designed. FAT32 was always seen as stop-gap measure for low-end consumer hardware when introduced in 1996; NTFS was introduced 3 years prior to handle terabyte-scale data for enterprise users. > Most likely FAT32 has a 32bit number for file size and ExFAT presumably has either a 64bit one Correct.
- zinekeller 4y agoI actually am disappointed that Microsoft has a chance to fix some inherent problems with FAT but didn't, even considering the main use case of a simple FS. Notably, it still has the notorious year 2100 bug (or 2108 bug, depending on the implementation), the metadata is weird and not at all straightforward, it's basically just extending FAT32 and some minor updates since Unicode and UTC are now here.
- cmurf 4y agoThe industry could have used UDF. Derived from ISO 9660, but it supports read-write random access storage. I'm guessing they didn't if FAT12/16/32->exFAT driver changes are comparatively simple, and/or results in a smaller code base to support FAT32 and exFAT on the same device (e.g. a camera).
- mikece 4y agoAnd on a camera that costs anywhere from USD$1000 to USD$6500 does the cost of an exFAT license really matter?
- vetinari 4y agoYes, it does. If you manufacture 100K of them, and save 10 dollars on every piece, you got an extra million in savings.
- xxpor 4y agoFrom the wiki, my understanding is the licence is $0.25 a unit, not $10?
- vetinari 4y agoIt was a number to illustrate a point, not exact figure for this specific case. Yes, the manufacturers will go great lengths to minimize variable costs. If they can shave $0.25, they will. At volumes, it matters.
- steeleduncan 4y agoIs there a linux kernel driver for this somewhere?
- ninefathom 4y agoI'm a bit puzzled as to how split files with name standardization is an "extension." It seems to me that SEGGER is simply proposing a de facto file naming convention, and offering a few free tools (including a few abstraction drivers) to encourage adoption. Can somebody fill me in, here- where's the value in what SEGGER is proposing, as opposed to what the entire IT community has already been doing for decades?
- mort96 4y agoWell, if we view FAT32 + this name convention as a new filesystem, then filesystem drivers could let you transparently operate on files bigger than 4GB (GiB?) and take care of the splitting for you. FAT32 + this convention would essentially become a filesystem which supports files up to around 4TB. You wouldn't have to make the choice between the patent-encumbered exFAT and the open but limited FAT32.
- jaclaz 4y agoI had the same impression, I cannot see how it is different from the tools that allow file splitting (for later archiving on floppies. CD's, DVD's, etc.) that exist since forever, when/if implemented in OS filesystem drivers then it will be more "transparent", but until then it seems to me not different from multi-part archives, such as rar or similar. If I may it would make more sense (to me at least) to use a directory and have a descriptor file, not entirely unlike multi-part vmdk's are implemented.
- mort96 4y agoI like the idea. Making it backwards compatible with FAT means that, in principle, regular FAT filesystem implementations could be transparently changed to support big fat files (hehe) transparently. However, reading the spec, it doesn't look fully backwards compatible? It seems like there are file structures which are possible to represent in FAT which aren't possible to represent in BigFAT. In FAT, I could have a 4GB-128kB size file called "hello.txt", and next to it, a file called "hello.txt.000.BigFAT". A FAT filesystem will show this as intended, but a BigFAT implementation will show it as one file' "hello.txt". That makes this a breaking change. I would kind of have hoped that they had found an unused but always-zero bit in some header which could be repurposed to identify whether a file has a continuation or not, or some other clever way of ensuring that you can represent all legal FAT32 file structures.
- chasil 4y agoThere are so many good filesystems out there. Is it really necessary to keep dragging FAT along? ReactOS is using btrfs, which has so many useful options that FAT will never see (zstd, xxhash, flash-aware options, snapshots, send/receive, etc.). This is positioned both for Linux and Windows. Microsoft itself restrains ReFS to enterprise use, and btrfs offers so much more functionality. We should stop using a file system from the '80s.
- mort96 4y agoCamera manufacturers and SD card manufacturers can't start shipping SD cards formatted with btrfs until Windows supports it out of the box. They can start shipping SD cards formatted with FAT32 and software/firmware which reads and writes FAT32+BigFAT.
- chasil 4y agoI guess what is needed is a BSD implementation of btrfs. Still, something similar to fuse might help with the licensing.
- 4y ago
- JAA1337 4y agoAwesome concept, especially for academia ... but is there a value proposition? I love seeing this, don't get me wrong. I am just curious is there are any real world applications for this?
- mort96 4y agoIt would be great to have a non-patent-encumbered simple file format that's supported everywhere. The fact that this is based on FAT32 might help adoption, everyone's computers can already at least read a BigFAT drive, and BigFAT support could be added at the application level for systems which don't support it at an OS level.
- prmoustache 4y agoAre the filesystem used on bsd and linux distros patent encumbered? Isn't UFS2 simple enough?
- mort96 4y agoUFS2 might be the technically perfect tool for the job, but that doesn't matter when Windows doesn't support it. A camera manufacturer or SD card manufacturer can't start shipping their customers SD cards formatted with UCS2 when it's not supported by Windows. They could start shipping FAT32 SD cards and software and firmware which can read and write FAT32+BigFAT.
- RealStickman_ 4y agoIn my opinion UDF [1] would be a great option. Although it's mainly used in DVDs, it can also be created as read/write capable. [2] [1] https://en.m.wikipedia.org/wiki/Universal_Disk_Format https://en.m.wikipedia.org/wiki/Universal_Disk_Format [2] https://duncanlock.net/blog/2013/05/13/using-udf-as-an-improved-filesystem-for-usb-flash-drives/ https://duncanlock.net/blog/2013/05/13/using-udf-as-an-impro...
- yyyk 4y ago
- kmeisthax 4y ago>Unfortunately, exFAT has been adopted by the SD Association as the default file system for SDXC cards larger than 32 GB. In our view, this should never have happened, as it forces anyone who wants to access SDXC cards to get a license from Microsoft, basically making this a field owned by Microsoft. So, this is a bit of a cultural/perception gap between FOSS developers and standards bodies. Most standards bodies have a patent policy of "as long as all the standards-essential patents are licensable for a uniform fee, we're good". Convincing patent holders to not extract royalties from their patents for the sake of easing the lives of FOSS implementers is much, much harder[0]. Microsoft isn't even the only SEP holder for SD, and the standard makes no attempt at being a royalty-free standard. In fact, early SD standards were NDA'd[1] and prohibited FOSS implementation at all. [0] In fact, so hard that the EU has a conspiracy theory that Google/AOM bullied a patent holder into doing this [1] Remember, SD cards were basically MMC with primitive DRM
- tumetab1 4y agoThe thing missing on the page is some kind of performance benchmark which I would love to read/see.
- CodesInChaos 4y agoAre the exFAT patents still a problem nowadays? > exFAT was a proprietary file system until 2019, when Microsoft released the specification and allowed OIN members to use their patents. https://en.wikipedia.org/wiki/ExFAT#Legal_status https://en.wikipedia.org/wiki/ExFAT#Legal_status
- NotYourLawyer 4y ago> We also support the eventual inclusion of a Linux kernel with exFAT support in a future revision of the Open Invention Network’s Linux System Definition, where, once accepted, the code will benefit from the defensive patent commitments of OIN’s 3040+ members and licensees. I don’t know exactly what that means. But it sounds like something different from “we hereby grant everybody a license to any and all exFAT patents.”
- deleted 4y ago[deleted]
- slavik81 4y agoThe patent will also expire in 2027 [1]. We can look forward to it being entirely unencumbered at that point. https://patents.google.com/patent/US20090164440?oq=US2009164440 https://patents.google.com/patent/US20090164440?oq=US2009164...
- ksec 4y agoI sometimes wonder if companies could choose to expire their patents earlier. Especially in cases when there are little to no strategic value to uphold them, but lots of potential value to unlock when they are gone.
- loeg 4y agoGood for OIN, but it doesn't help non-Linux systems.
- phkahler 4y agoThe question I have is, why Segger? When I saw this I was like "the debugger company?!?!" Clearly this wouldn't fall under their business, so it makes sense for them to open it up, but why did they do it in the first place?
- matja 4y agoThey have an entire RTOS ecosystem which supports a gazillon different microcontrollers.
- weinzierl 4y agoThey offer their own file system implementation (emFile) which supports either their own storage format (EFS) or FAT. The BigFAT article is posted in the emFile section of their website. My suspicion is that customers are bugging them to support large files in emFile and they don't want to pay the license fee for exFAT. I think they even can't do that with their current licensing model, which is one-time per product (not item) or product-family payment. EDIT: I tried to find out if Microsoft's exFAT is licensed per product or per unit and I found that it used to be a 300000 USD flat fee in 2009 but seems to be free since 2019. So my theory from above has no basis and I wonder why Segger does not simply implement exFAT?
- noAnswer 4y agoIt is not true that is "free since 2019". Source: https://www.microsoft.com/en-us/legal/intellectualproperty/tech-licensing/programs https://www.microsoft.com/en-us/legal/intellectualproperty/t... https://www.paragon-software.com/exfat-license/ https://www.paragon-software.com/exfat-license/ https://en.wikipedia.org/wiki/ExFAT#Legal_status https://en.wikipedia.org/wiki/ExFAT#Legal_status You maybe of the hook if you use Linux >= 5.7. And it seem that you are of the hook if you are a member of the Open Invention Network (OIN). But SEGGER's embOS is not based on Linux and their costumers a OEMs themselves. So their costumers would need to be OIN members or pay royalties to MS.
- weinzierl 4y ago
- scohesc 4y agoWould it not be possible to create a filesystem with modern capabilities but with backwards compatibility with FAT? Why can't we just have "legacy" commands built into the ReFS filesystem that process any FAT filesystem access? I'm very ignorant to this but I'd love some insight from someone vastly more knowledgeable than me.
- tenebrisalietum 4y agoA filesystem translates a filename-based streaming I/O API to the way a disk talks, which is "Read/write 512 times X bytes of data at disk block N." The I/O API or "commands" are the same; different filesystems will implement it differently.
- scohesc 4y agoThanks! I wasn't aware of the intricacies of this stuff - I'll have to do some more reading in my free time.
- quickthrower2 4y agoIs the big file handling seemless? If not why not just split files and use regular FAT32. And what about converting FAT32 to a linux partition? Or buy a new disk and move data over to that. Edit: it is a genuine question. downvote implies not but honestly it is.
- bArray 4y agoIs this only compatible with FAT32, or is it also compatible with FAT12/16? It would be very cool if this would support floppy disks. Regarding the format, once you convert it, does the target device need to have a driver to support the format? It mentions that this would allow for > 4GB files for TVs, but these are typically non-updated very out of date OSes. I think MS missed a trick by not making the boot sector also contain a simplistic driver, although it would have been a push to keep it all down at 512 bytes.
- Dylan16807 4y ago> Is this only compatible with FAT32, or is it also compatible with FAT12/16? It would be very cool if this would support floppy disks. It's simple enough to work on basically anything, but for what purpose? The max file size on FAT12/16 is the same as the max drive size. And FAT32 is very easy to implement for any system dealing with multiple megabytes.
- bArray 4y agoAs I replied to the comment, it's about having a filesystem that scales from floppy disks to hard drives. This is quite important for hobby OSes.
- pantalaimon 4y agoHow do you get files > 4 GiB on a floppy disk?
- bArray 4y agoYou don't of course, it's about having a filesystem that works from floppy disk all the way to hard drive.
- stuaxo 4y agoIf they want it to spread they should also write a fuse implementation and think about operating system support for Linux or BSD.
- sampa 4y agoIf only they released it back when exFAT was released. Now it has no future.
- tekchip 4y agoHow is this not BackFAT?