9 ms·
The thinking behind the 32GB Windows Format limit on FAT32
- woliveirajr 6y agoTL;DR: "simple lack of foresight combined with the age-old problem of the temporary solution becoming de-facto permanent"
- simonblack 6y agoExactly. Just another case of "640K should be enough for anybody". But we have to admit that storage has leapt hugely in size and very rapidly. In 1990 or thereabouts, a common hard disk size was about 30 megabytes. Today I have files that are many times that size, and we think that that is quite normal. A mere five or so years later around 1996, that size had jumped to 4000 megabytes! These days my laptop has 6 million megabytes. We can't blame filesystem planners for lack of foresight. They were blind-sided by progress just like the rest of us were.
- mnw21cam 6y agoI have a single file that is 6 million megabytes in size.
- Thristle 6y agowho/what reads that file?
- mnw21cam 6y agotar
- AnIdiotOnTheNet 6y agoThat stretches the definition of "a single file" when it is basically a filesystem itself.
- bilekas 6y agoNot necessarily, it's all relative really. But the point is, the limitations or expected limitations are only relevant until they're not.. Also why not tar a drive for archive if I want :}
- jcranmer 6y agoI haven't managed a 6TB file before, but I did have a >100GB log file I ended up parsing with awk just to be sure that the memory consumption on the consuming process wouldn't go crazy.
- petschge 6y agoNot my file, but I have produced HDF5 files in that size range using simulation codes on supercomputers and paraview can open them and render 3d view as video from it (eventually).
- ben509 6y agoHe had already seen exponential growth in storage capacity and could easily extrapolate from that. Moore's law was coined in the '60s, after all. This wasn't a lack of foresight, this was being prudent. If I can't come close to testing an upper limit, it's correct to acknowledge it may degrade in a way I haven't anticipated. Especially since he was writing a GUI which should generally avoid handing users footguns, and which can be updated over time.
- jandrese 6y agoMore directly: He only expected his code to be used in Windows NT and made the fatal flaw of thinking that just because he slapped it together in an hour it would be easy for someone to rewrite in the next version of Windows to up the limit. Instead the code sat untouched for decades as the limit slowly became more and more unreasonable. IMHO the real issue is the artificial limit. Instead of straight up disallowing you from creating large filesystems, it should have had a dialog box warning about small files being inefficient but letting you make it anyway.
- jackric 6y ago>Instead the code sat untouched for decades as the limit slowly became more and more unreasonable. My projects likely won't span decades but when I know I'm putting in some opinionated value I leave behind a comment like: // XXX picking 32gb because it'll probably be 5 years until we have >32gb hard drives. Change this when the world changes!
- m000 6y agoTIL: In 2019, Microsoft released the exFAT [1] specs to a large group of vendors [2], including prominent Linux vendors. I.e. exFAT should now be preferred over FAT/NTFS for media used across different operating systems. [1] https://en.wikipedia.org/wiki/ExFAT https://en.wikipedia.org/wiki/ExFAT [2] https://en.wikipedia.org/wiki/Open_Invention_Network https://en.wikipedia.org/wiki/Open_Invention_Network
- gruez 6y ago>TIL: In 2019, Microsoft released the exFAT [1] specs [...] I.e. exFAT should now be preferred over FAT/NTFS for media used across different operating systems. You seem to be very optimistic about how fast these get implemented into embedded devices.
- netsharc 6y agoI have a 2011 car which has a USB port where I can plug a USB stick with MP3s (remember those, kids?) on them, I doubt it will ever be made ExFAT-compatible. OTOH, I do have the Windows logo on my steering wheel, and somewhere inside the car there's a Windows CE system installed (it doesn't have a multi-color LCD screen and interacts using speech recognition, so it's not really obvious).
- bilekas 6y agoI would imagine if embedded devices are being assembled and configured with filesize concerns, then exFAT would be the natural implementation they would opt for. Or do you mean updating/transitioning legacy embedded? The spec doesn't mention backwards compat, but does mention as a goal >Retain the simplicity of FAT-based file systems. I would like to test out if an input embed system is exFAT compat. And what might be required to update.
- wongarsu 6y agoEmbedded devices often have a lifespan measured in decades. And for short-lived devices like consumer all-in-one printers (~4 years) it's not like every new model has a new firmware written from scratch. They get whatever the last model had, with a few updates for new device features. That might include exFAT, or it might not.
- bluedays 6y agoThe original video this article is referencing is here if anyone wants to just watch it instead: https://www.youtube.com/watch?v=bikbJPI-7Kg https://www.youtube.com/watch?v=bikbJPI-7Kg
- dbt00 6y agoThe choice he made was probably a reasonable balance -- the alternative was reinventing the filesystem, which was probably a bit out of spec for implementing the GUI format dialog. And eventually happened, but it took a while.
- Thristle 6y agoIts not reinventing. Its just setting a limit on max cluster slack. Like he says in the video, you can totally use the CLI and format any disk size you want, its just a soft block
- hocuspocus 6y agoIf my memory serves well, when 40+ GB HDDs started being common, you were either using Windows 2000 and NTFS already, or it was probably a good practice to have several partitions, given the overall unreliability of Windows 9x and the not so infrequent clean reinstallations. And then came Windows XP and the issue was moot. It's a bit of a shame Microsoft didn't push and open up exFAT earlier and allowed FAT32 to become an issue again, only with removable flash drives and memory cards this time.
- AnIdiotOnTheNet 6y agoThe problem with removable drives is that no one came up with a better filesystem than FAT32 that could be implemented easily by everyone. All the *NIX filesystems embedded permissions that were useless on non-POSIX systems and caused issues on POSIX ones when you actually moved a drive from one computer to another. So MS extended FAT again and kept it under its IP hat, as was their wont[0] at the time. [0] Non-native english speakers may be confused by this: https://www.merriam-webster.com/dictionary/as%20is%20someone%27s%20wont https://www.merriam-webster.com/dictionary/as%20is%20someone...
- josefx 6y agoFAT embedded a long/short name mapping algorithm that a modern system shouldn't need, but Microsoft had patents on. The main feature of FAT is simple native support on all platforms, Microsoft never had a financial incentive to add support for alternatives on Windows.
- kasabali 6y ago> It's a bit of a shame Microsoft didn't push and open up exFAT earlier and allowed FAT32 to become an issue again, only with removable flash drives and memory cards this time Erm, because that was the idea? SDXC is a big racket and Microsoft has been collecting patent fees from any kind of consumer device that's capable of reading memory cards thanks to the filesystem requirements in the standard.
- rlv-dan 6y agoHe made a decision and assumed when making future NT versions this would be revised and updated with the times. I can only image that future teams looked at this and thought "someone chose 32GB for a reason, better not change it"...
- Zardoz84 6y agoWell, Windows 95/98/ME, 2000 and XP shared the same old font installer that Windows 3.11 . I wonder why no body changed it before.
- macjohnmcc 6y agoLikely if it ain't broke don't fix it.
- Delk 6y agoMaybe, but it could also be about just not getting around to doing it. Changes, especially in a large system with a large user base, require extensive testing, and even small changes may come with a lot of red tape. I don't know how things work at Microsoft, but I wouldn't be surprised if making changes to the UI for a technology that's being superseded anyway just didn't make anybody's list.
- TwoNineFive 6y agoIn other words, he failed to comment his code adequately.
- creeble 6y agoWhat is the practical difference between exFAT and FAT32 when you can format FAT32 up to many GB? Is it just that they are different formats, and exFAT has taken over, in terms of pre-formatted USB media?
- BlueTemplar 6y agoFAT32 also has a 4 Go file size limit (think large video/VM files, DVD/Blu-ray images…)
- ike77 6y agoMaybe it's not for the worst, as otherwise there might have been less push for exFAT and this inefficiency would have persisted much later..?
- wazoox 6y agoActually NT4.0 did not support FAT32 at all (unless you installed a little-known driver from Sysinternals I've never actually seen used in the wild). So maybe his work was on Windows 2000?
- greghendershott 6y agoI rarely read The Register. Is that the usual tone -- as if the author wishes they were Lady Whistledown, the gossip writer on Bridgerton?
- tym0 6y agoIt is, it's similar to Private Eye in the UK or Le Canard enchaîné in France.
- arp242 6y agoYes, pretty much.
- QuercusMax 6y agoMaybe with a little Variety mixed in, but you're not far off the mark. Definitely in the standard British vein of snarky tabloid writing.
- selimnairb 6y agoHopefully exFAT implementations will get more robust. At least with the macOS implementation, it has always been prone to reporting corruption when devices are not ejected properly. I can usually fix it by running check disk or whatever on Windows, but sometimes not.
- MrDOS 6y agoI've had more filesystem corruption trouble with exFAT on Linux than with any other filesystem, ever, including Btrfs. Really crossing my fingers and hoping for the quality of the Samsung driver in Linux 5.7. It's going to be great to have an option for high-capacity, cross-platform read/write storage. The day I never have to use NTFS-3G again cannot come too soon.
- pantalaimon 6y ago> The day I never have to use NTFS-3G again cannot come too soon. That day might be soon https://lkml.org/lkml/2020/12/31/126 https://lkml.org/lkml/2020/12/31/126
- MrDOS 6y agoI know everyone talks about 2020 having felt like a long year, but I had totally forgotten that Paragon had started trying to submit that back in August[0]. I'm really glad to see this is progressing. What a monumental set of changes. This won't change the NTFS situation on macOS, of course but even in a post-exFAT world, NTFS will remain a relevant concern for Linux/Windows dual-booting, and I'd love to see this get merged. [0]: https://lore.kernel.org/linux-fsdevel/2911ac5cd20b46e397be506268718d74@paragon-software.com/ https://lore.kernel.org/linux-fsdevel/2911ac5cd20b46e397be50...
- kasabali 6y agoWere you using the FUSE driver?
- MrDOS 6y ago
- doser 6y agoMany embedded devices can read FAT32 and that's it. If you need to use a drive larger than 32GB for these, there are tools that will let you format them as FAT32 from within Windows. One example: FAT32Format GUI - https://www.softpedia.com/get/System/Hard-Disk-Utils/FAT32format-GUI.shtml https://www.softpedia.com/get/System/Hard-Disk-Utils/FAT32fo...
- BuildTheRobots 6y agoI hit this a couple of days ago when helping a friend set up their new DJ decks. The decks only supported FAT32 but would happily read a 128gb USB stick. The article states that you can format as fat32 from the command line - this wasn't my experience. Disk Util (cli) saw my raw partition and started formatting it as FAT32, but seemed to fail about an hour in saying the disk was too big. I ended up finding a random abandoned GUI utility which seemed to work, though it's extremely infuriating.
- fomine3 6y agoI suspect the command the article said is diskpart.
- kasabali 6y agoIt also conveniently allowed Microsoft to collect license fees from ExFAT patents on all SDXC devices.
- tpoacher 6y ago"Prototypes have a bad habit of becoming the product".