4 ms·

I understand the rationale of having a relatively simple filesystem specification so that it can be implemented under the constraints of embedded systems, but u

by matja 4y ago

I understand the rationale of having a relatively simple filesystem specification so that it can be implemented under the constraints of embedded systems, but using a filesystem that has no journal or checksums on the cheapest and flakiest of storage devices (hi, bottom-of-the-barrel SD cards and USB memory sticks) seems a recipe for disaster in terms of data resiliency. I use ZFS on SD cards and USB memory sticks when I am able to, and the number of checksum errors are ... not rare. Even for high-quality NVMe SSDs on (lower quality) USB-NVMe bridge controllers, corruption everywhere.

ciupicri 4y ago
Doesn't most flash memory already have some builtin error detection and possibly correction logic already?
- est31 4y ago
  It is supposed to have, but apparently cheap flash storage does not have this logic, or it's not working. That's what GP is referring to.
- Spooky23 4y ago
  Flash sticks are like $2 wholesale parts. The packaging probably costs more than the product in many cases. Every expense has been spared - I wouldn’t trust these things at all unless you are using one of the more expensive SSD devices or something certified for Windows ToGo, etc.
- jeffbee 4y ago
  All flash memory devices are wall-to-wall error correction. It's basically math on a stick. The raw media error rates are 1% to 10% which are crazy high. That flash works at all is a miracle of linear algebra.
M95D 4y ago
Actually FAT filesystems with two FAT entries can be implemented in a transaction safe way: 1) write allocations on first FAT, 2) write file data, 3) write the second FAT. This is the same as a journal that stores metadata only (what most other journaled filesystems do). This is probably what Microsoft calls "TexFAT", but doesn't describe in any way. Good written data (not corrupted while being written) will last a long time, probably more than any magnetic media, so data resiliency is good.
- OneLeggedCat 4y ago
  > Good written data (not corrupted while being written) will last a long time, probably more than any magnetic media, so data resiliency is good Are you meaning generally any kind of solid-state storage here?
  - M95D 4y ago
    I'm not sure about TLC and QLC, but SLC and MLC seems very good so far.
- kevin_thibedeau 4y ago
  > will last a long time, probably more than any magnetic media, so data resiliency is good. Flash storage will not last as long as magnetic media. You've got 50 years, tops, and even then it requires some luck to not have any bit rot.
  - M95D 4y ago
    > Flash storage will not last as long as magnetic media. You've got 50 years, tops, and even then it requires some luck to not have any bit rot. Which magnetic media lasted 50 years? I don't belive HDDs from the '70s are readable today. I tried to read a QIC-120 last year (2021). The tape was dated 1992 (I think), so it was almost 30 years old. The tape jammed inside the cartrige.
  - hulitu 4y ago
    50 years, for flash ? without wear leveling ? Do you have some links ? Over 10 years for flash is difficult.
rwmj 4y ago
The real issue that the article only hints at is Windows support. Windows only supports a small handful of filesystems (FAT*, ExFAT, NTFS, ReFS, ISO9660 & UDF - I think that's it!). So if you have to interoperate with Windows then you have to use one of those filesystems. If you don't need to interoperate with Windows then you have many better choices.
- ChuckNorris89 4y ago
  > Windows only supports a small handful of filesystems ( [...] If you don't need to interoperate with Windows then you have many better choices Are the filesystems choices mare varied if you need to operate with MacOS? Whenever my dad brings me one of his FAT32 USB drives with "issues", I find a bunch of hidden dot files and folders that were not there before and realize it's been plugged in a Mac. I really would like to know what MacOS does to USB flash drives.
  - GekkePrutser 4y ago
    The dotfiles are for metadata that the non-Mac filesystems can't store. Coloured labels, icon position, longer filenames, creator info etc. Windows actually does almost the same thing. Thumbs.db etc. And it also creates shadow files for those with metadata when saving to FAT32. You just don't see them on windows as they are hidden, just like dotfiles are on macOS.
    - Dylan16807 4y ago
      The last version of windows that made thumbs.db was XP. And what metadata do you mean? Permissions and alternate streams get purged, don't they? What else is there? You have to go out of your way to get a desktop.ini and those are just as obtrusive on NTFS. Long file names are stored in a weird way but that's very standardized and doesn't make extra files.
      - panzi 4y ago
        There's still the "System Volume Information" folder. Whatever that is. (I use Linux, but see it on USB drives that where plugged into Windows machines.)
pantalaimon 4y ago
You can also get ext4 (with journalling) rather small so it's usable on an MCU. https://github.com/gkostka/lwext4 https://github.com/gkostka/lwext4

by matja 4y ago

ciupicri 4y ago

Doesn't most flash memory already have some builtin error detection and possibly correction logic already?

est31 4y ago
It is supposed to have, but apparently cheap flash storage does not have this logic, or it's not working. That's what GP is referring to.
Spooky23 4y ago
Flash sticks are like $2 wholesale parts. The packaging probably costs more than the product in many cases. Every expense has been spared - I wouldn’t trust these things at all unless you are using one of the more expensive SSD devices or something certified for Windows ToGo, etc.
jeffbee 4y ago
All flash memory devices are wall-to-wall error correction. It's basically math on a stick. The raw media error rates are 1% to 10% which are crazy high. That flash works at all is a miracle of linear algebra.

M95D 4y ago

Actually FAT filesystems with two FAT entries can be implemented in a transaction safe way: 1) write allocations on first FAT, 2) write file data, 3) write the second FAT. This is the same as a journal that stores metadata only (what most other journaled filesystems do). This is probably what Microsoft calls "TexFAT", but doesn't describe in any way. Good written data (not corrupted while being written) will last a long time, probably more than any magnetic media, so data resiliency is good.

OneLeggedCat 4y ago
> Good written data (not corrupted while being written) will last a long time, probably more than any magnetic media, so data resiliency is good Are you meaning generally any kind of solid-state storage here?
- M95D 4y ago
  I'm not sure about TLC and QLC, but SLC and MLC seems very good so far.
kevin_thibedeau 4y ago
> will last a long time, probably more than any magnetic media, so data resiliency is good. Flash storage will not last as long as magnetic media. You've got 50 years, tops, and even then it requires some luck to not have any bit rot.
- M95D 4y ago
  > Flash storage will not last as long as magnetic media. You've got 50 years, tops, and even then it requires some luck to not have any bit rot. Which magnetic media lasted 50 years? I don't belive HDDs from the '70s are readable today. I tried to read a QIC-120 last year (2021). The tape was dated 1992 (I think), so it was almost 30 years old. The tape jammed inside the cartrige.
- hulitu 4y ago
  50 years, for flash ? without wear leveling ? Do you have some links ? Over 10 years for flash is difficult.

rwmj 4y ago

The real issue that the article only hints at is Windows support. Windows only supports a small handful of filesystems (FAT*, ExFAT, NTFS, ReFS, ISO9660 & UDF - I think that's it!). So if you have to interoperate with Windows then you have to use one of those filesystems. If you don't need to interoperate with Windows then you have many better choices.

ChuckNorris89 4y ago
> Windows only supports a small handful of filesystems ( [...] If you don't need to interoperate with Windows then you have many better choices Are the filesystems choices mare varied if you need to operate with MacOS? Whenever my dad brings me one of his FAT32 USB drives with "issues", I find a bunch of hidden dot files and folders that were not there before and realize it's been plugged in a Mac. I really would like to know what MacOS does to USB flash drives.
- GekkePrutser 4y ago
  The dotfiles are for metadata that the non-Mac filesystems can't store. Coloured labels, icon position, longer filenames, creator info etc. Windows actually does almost the same thing. Thumbs.db etc. And it also creates shadow files for those with metadata when saving to FAT32. You just don't see them on windows as they are hidden, just like dotfiles are on macOS.
  - Dylan16807 4y ago
    The last version of windows that made thumbs.db was XP. And what metadata do you mean? Permissions and alternate streams get purged, don't they? What else is there? You have to go out of your way to get a desktop.ini and those are just as obtrusive on NTFS. Long file names are stored in a weird way but that's very standardized and doesn't make extra files.
    - panzi 4y ago
      There's still the "System Volume Information" folder. Whatever that is. (I use Linux, but see it on USB drives that where plugged into Windows machines.)

pantalaimon 4y ago

You can also get ext4 (with journalling) rather small so it's usable on an MCU. https://github.com/gkostka/lwext4 https://github.com/gkostka/lwext4