4 ms·

It’s interesting to me that Linux doesn’t have support for putting ext/xfs/… on top of MTD/UBI devices (i.e. making wear leveling more independent). I imagine m

It’s interesting to me that Linux doesn’t have support for putting ext/xfs/… on top of MTD/UBI devices (i.e. making wear leveling more independent). I imagine many people installing on normal SSDs with built-in controllers use XFS. Not sure why we don’t see more raw NAND w/ a basic PCI-e controller with Linux doing all the wear leveling & whatnot given the problems vendors have flushing. You could even have an attached M3 that Linux could load companion software on for accelerating performance to make it even more on par with traditional SSDs

bradfa 3y ago
You can use a normal file system on top of UBI. There's some rather large caveats, however: http://www.linux-mtd.infradead.org/doc/ubi.html#L_ubiblock http://www.linux-mtd.infradead.org/doc/ubi.html#L_ubiblock
duskwuff 3y ago
> It’s interesting to me that Linux doesn’t have support for putting ext/xfs/… on top of MTD/UBI devices Raw NAND has a number of characteristics which are incredibly unpleasant for a traditional filesystem to deal with, including: 1) No small writes. You usually have to program a whole page at a time, which is often larger than a sector on a hard disk (~16 KiB). Filesystems which are accustomed to being able to flush single-byte writes to disk will be very disappointed by this. 2) No overwriting. Once a page has been written to, you can't write to it again until it's been erased. This makes even emulating small writes difficult. 3) Erase is a big hammer. It operates on a large (32 - 128 MiB) group of pages and erases them all at once. 4) Bad sectors. Sometimes an erase block is just bad on chip and you have to work around it. And, as flash wears out, blocks can go bad, and you'll just have to hope there's a spare available. Bottom line is -- if you want to write a filesystem that works with flash directly, you have to write it to work that way from the start. Modifying a traditional filesystem like ext4 to run on flash would effectively be a complete rewrite.
- SkeuomorphicBee 3y ago
  Out of curiosity, are you sure about #1 ? I all because a long long time ago I worked with some raw NAND on embedded devices and I remember doing small writes, in fact I could do single byte writes (write only, the erase was in big blocks as you said).
  - Deeg9rie9usi 3y ago
    On NAND there is the NOP property (Number of programs). For SLC it is usually 4. So, you can program a single page in up to 4 steps on SLC NAND. If you write less than that, you're wasting space because you need to jump to the next (sub)page. Re-programming is not allowed on NAND.
  - duskwuff 3y ago
    Partial page programming was supported on some older SLC NAND, but is usually unsupported on modern MLC(/TLC/QLC) flash.
- vlovich123 3y ago
  > Modifying a traditional filesystem like ext4 to run on flash would effectively be a complete rewrite. I mean that’s patently incorrect though… All modern PCs are running on raw NAND with a flash controller mediating things to present as a legacy block device and ext/xfs/nags etc runs just fine this way. Heck, even Apple OSes, which actually afaik do have their own in-house flash controller, had HFS running on top of the translation layer rather than baking those requirements into the FS (I think this is also true for APFS but not 100% certain). So while everything you wrote about the complexities of raw NAND are true, it doesn’t follow that the filesystem MUST handle all that complexity itself. Something designed to take full advantage might perform better, but that’s always true and seeing cheaper storage solutions with reliable fsync might be a worthwhile trade off (+ you can always run a filesystem that’s better designed to accommodate the underlying HW details)
  - IAmLiterallyAB 3y ago
    This is true, Linux developers could write their own fully featured FTL. However, it's difficult. To quote the MTD docs "Unfortunately it is a rather difficult task to create a good FTL layer and nobody still managed to implement one for Linux."
gdgghhhhh 3y ago
To support a block based filesystem on top of MTD an advanced FTL (Flash Translation Layer) is needed. Sadly most algorithms for decent FTLs are patented. SSD vendors try hard keeping their magic sauce there own. In Linux we habe some trivial FTLs implemented, but even these are toxic.
- vlovich123 3y ago
  I suspect that while vendors would like you to believe it’s all patented secret sauce, I’m more skeptical. There’s too many vendors and they all perform roughly similarly with the controller chip speed and bus speed being more impactful. The FTL might be more impactful for disk longevity but I wouldn’t put much stock in consumer device numbers here. Even if Linux starts with an initially worse implementation, over time it should succeed because of cheaper costs multiplied by in aggregate a greater investment from stakeholders than any single private stakeholder could match. There’s a reason Apple and Amazon have written their own controllers and it’s largely because vendor solutions actually suck quite a bit when you start digging in.
  - Deeg9rie9usi 3y ago
    And neither Apple nor Google published their FTLs...
garaetjjte 3y ago
I think it is possible to create emulated block device stacked on UBI.

bradfa 3y ago

You can use a normal file system on top of UBI. There's some rather large caveats, however: http://www.linux-mtd.infradead.org/doc/ubi.html#L_ubiblock http://www.linux-mtd.infradead.org/doc/ubi.html#L_ubiblock

duskwuff 3y ago

> It’s interesting to me that Linux doesn’t have support for putting ext/xfs/… on top of MTD/UBI devices Raw NAND has a number of characteristics which are incredibly unpleasant for a traditional filesystem to deal with, including: 1) No small writes. You usually have to program a whole page at a time, which is often larger than a sector on a hard disk (~16 KiB). Filesystems which are accustomed to being able to flush single-byte writes to disk will be very disappointed by this. 2) No overwriting. Once a page has been written to, you can't write to it again until it's been erased. This makes even emulating small writes difficult. 3) Erase is a big hammer. It operates on a large (32 - 128 MiB) group of pages and erases them all at once. 4) Bad sectors. Sometimes an erase block is just bad on chip and you have to work around it. And, as flash wears out, blocks can go bad, and you'll just have to hope there's a spare available. Bottom line is -- if you want to write a filesystem that works with flash directly, you have to write it to work that way from the start. Modifying a traditional filesystem like ext4 to run on flash would effectively be a complete rewrite.

SkeuomorphicBee 3y ago
Out of curiosity, are you sure about #1 ? I all because a long long time ago I worked with some raw NAND on embedded devices and I remember doing small writes, in fact I could do single byte writes (write only, the erase was in big blocks as you said).
- Deeg9rie9usi 3y ago
  On NAND there is the NOP property (Number of programs). For SLC it is usually 4. So, you can program a single page in up to 4 steps on SLC NAND. If you write less than that, you're wasting space because you need to jump to the next (sub)page. Re-programming is not allowed on NAND.
- duskwuff 3y ago
  Partial page programming was supported on some older SLC NAND, but is usually unsupported on modern MLC(/TLC/QLC) flash.
vlovich123 3y ago
> Modifying a traditional filesystem like ext4 to run on flash would effectively be a complete rewrite. I mean that’s patently incorrect though… All modern PCs are running on raw NAND with a flash controller mediating things to present as a legacy block device and ext/xfs/nags etc runs just fine this way. Heck, even Apple OSes, which actually afaik do have their own in-house flash controller, had HFS running on top of the translation layer rather than baking those requirements into the FS (I think this is also true for APFS but not 100% certain). So while everything you wrote about the complexities of raw NAND are true, it doesn’t follow that the filesystem MUST handle all that complexity itself. Something designed to take full advantage might perform better, but that’s always true and seeing cheaper storage solutions with reliable fsync might be a worthwhile trade off (+ you can always run a filesystem that’s better designed to accommodate the underlying HW details)
- IAmLiterallyAB 3y ago
  This is true, Linux developers could write their own fully featured FTL. However, it's difficult. To quote the MTD docs "Unfortunately it is a rather difficult task to create a good FTL layer and nobody still managed to implement one for Linux."

gdgghhhhh 3y ago

To support a block based filesystem on top of MTD an advanced FTL (Flash Translation Layer) is needed. Sadly most algorithms for decent FTLs are patented. SSD vendors try hard keeping their magic sauce there own. In Linux we habe some trivial FTLs implemented, but even these are toxic.

vlovich123 3y ago
I suspect that while vendors would like you to believe it’s all patented secret sauce, I’m more skeptical. There’s too many vendors and they all perform roughly similarly with the controller chip speed and bus speed being more impactful. The FTL might be more impactful for disk longevity but I wouldn’t put much stock in consumer device numbers here. Even if Linux starts with an initially worse implementation, over time it should succeed because of cheaper costs multiplied by in aggregate a greater investment from stakeholders than any single private stakeholder could match. There’s a reason Apple and Amazon have written their own controllers and it’s largely because vendor solutions actually suck quite a bit when you start digging in.
- Deeg9rie9usi 3y ago
  And neither Apple nor Google published their FTLs...

garaetjjte 3y ago

I think it is possible to create emulated block device stacked on UBI.