4 ms·

A small remark for the casual readers: This is not about SSDs, this is for raw NAND flash access. On an embedded device, you might have a piece of NAND flash d

A small remark for the casual readers: This is not about SSDs, this is for raw NAND flash access. On an embedded device, you might have a piece of NAND flash directly attached to the CPU e.g. via a parallel bus or SPI. Usually with a controller somewhere in between, or integrated in the SoCs. In Linux, there is a subsystem called MTD (memory technology devices) for dealing with not-quite-block storage that provides a thin hardware abstraction layer with an internal driver framework for NAND flash and backends for all sorts of controllers. Some particularly cheap flash chips/controller do not support hardware bit-error correction. There are IIRC 2 software engines that a driver can use as a fallback: hamming and BCH[1] based (the former is documented here). The MTD subsystem is really just a thin abstraction layer tough and provides a uniform API for e.g. page read/write & block erase for the flash chip it talks to. There is another subsystem called UBI (Unsorted Block Images) that can be stacked on top of MTD. It takes care of wear-leveling, bad block management and implements LVM-style logical volume partitioning. It also does not emulate a block device, but acts more like a flash device with idealized properties. There are 2 filesystems in Linux which are designed to deal with the weirdness of those devices: the older JFFS2, which stacks directly on top of MTD, and UBIFS on top of UBI. [1] https://en.wikipedia.org/wiki/BCH_code https://en.wikipedia.org/wiki/BCH_code

anthk 3y ago
This was used in the PocketCHIP for instance. They changed SLC to MLC and they error ratio got much smaller. AKA shit happened if you let the device shutdown by hardware due to low battery usage, you would loss lots of files.
vlovich123 3y ago
It’s interesting to me that Linux doesn’t have support for putting ext/xfs/… on top of MTD/UBI devices (i.e. making wear leveling more independent). I imagine many people installing on normal SSDs with built-in controllers use XFS. Not sure why we don’t see more raw NAND w/ a basic PCI-e controller with Linux doing all the wear leveling & whatnot given the problems vendors have flushing. You could even have an attached M3 that Linux could load companion software on for accelerating performance to make it even more on par with traditional SSDs
- bradfa 3y ago
  You can use a normal file system on top of UBI. There's some rather large caveats, however: http://www.linux-mtd.infradead.org/doc/ubi.html#L_ubiblock http://www.linux-mtd.infradead.org/doc/ubi.html#L_ubiblock
- duskwuff 3y ago
  > It’s interesting to me that Linux doesn’t have support for putting ext/xfs/… on top of MTD/UBI devices Raw NAND has a number of characteristics which are incredibly unpleasant for a traditional filesystem to deal with, including: 1) No small writes. You usually have to program a whole page at a time, which is often larger than a sector on a hard disk (~16 KiB). Filesystems which are accustomed to being able to flush single-byte writes to disk will be very disappointed by this. 2) No overwriting. Once a page has been written to, you can't write to it again until it's been erased. This makes even emulating small writes difficult. 3) Erase is a big hammer. It operates on a large (32 - 128 MiB) group of pages and erases them all at once. 4) Bad sectors. Sometimes an erase block is just bad on chip and you have to work around it. And, as flash wears out, blocks can go bad, and you'll just have to hope there's a spare available. Bottom line is -- if you want to write a filesystem that works with flash directly, you have to write it to work that way from the start. Modifying a traditional filesystem like ext4 to run on flash would effectively be a complete rewrite.
  - SkeuomorphicBee 3y ago
    Out of curiosity, are you sure about #1 ? I all because a long long time ago I worked with some raw NAND on embedded devices and I remember doing small writes, in fact I could do single byte writes (write only, the erase was in big blocks as you said).
    - Deeg9rie9usi 3y ago
      On NAND there is the NOP property (Number of programs). For SLC it is usually 4. So, you can program a single page in up to 4 steps on SLC NAND. If you write less than that, you're wasting space because you need to jump to the next (sub)page. Re-programming is not allowed on NAND.
dmitrygr 3y ago
> There are 2 filesystems in Linux which are designed to deal with the weirdness of those devices You forgot yaffs2 which was commonly used in the early days of android and runs on top of MTD
- Deeg9rie9usi 3y ago
  yaffs2 was never upstream.
o11c 3y ago
That said, it should be noted that there are still USB flash drives in the wild that do no error detection. It's a pity that `badblocks` is so poorly supported these days ...
- supertrope 3y ago
  I thought cheap flash devices are only usable products because of error correction. USB flash drives and memory cards use reject tier NAND and compensate by pre-marking bad sectors and heavy ECC. Your files exist as a probabilistic approximation.
  - o11c 3y ago
    I suppose it's possible that they simply do not enough error corruption, but given that I don't get IO errors, only wrong data, I doubt it. Typical algorithms work like: * 1 bad bit is corrected * 2 bad bits produce an error. Since there's nothing else meaningful to do, this can only propagate to the higher layer. * 3 or more bad bits may produce wrong data or an error so we wouldn't expect to hit case 3 without hitting case 2.

anthk 3y ago

This was used in the PocketCHIP for instance. They changed SLC to MLC and they error ratio got much smaller. AKA shit happened if you let the device shutdown by hardware due to low battery usage, you would loss lots of files.

vlovich123 3y ago

It’s interesting to me that Linux doesn’t have support for putting ext/xfs/… on top of MTD/UBI devices (i.e. making wear leveling more independent). I imagine many people installing on normal SSDs with built-in controllers use XFS. Not sure why we don’t see more raw NAND w/ a basic PCI-e controller with Linux doing all the wear leveling & whatnot given the problems vendors have flushing. You could even have an attached M3 that Linux could load companion software on for accelerating performance to make it even more on par with traditional SSDs

bradfa 3y ago
You can use a normal file system on top of UBI. There's some rather large caveats, however: http://www.linux-mtd.infradead.org/doc/ubi.html#L_ubiblock http://www.linux-mtd.infradead.org/doc/ubi.html#L_ubiblock
duskwuff 3y ago
> It’s interesting to me that Linux doesn’t have support for putting ext/xfs/… on top of MTD/UBI devices Raw NAND has a number of characteristics which are incredibly unpleasant for a traditional filesystem to deal with, including: 1) No small writes. You usually have to program a whole page at a time, which is often larger than a sector on a hard disk (~16 KiB). Filesystems which are accustomed to being able to flush single-byte writes to disk will be very disappointed by this. 2) No overwriting. Once a page has been written to, you can't write to it again until it's been erased. This makes even emulating small writes difficult. 3) Erase is a big hammer. It operates on a large (32 - 128 MiB) group of pages and erases them all at once. 4) Bad sectors. Sometimes an erase block is just bad on chip and you have to work around it. And, as flash wears out, blocks can go bad, and you'll just have to hope there's a spare available. Bottom line is -- if you want to write a filesystem that works with flash directly, you have to write it to work that way from the start. Modifying a traditional filesystem like ext4 to run on flash would effectively be a complete rewrite.
- SkeuomorphicBee 3y ago
  Out of curiosity, are you sure about #1 ? I all because a long long time ago I worked with some raw NAND on embedded devices and I remember doing small writes, in fact I could do single byte writes (write only, the erase was in big blocks as you said).
  - Deeg9rie9usi 3y ago
    On NAND there is the NOP property (Number of programs). For SLC it is usually 4. So, you can program a single page in up to 4 steps on SLC NAND. If you write less than that, you're wasting space because you need to jump to the next (sub)page. Re-programming is not allowed on NAND.

dmitrygr 3y ago

> There are 2 filesystems in Linux which are designed to deal with the weirdness of those devices You forgot yaffs2 which was commonly used in the early days of android and runs on top of MTD

Deeg9rie9usi 3y ago
yaffs2 was never upstream.

o11c 3y ago

That said, it should be noted that there are still USB flash drives in the wild that do no error detection. It's a pity that `badblocks` is so poorly supported these days ...

supertrope 3y ago
I thought cheap flash devices are only usable products because of error correction. USB flash drives and memory cards use reject tier NAND and compensate by pre-marking bad sectors and heavy ECC. Your files exist as a probabilistic approximation.
- o11c 3y ago
  I suppose it's possible that they simply do not enough error corruption, but given that I don't get IO errors, only wrong data, I doubt it. Typical algorithms work like: * 1 bad bit is corrected * 2 bad bits produce an error. Since there's nothing else meaningful to do, this can only propagate to the higher layer. * 3 or more bad bits may produce wrong data or an error so we wouldn't expect to hit case 3 without hitting case 2.