4 ms·

You cannot do single byte writes on NAND. You have erase blocks (typically several 100k to M range) that are divided into pages (typically a few to several doze

You cannot do single byte writes on NAND. You have erase blocks (typically several 100k to M range) that are divided into pages (typically a few to several dozen k), some devices have sub-pages but that's about it, that's the smallest unit of data you can write. If you attempt to clear bits from 1 to 0 on an already written page, it will generally not work. I say generally, because I have actually tried this using raw NAND flash on an embedded Linux device. I found a chip, where it did work, but not very reliably. I also had an MLC chip, where it did not work, but instead caused random bit-flips on seemingly unrelated pages. I used this trick to systematically hammer down on the pages and figure out the pairings (only document in the NDA version of the datasheet), but failed to reproduce this on a different, more expensive MLC NAND. But even if appending in this fashion did somehow work, you would still need to update a data structure somewhere else anyway, telling you the exact amount of data written (including cases where you actually append 0xff bytes).

kragen 2y ago
thank you so much! this is something i've been wondering about for years, and figuring it out obviously took you a huge amount of work normally with raw nand you use ecc, right? was there any ecc happening in your tests, either on the nand chip or in the linux driver? it seems like if you started with a page with valid ecc data and then tried to bash an arbitrary payload 1 bit in it to 0, you'd have a real challenge updating the ecc data to be consistent with the updated payload
- st_goliath 2y ago
  > normally with raw nand you use ecc, right? yes, reasonably priced ones have an internal ECC engine, very cheap ones don't. The Linux NAND framwork has software ECC engines for those (see also: previous discussion on HN[1]). > was there any ecc happening in your tests? I deliberately turned it off. Linux has an ioctl for that. If the chip supports it, you can read/write without ECC or even directly into the OOB area where ECC data would normally be stored. The problem with the more expensive MLC NAND was that it had some smarts built in. It could discover overwrite attempts, as well as certain write patterns and apply a scrambling mask to the bits, so that the stored bits would (hopefully) not diffuse over to their neighbors. > if you started with a page with valid ecc data and then tried to bash an arbitrary payload 1 bit in it to 0, you'd have a real challenge updating the ecc data to be consistent with the updated payload If you are implementing a raw flash driver, there is a "nandbiterrs" test that does roughly what you describe: it bashes an increasing number of bit errors into a page and reads it back with ECC re-enabled, checking how many errors are successfully corrected during read, before the ECC engine gives up. [1] https://news.ycombinator.com/item?id=38361139 https://news.ycombinator.com/item?id=38361139
  - kragen 2y ago
    thank you very much; this is priceless wisdom unfortunately the daunting task in front of me is to do this without linux or gcc, because what i want is a self-hosted operating system on a microcontroller that happens not to have an mmu or enough ram for gcc. my experience with sd cards in the past has been depressingly terrible reliability so i bought some slc nand chips with a public datasheet and without built-in ecc logic
SideQuark 2y ago
>> Couldn't you perform a file append without the need for a block erase? > You cannot do single byte writes on NAND. This isn't true - that the OP wrote here certainly does work (at least for many places I've tested and used it). I've used it to do all sorts of low level logging in small devices (and I've thoroughly tested many, many chips to complete destruction to see how things degrade). I have not yet seen a place it fails and I've done it on dozens or projects. I guess there may be places wear leveling fights with this, but if you dig into the particular device you have you can usually find low level details to make it work. The places I've pushed it are all commercial projects for which we characterize long term failure by running the flash parts fast enough to destroy the flash, to ensure that once in the field the expected lifetimes meet any guarantees we need to provide. Also, if you've never ran a flash chip to death, try it - it's interesting. Get a small SD card and hammer it withRead/write cycles until you start getting errors, then log and watch how those errors propagate. Good stuff :) Every chip I've tested does indeed let you erase a block (which sets all bits to 0 or 1 depending on chip), then append bits to that block by changing the bits you need. To append, you don't need to update some other data structure, just append a single bit of the correct type, then a bit scan of the page tells you where the data ends. Or, if you have another page telling how many "slots" of yourData are used, again append s single bit. Then small appends only cost a bit. On top of this implement wear leveling (unless the chip does - most bigger ones do, but some low level devices don't). Other projects I've worked on professionally are products for desoldering flash from captured and destroyed enemy devices which are then read raw and reconstructed, fault tolerant and error shielding for NASA space ops with flash, reverse engineering proprietary flash protocols for Secret Service (weird, right? but they pay) uses. I somewhat often get called for consulting on precisely these types of issues. So I do know a bit about low level flash abuse :)
- st_goliath 2y ago
  I was talking about bare NAND flash, not eMMC or anything otherwise "fancy" with builtin FTL. What you say is certainly true for those. None of the projects you describe sound "cheap" or like they involve devices first-and-foremost optimized for cost, and are unlikely to involve such dirt-cheap, raw NAND devices that are becoming increasingly rare even in the super-cheap embedded space. The original context of the discussion (see: a few comments up) is on-die NAND flash, inside SD cards, which TFA is about. The flash chips I brought up for comparison were from an extremely cost-optimized, embedded device that I worked on ~2016. The more expensive one already being on the upper end, regarding smarts & cost, but still exposing the flash directly (plus ECC & bad block management, but no FTL). I mostly work with industrial devices, and also maintain the Linux user space for raw flash (mtd-utils). I have seen my fair share of broken flash devices over the years (often in-field) and also have a my own little pile of broken industrial SD cards on my desk :)

kragen 2y ago

thank you so much! this is something i've been wondering about for years, and figuring it out obviously took you a huge amount of work normally with raw nand you use ecc, right? was there any ecc happening in your tests, either on the nand chip or in the linux driver? it seems like if you started with a page with valid ecc data and then tried to bash an arbitrary payload 1 bit in it to 0, you'd have a real challenge updating the ecc data to be consistent with the updated payload

st_goliath 2y ago
> normally with raw nand you use ecc, right? yes, reasonably priced ones have an internal ECC engine, very cheap ones don't. The Linux NAND framwork has software ECC engines for those (see also: previous discussion on HN[1]). > was there any ecc happening in your tests? I deliberately turned it off. Linux has an ioctl for that. If the chip supports it, you can read/write without ECC or even directly into the OOB area where ECC data would normally be stored. The problem with the more expensive MLC NAND was that it had some smarts built in. It could discover overwrite attempts, as well as certain write patterns and apply a scrambling mask to the bits, so that the stored bits would (hopefully) not diffuse over to their neighbors. > if you started with a page with valid ecc data and then tried to bash an arbitrary payload 1 bit in it to 0, you'd have a real challenge updating the ecc data to be consistent with the updated payload If you are implementing a raw flash driver, there is a "nandbiterrs" test that does roughly what you describe: it bashes an increasing number of bit errors into a page and reads it back with ECC re-enabled, checking how many errors are successfully corrected during read, before the ECC engine gives up. [1] https://news.ycombinator.com/item?id=38361139 https://news.ycombinator.com/item?id=38361139
- kragen 2y ago
  thank you very much; this is priceless wisdom unfortunately the daunting task in front of me is to do this without linux or gcc, because what i want is a self-hosted operating system on a microcontroller that happens not to have an mmu or enough ram for gcc. my experience with sd cards in the past has been depressingly terrible reliability so i bought some slc nand chips with a public datasheet and without built-in ecc logic

SideQuark 2y ago

>> Couldn't you perform a file append without the need for a block erase? > You cannot do single byte writes on NAND. This isn't true - that the OP wrote here certainly does work (at least for many places I've tested and used it). I've used it to do all sorts of low level logging in small devices (and I've thoroughly tested many, many chips to complete destruction to see how things degrade). I have not yet seen a place it fails and I've done it on dozens or projects. I guess there may be places wear leveling fights with this, but if you dig into the particular device you have you can usually find low level details to make it work. The places I've pushed it are all commercial projects for which we characterize long term failure by running the flash parts fast enough to destroy the flash, to ensure that once in the field the expected lifetimes meet any guarantees we need to provide. Also, if you've never ran a flash chip to death, try it - it's interesting. Get a small SD card and hammer it withRead/write cycles until you start getting errors, then log and watch how those errors propagate. Good stuff :) Every chip I've tested does indeed let you erase a block (which sets all bits to 0 or 1 depending on chip), then append bits to that block by changing the bits you need. To append, you don't need to update some other data structure, just append a single bit of the correct type, then a bit scan of the page tells you where the data ends. Or, if you have another page telling how many "slots" of yourData are used, again append s single bit. Then small appends only cost a bit. On top of this implement wear leveling (unless the chip does - most bigger ones do, but some low level devices don't). Other projects I've worked on professionally are products for desoldering flash from captured and destroyed enemy devices which are then read raw and reconstructed, fault tolerant and error shielding for NASA space ops with flash, reverse engineering proprietary flash protocols for Secret Service (weird, right? but they pay) uses. I somewhat often get called for consulting on precisely these types of issues. So I do know a bit about low level flash abuse :)

st_goliath 2y ago
I was talking about bare NAND flash, not eMMC or anything otherwise "fancy" with builtin FTL. What you say is certainly true for those. None of the projects you describe sound "cheap" or like they involve devices first-and-foremost optimized for cost, and are unlikely to involve such dirt-cheap, raw NAND devices that are becoming increasingly rare even in the super-cheap embedded space. The original context of the discussion (see: a few comments up) is on-die NAND flash, inside SD cards, which TFA is about. The flash chips I brought up for comparison were from an extremely cost-optimized, embedded device that I worked on ~2016. The more expensive one already being on the upper end, regarding smarts & cost, but still exposing the flash directly (plus ECC & bad block management, but no FTL). I mostly work with industrial devices, and also maintain the Linux user space for raw flash (mtd-utils). I have seen my fair share of broken flash devices over the years (often in-field) and also have a my own little pile of broken industrial SD cards on my desk :)