4 ms·

> For example, lets consider that we wrote to the sector 1024 on the SDCARD, and this address got initially mapped to physical sector 1024 itself on flash. Late

> For example, lets consider that we wrote to the sector 1024 on the SDCARD, and this address got initially mapped to physical sector 1024 itself on flash. Later we overwrite the contents, but this time write cannot go to the same physical address because it needs to be erased before programmed again. Maybe someone here will know the answer to something I've been wondering for a while: When appending to a file with writes smaller than a sector, does this remapping process trigger every time? Or is the controller smart enough that it can append without remapping, even though to overwrite would need remapping? Obviously for an SD card used for saving photos or videos you could write a sector at a time - but for an SD card in a temperature logger or similar, your appends might be a lot smaller.

st_goliath 2y ago
Pages that are already written cannot be overwritten, at least not without a full block erase, so it has to do a remap to append data. Directly logging to an SD card (i.e. lots of small writes) is the fastest way to destroy it.
- michaelt 2y ago
  As I understand it, you need a block erase to change bits from 0 to 1 then you can selectively set bits back from 1 to 0. Couldn't you perform a file append without the need for a block erase? By keeping the unwritten area as all ones until written?
  - st_goliath 2y ago
    You cannot do single byte writes on NAND. You have erase blocks (typically several 100k to M range) that are divided into pages (typically a few to several dozen k), some devices have sub-pages but that's about it, that's the smallest unit of data you can write. If you attempt to clear bits from 1 to 0 on an already written page, it will generally not work. I say generally, because I have actually tried this using raw NAND flash on an embedded Linux device. I found a chip, where it did work, but not very reliably. I also had an MLC chip, where it did not work, but instead caused random bit-flips on seemingly unrelated pages. I used this trick to systematically hammer down on the pages and figure out the pairings (only document in the NDA version of the datasheet), but failed to reproduce this on a different, more expensive MLC NAND. But even if appending in this fashion did somehow work, you would still need to update a data structure somewhere else anyway, telling you the exact amount of data written (including cases where you actually append 0xff bytes).
    - kragen 2y ago
      thank you so much! this is something i've been wondering about for years, and figuring it out obviously took you a huge amount of work normally with raw nand you use ecc, right? was there any ecc happening in your tests, either on the nand chip or in the linux driver? it seems like if you started with a page with valid ecc data and then tried to bash an arbitrary payload 1 bit in it to 0, you'd have a real challenge updating the ecc data to be consistent with the updated payload
      - st_goliath 2y ago
        > normally with raw nand you use ecc, right? yes, reasonably priced ones have an internal ECC engine, very cheap ones don't. The Linux NAND framwork has software ECC engines for those (see also: previous discussion on HN[1]). > was there any ecc happening in your tests? I deliberately turned it off. Linux has an ioctl for that. If the chip supports it, you can read/write without ECC or even directly into the OOB area where ECC data would normally be stored. The problem with the more expensive MLC NAND was that it had some smarts built in. It could discover overwrite attempts, as well as certain write patterns and apply a scrambling mask to the bits, so that the stored bits would (hopefully) not diffuse over to their neighbors. > if you started with a page with valid ecc data and then tried to bash an arbitrary payload 1 bit in it to 0, you'd have a real challenge updating the ecc data to be consistent with the updated payload If you are implementing a raw flash driver, there is a "nandbiterrs" test that does roughly what you describe: it bashes an increasing number of bit errors into a page and reads it back with ECC re-enabled, checking how many errors are successfully corrected during read, before the ECC engine gives up. [1] https://news.ycombinator.com/item?id=38361139 https://news.ycombinator.com/item?id=38361139
        kragen 2y ago
        thank you very much; this is priceless wisdom unfortunately the daunting task in front of me is to do this without linux or gcc, because what i want is a self-hosted operating system on a microcontroller that happens not to have an mmu or enough ram for gcc. my experience with sd cards in the past has been depressingly terrible reliability so i bought some slc nand chips with a public datasheet and without built-in ecc logic
  - marcosdumay 2y ago
    AFAIK, it's not clear how much difference in wear a series of writes cause when compared to writes and erases. I do know that very old flash firmwares applied that idea, and that new flash evolved in a direction where the wear caused by an erase is closer to the one caused by a write. If modern firmwares do not use it, it's probably because there isn't a lot to gain.
  - wmf 2y ago
    Probably not for a variety of reasons. The SSD itself does not know about files or appends; only block reads and writes. If the OS fills the unused space with zeros then it's going to write a block with trailing zeros, not ones. Most SSDs also scramble or encrypt data before writing to flash.
- 01HNNWZ0MV43FF 2y ago
  That could explain why my dashcam is working fine after ten years whereas my previous job's IoT devices regularly ate SD cards. (Then again, the IoT devices were also overheating badly whereas the dashcam was designed for its purpose, kept cool while running, and runs on a low duty cycle) What are you supposed to do, then? Buffer up writes and accept that you'll lose logs a minute or so before a power outage? As much hell as we had with those SD cards I'd rather put tiny iPod spinners in them...
  - marcosdumay 2y ago
    Losing the logs on a power outage may be a good compromise. Alternatively, you can buffer on a battery-powered RAM unit, accept that your cards will be short-lived, immediately send the logs through some network, not log... When you are designing the hardware, you have lots and lots of choices.
    - 01HNNWZ0MV43FF 2y ago
      Yeah. This company wasn't big enough to entertain battery backups, and the product was not expensive enough to add hardware that didn't bring in more sales. And we logged too much to pull all the logs over WAN all the time, resulting in a clever hack for remotely browsing logs with (relative) security Honestly we could have batched up the logs.
  - antonkochubey 2y ago
    There are industrial microSD's with pSLC memory, e.g. Kingston's SDCIT2 line. Those are significantly more durable - in a few years of using them I haven't managed to kill a single one. Much pricier per-GB, though.
- goodburb 2y ago
  I've seen some flash drives that prepare a pool of free AU blocks which is common with Phison controllers They prepare and erase a new pool only when it's full causing bursts in speeds and long periods of unresponsiveness (10 sec) while writing, reading is blocked as well. You can see it as a zigzag in file transfer speed graphs. This is not to be confused with OS buffer flushing. While I couldn't find any official info for the reasons behind it, it's assumed that the erase speed is slow on low quality flash and hence the controller has to pool in order to erase pages in parallel, increasing speed when copying small files between computers on plug and unplug. In my case it was about 500MB before before freezing on a 128GB flash drive.
mystified5016 2y ago
It sort of depends, but in general flash cannot write a page without erasing it first. Sometimes you can set, but not un-set bits (can be a 0 or 1 depending, usually 0 is 'set') without erasing a page. At the lowest level this means to update a page you have to first read it all back out, apply your modifications, then write back in a single operation. You must always write the full page size. In high level contexts like SD cards or SSDs being driven by an operating system, all of this is handled transparently in the background. If you're using raw flash in an embedded context, you usually have to manage this yourself (or let a library do it for you).

st_goliath 2y ago

Pages that are already written cannot be overwritten, at least not without a full block erase, so it has to do a remap to append data. Directly logging to an SD card (i.e. lots of small writes) is the fastest way to destroy it.

michaelt 2y ago
As I understand it, you need a block erase to change bits from 0 to 1 then you can selectively set bits back from 1 to 0. Couldn't you perform a file append without the need for a block erase? By keeping the unwritten area as all ones until written?
- st_goliath 2y ago
  You cannot do single byte writes on NAND. You have erase blocks (typically several 100k to M range) that are divided into pages (typically a few to several dozen k), some devices have sub-pages but that's about it, that's the smallest unit of data you can write. If you attempt to clear bits from 1 to 0 on an already written page, it will generally not work. I say generally, because I have actually tried this using raw NAND flash on an embedded Linux device. I found a chip, where it did work, but not very reliably. I also had an MLC chip, where it did not work, but instead caused random bit-flips on seemingly unrelated pages. I used this trick to systematically hammer down on the pages and figure out the pairings (only document in the NDA version of the datasheet), but failed to reproduce this on a different, more expensive MLC NAND. But even if appending in this fashion did somehow work, you would still need to update a data structure somewhere else anyway, telling you the exact amount of data written (including cases where you actually append 0xff bytes).
  - kragen 2y ago
    thank you so much! this is something i've been wondering about for years, and figuring it out obviously took you a huge amount of work normally with raw nand you use ecc, right? was there any ecc happening in your tests, either on the nand chip or in the linux driver? it seems like if you started with a page with valid ecc data and then tried to bash an arbitrary payload 1 bit in it to 0, you'd have a real challenge updating the ecc data to be consistent with the updated payload
    - st_goliath 2y ago
      > normally with raw nand you use ecc, right? yes, reasonably priced ones have an internal ECC engine, very cheap ones don't. The Linux NAND framwork has software ECC engines for those (see also: previous discussion on HN[1]). > was there any ecc happening in your tests? I deliberately turned it off. Linux has an ioctl for that. If the chip supports it, you can read/write without ECC or even directly into the OOB area where ECC data would normally be stored. The problem with the more expensive MLC NAND was that it had some smarts built in. It could discover overwrite attempts, as well as certain write patterns and apply a scrambling mask to the bits, so that the stored bits would (hopefully) not diffuse over to their neighbors. > if you started with a page with valid ecc data and then tried to bash an arbitrary payload 1 bit in it to 0, you'd have a real challenge updating the ecc data to be consistent with the updated payload If you are implementing a raw flash driver, there is a "nandbiterrs" test that does roughly what you describe: it bashes an increasing number of bit errors into a page and reads it back with ECC re-enabled, checking how many errors are successfully corrected during read, before the ECC engine gives up. [1] https://news.ycombinator.com/item?id=38361139 https://news.ycombinator.com/item?id=38361139
      - kragen 2y ago
        thank you very much; this is priceless wisdom unfortunately the daunting task in front of me is to do this without linux or gcc, because what i want is a self-hosted operating system on a microcontroller that happens not to have an mmu or enough ram for gcc. my experience with sd cards in the past has been depressingly terrible reliability so i bought some slc nand chips with a public datasheet and without built-in ecc logic
- marcosdumay 2y ago
  AFAIK, it's not clear how much difference in wear a series of writes cause when compared to writes and erases. I do know that very old flash firmwares applied that idea, and that new flash evolved in a direction where the wear caused by an erase is closer to the one caused by a write. If modern firmwares do not use it, it's probably because there isn't a lot to gain.
- wmf 2y ago
  Probably not for a variety of reasons. The SSD itself does not know about files or appends; only block reads and writes. If the OS fills the unused space with zeros then it's going to write a block with trailing zeros, not ones. Most SSDs also scramble or encrypt data before writing to flash.
01HNNWZ0MV43FF 2y ago
That could explain why my dashcam is working fine after ten years whereas my previous job's IoT devices regularly ate SD cards. (Then again, the IoT devices were also overheating badly whereas the dashcam was designed for its purpose, kept cool while running, and runs on a low duty cycle) What are you supposed to do, then? Buffer up writes and accept that you'll lose logs a minute or so before a power outage? As much hell as we had with those SD cards I'd rather put tiny iPod spinners in them...
- marcosdumay 2y ago
  Losing the logs on a power outage may be a good compromise. Alternatively, you can buffer on a battery-powered RAM unit, accept that your cards will be short-lived, immediately send the logs through some network, not log... When you are designing the hardware, you have lots and lots of choices.
  - 01HNNWZ0MV43FF 2y ago
    Yeah. This company wasn't big enough to entertain battery backups, and the product was not expensive enough to add hardware that didn't bring in more sales. And we logged too much to pull all the logs over WAN all the time, resulting in a clever hack for remotely browsing logs with (relative) security Honestly we could have batched up the logs.
- antonkochubey 2y ago
  There are industrial microSD's with pSLC memory, e.g. Kingston's SDCIT2 line. Those are significantly more durable - in a few years of using them I haven't managed to kill a single one. Much pricier per-GB, though.
goodburb 2y ago
I've seen some flash drives that prepare a pool of free AU blocks which is common with Phison controllers They prepare and erase a new pool only when it's full causing bursts in speeds and long periods of unresponsiveness (10 sec) while writing, reading is blocked as well. You can see it as a zigzag in file transfer speed graphs. This is not to be confused with OS buffer flushing. While I couldn't find any official info for the reasons behind it, it's assumed that the erase speed is slow on low quality flash and hence the controller has to pool in order to erase pages in parallel, increasing speed when copying small files between computers on plug and unplug. In my case it was about 500MB before before freezing on a 128GB flash drive.

mystified5016 2y ago

It sort of depends, but in general flash cannot write a page without erasing it first. Sometimes you can set, but not un-set bits (can be a 0 or 1 depending, usually 0 is 'set') without erasing a page. At the lowest level this means to update a page you have to first read it all back out, apply your modifications, then write back in a single operation. You must always write the full page size. In high level contexts like SD cards or SSDs being driven by an operating system, all of this is handled transparently in the background. If you're using raw flash in an embedded context, you usually have to manage this yourself (or let a library do it for you).