7 ms·
SDcard wear leveling and translation layers (2014)
- deleted 2y ago[deleted]
- deleted 2y ago[deleted]
- st_goliath 2y agoAt a conference, I once met somebody who said his team had the joy of dismantling broken SD cards and figuring out why they broke. One of the anecdotes I distinctly recall was about a batch of cards that (for obvious cost saving reasons) used the actual data flash to store its own firmware as well. IIRC due to a bug in the wear leveling accounting, once the card got sufficiently full, the wear leveling code ended up partially overwriting the firmware itself.
- billpg 2y agoI recall once having an SD card with a mysterious file in the root folder with system and hidden attributes on it. I decided to delete the file and some time later (months), the card suddenly stopped working. It maybe stopped working once the OS decided the bytes were available for use (which it would only have done without the file marking that space as unavailable) and overwrote that section. I suspect defragmenting or otherwise moving the file about for whatever reason would have had the same effect.
- karamanolev 2y agoWouldn't a regular filesystem format of the card have the same effect - wipe the file? Wouldn't such SD cards be fatally flawed for most applications? My camera, drone, etc. all format the cards.
- billpg 2y agoI vaguely recall thinking the file was just some sort of serial number or stock tracking metadata when I deleted it. It might have instead failed for quite mundane reasons such as low quality manufacture. I don't know. Exposing the firmware in this fashion would be a bad idea for all of these reasons. It is a good thing that SD card manufacturers don't do this any more, if they ever did.
- vardump 2y agoFirmware definitely would not be on the filesystem layer or even accessible on the block device.
- 01HNNWZ0MV43FF 2y agoYou'd think yeah
- tiberious726 2y agoIt isn't. Before a chip has firmware loaded, it can't decode filesystems.
- 15155 2y agoEven after a chip has firmware loaded, which controller can make logical sense of a filesystem?
- goodburb 2y agoAccording to SDXC specs, the controller looks for exFAT markers to relocate the table and bitmap to smaller AU units. exfatprogs can pack bitmap with table for SDHC and lower: https://manpages.debian.org/experimental/exfatprogs/mkfs.exfat.8.en.html https://manpages.debian.org/experimental/exfatprogs/mkfs.exf...
- samatman 2y agoThat doesn't at all mean that the firmware can't be exposed through the filesystem. Is that a bad idea? Almost certainly. Is it possible? Absolutely yes.
- Sakos 2y agoMan, I'd love to work in that team for a year. I've had so many SD cards fail on me (most recently a 1TB SanDisk Extreme which was used in my Steam Deck) and it feels like most of them are junk.
- robotnikman 2y agoIIRC the denser the memory on the cards, the more likely problems are to arise. I forget the specifics but it has to do with the physics involving electronics on such a small scale. That's why industrial grade cards tend to be in much smaller capacities, but they are much longer lasting.
- Sakos 2y agoI wish we'd move away from microSD cards to a larger format if it meant more durable storage. I hate these things. I just end up keeping them in an SDcard adapter when they're not in active use so I don't lose them. USB sticks seem to have the same issue with durability though. I've also had multiple of those fail on me. I'm currently eyeing those SSD USB sticks and I wonder how much more durable they might be.
- robotnikman 2y agoThe regular SD card form factor seems great when it comes to more durable storage, but obviously would not be ideal for smaller devices. I've heard that many USB sticks use lower quality binned chips so that may be why they usually fail more often despite their size, but this is just anecdotal from what I've heard.
- chasil 2y agoI read elsewhere here that if you fdisk/gdisk a partition that is one third of the drive and limit yourself to that space, then the drive will exhibit the reliability of SLC media, and last far longer than rated.
- 2Gkashmiri 2y agoHave an SD card that was copying file on to and in the middle of transfer the card went kaput and hasnt seen since. No machine I try it on even acknowledges it existence. The card is almost a decade old at this point but is there a way to recover data ?
- robotnikman 2y agoBest to consult a data recovery specialist, will cost you an arm and a leg though. 10 years without power makes me think most of it wont be recoverable even if they can get to the data on it, but then again it depends on a lot of factors when it comes to flash memory.
- eschneider 2y agoSome amount of the data is almost certainly recoverable, but getting it done commercially is going to be $$$.
- goodburb 2y agoIIRC SD cards and most SSDs don't go through error check routine when powered, it has to be specified. Some do use the power-on time and run a routine if left on for too long. You usually you have to read the entire drive to maintain, in my case S.M.A.R.T. reported changes to reserve blocks after doing that on an old drive. I have a 16 year old 4GB SDcard with all data still intact.
- 2Gkashmiri 2y ago10 years in use. On a phone.
- mystified5016 2y agoSometimes you can recover data over the SPI interface instead of the proprietary SD interface. Essentially, plug it into an arduino with one of the many SD card libraries and slowly dump the data over the serial link. I'm unsure if this is just urban legend or not, though. I've never tried myself.
- londons_explore 2y agoI am really disappointed by SD card reliability. It seems even a little wear and the whole thing slows down massively and eventually throws read/write errors. I would like to see a card design which, instead of failing when there is flash wear, instead just gets smaller. For compatibility with existing OS's, that would take the form of a self-partition-resizing sd card. It would understand fat32, ext3, NTFS, afs etc, and when the card is next powered up the partition would be slightly smaller and files physically at the 'end' would be moved inwards. For newer OS's, a new API could be introduced which tells the OS 'this card is smaller now, please give me some blocks to mark as unavailable'. The now-smaller card can use the removed-and-worn-out blocks to store error correction data for the remaining blocks. Ie. Additional error correction data on top of the data already stored within each page. That effectively dramatically increases the lifespan of each page, at the cost of reduced IO performance.
- actionfromafar 2y agoIs there any Linux compatible file system which will be mounted redundantly without any special mount options? So that for instance a 64 gigabyte would have a filesystem with 32 gigabyte space but with a lot of redundancy? I know there are many ways to achieve similar outcomes but all I know of relies on knowing beforehand how to mount the thing.
- londons_explore 2y agoUnfortunately because of the way the SD card - computer interface works, storing everything twice on the same card doesn't actually get you much more reliability. The card is unaware of which bits of data are duplicates, and therefore can easily end up storing both copies of some data on areas of the physical flash that are weak - and this is even more likely considering they will probably be written around the same timestamp.
- kragen 2y agoi've heard some cards do duplicate data detection? that way they can avoid storing multiple copies of the same data, so for example copying a large file is fast
- michaelt 2y ago> For example, lets consider that we wrote to the sector 1024 on the SDCARD, and this address got initially mapped to physical sector 1024 itself on flash. Later we overwrite the contents, but this time write cannot go to the same physical address because it needs to be erased before programmed again. Maybe someone here will know the answer to something I've been wondering for a while: When appending to a file with writes smaller than a sector, does this remapping process trigger every time? Or is the controller smart enough that it can append without remapping, even though to overwrite would need remapping? Obviously for an SD card used for saving photos or videos you could write a sector at a time - but for an SD card in a temperature logger or similar, your appends might be a lot smaller.
- st_goliath 2y agoPages that are already written cannot be overwritten, at least not without a full block erase, so it has to do a remap to append data. Directly logging to an SD card (i.e. lots of small writes) is the fastest way to destroy it.
- michaelt 2y agoAs I understand it, you need a block erase to change bits from 0 to 1 then you can selectively set bits back from 1 to 0. Couldn't you perform a file append without the need for a block erase? By keeping the unwritten area as all ones until written?
- st_goliath 2y agoYou cannot do single byte writes on NAND. You have erase blocks (typically several 100k to M range) that are divided into pages (typically a few to several dozen k), some devices have sub-pages but that's about it, that's the smallest unit of data you can write. If you attempt to clear bits from 1 to 0 on an already written page, it will generally not work. I say generally, because I have actually tried this using raw NAND flash on an embedded Linux device. I found a chip, where it did work, but not very reliably. I also had an MLC chip, where it did not work, but instead caused random bit-flips on seemingly unrelated pages. I used this trick to systematically hammer down on the pages and figure out the pairings (only document in the NDA version of the datasheet), but failed to reproduce this on a different, more expensive MLC NAND. But even if appending in this fashion did somehow work, you would still need to update a data structure somewhere else anyway, telling you the exact amount of data written (including cases where you actually append 0xff bytes).
- faragon 2y agoWhy there's not generalized discard/trim capabilities on USB/SD devices? It should be "easy" to agree some standard on that.
- goodburb 2y agoSome A2 SD cards have command queueing for random read speed improvement and TRIM enabled. It requires kernel support and hardware acceleration being optional. https://forums.raspberrypi.com/viewtopic.php?t=367459 https://forums.raspberrypi.com/viewtopic.php?t=367459
- formerly_proven 2y agoEven more strangely, proprietary SxS/XQD cards also don't support discard/trim, neither for devices nor from a PC. Only the more recent CFX cards support trim (most of them, but not with every reader, and formatting in a camera doesn't seem to issue them, perhaps to aid with data recovery in case of accidental formats).
- radicality 2y agoAre CFExpress cards similar/comparable at all to SD cards? Afaik CFExpress uses nvme, which will provide more features. Also, interesting thing one can do if you have a need for a CFExpress card (like a camera) and want to save some money. You can buy an adapter online such as the linked one, and then buy an m2.2230 ssd and put it in there. That’s how I made myself a cheaper 1TB CFExpress B card for my camera, and it’s doing just fine, even with 8k30fps at ~3.5Gb/s. https://www.bhphotovideo.com/c/product/1685487-REG/monster_adapter_sa_cfe1_2230_m_2_ssd_to.html https://www.bhphotovideo.com/c/product/1685487-REG/monster_a...
- a-l-e-c 2y agoDoes anyone have access to the full specs of some SD cards? To get access to the full specifications it seems like you're forced to join the SD Association: 2500$ p/year
- riiii 2y agoI heard Anna's Archive might have SD Specifications.
- HeatrayEnjoyer 2y agoWhat is the point of charging for it, especially such an outrageous amount?
- ssl-3 2y agoThe SD Association is a non-profit organization, but that doesn't mean that they don't have expenses to pay and mouths to feed. It's $6.84 per day for a membership. If that's an outrageous amount to you, then: Please remember that you are absolutely free to direct your rage into creating your own competing portable flash storage systems, and to license them in any way that you wish. You may even include a free pony, if you choose. It's your world.