4 ms·

I am really disappointed by SD card reliability. It seems even a little wear and the whole thing slows down massively and eventually throws read/write errors

I am really disappointed by SD card reliability. It seems even a little wear and the whole thing slows down massively and eventually throws read/write errors. I would like to see a card design which, instead of failing when there is flash wear, instead just gets smaller. For compatibility with existing OS's, that would take the form of a self-partition-resizing sd card. It would understand fat32, ext3, NTFS, afs etc, and when the card is next powered up the partition would be slightly smaller and files physically at the 'end' would be moved inwards. For newer OS's, a new API could be introduced which tells the OS 'this card is smaller now, please give me some blocks to mark as unavailable'. The now-smaller card can use the removed-and-worn-out blocks to store error correction data for the remaining blocks. Ie. Additional error correction data on top of the data already stored within each page. That effectively dramatically increases the lifespan of each page, at the cost of reduced IO performance.

actionfromafar 2y ago
Is there any Linux compatible file system which will be mounted redundantly without any special mount options? So that for instance a 64 gigabyte would have a filesystem with 32 gigabyte space but with a lot of redundancy? I know there are many ways to achieve similar outcomes but all I know of relies on knowing beforehand how to mount the thing.
- londons_explore 2y ago
  Unfortunately because of the way the SD card - computer interface works, storing everything twice on the same card doesn't actually get you much more reliability. The card is unaware of which bits of data are duplicates, and therefore can easily end up storing both copies of some data on areas of the physical flash that are weak - and this is even more likely considering they will probably be written around the same timestamp.
  - kragen 2y ago
    i've heard some cards do duplicate data detection? that way they can avoid storing multiple copies of the same data, so for example copying a large file is fast
    - londons_explore 2y ago
      Possible, but I haven't heard the same. Sometimes copying a large file feels fast because the OS has a huge cache, and it'll just take a long time in the background later when you unmount the disk.
      - kragen 2y ago
        that is definitely a thing that happens
- vbezhenar 2y ago
  mdraid? Create two partitions and build a raid on top of them. Or any of advanced filesystems with built-in raid capabilities like bcachefs, btrfs or zfs.
  - ssl-3 2y ago
    Using mdraid requires special considerations when mounting in the future, doesn't it?
- ssl-3 2y ago
  ZFS can do this. Just set copies=2 on a dataset, and it'll always keep 2 copies of everything written thenceforth (with ~twice the storage usage). And since it's a dataset setting instead of a pool setting, it can be applied selectively on a storage device. "Oh, those photos are also backed up to Google Photos. It's maybe not ideal, but it's a lot of data and not worth keeping 2 copies those locally, so I'll leave those at the default of copies=1. But the this collection of code and short stories and I've written? That's pretty important to me, and it's actually kind of small -- I'll set that as copies=2. Actually, maybe copies=3 would be even better..." The setting will stick with it between machines/sessions/whatevers without further user input. Things with copies=2 get mounted the same way as anything else is with ZFS in a given environment. ZFS already detects bitrot very well even with copies=1, but it can't fix bitrot with chechsums alone. With copies=2, it has a non-zero chance at actually fixing it (or at least being completely readable so the data can be transitioned to the next storage device, completely intact). https://docs.oracle.com/cd/E19253-01/819-5461/gevpg/index.html https://docs.oracle.com/cd/E19253-01/819-5461/gevpg/index.ht...
  - account42 2y ago
    Is there even that much value to keeping multiple copies on the same device? I fail to see what this adds over backups which can be done with any filesystem.
    - HankB99 2y ago
      If you have a pool consisting of a single device (such as a drive in a laptop) the second copy could protect from loss of a file that includes storage that can not be read. In a multiple device pool (with some redundancy) it would IMO be a extra and unneeded layer of protection.
    - ssl-3 2y ago
      RAID is good (if one pays for it and makes it work). Backups are better (if they actually get done, and if they are actually usable for recovery). Backups + RAID is better yet. But OP asked for a particular set of things that do not have any direct relationship to the traditional concepts of either backups or RAID, and ZFS can provide that set of things using copies=2. It isn't for us to decide the value of this for them.
goodburb 2y ago
Maybe there are more lemons with SD cards. The fact that they're used in dash cams, exposed to high temperature thermal cycles, and recording tens of terabytes of data which can be more than the typical consumer HDD/SSD in it's lifetime is quite impressive.
- weweweoo 2y ago
  Recently I found a long lost go-pro style camera using metal detector, after it had fallen off an RC plane, which crashed into a tree in dense forest. Over nearly a decade the camera had been buried several centimeters into the ground, and the card inside the camera wasn't particularly well protected (just an open slot, not waterproof or anything), which made it exposed to lots of water and freezing winters for nearly a decade. After some cleaning the SD card worked perfectly, and all the data was intact. It even had the crash recorded in it. I was impressed by that. The camera didn't survive though.
  - Scoutmaster 2y ago
    I recommend everyone take a picture of their contact information with their digital cameras, that way if it gets lost and found, people know who to at least send the pictures to if the camera is destroyed. I imagine the RC plane's owner would get a real kick if years later the camera was found and they got the footage back. (I used to work at Philmont Scout Ranch and would do a lot of detective work to get people their cameras back.)
deleted 2y ago
[deleted]
ploxiln 2y ago
SD Cards are just so damn cheap. Less than $20 USD for a 128GB microsd is unbelievable (cost and size). Only the most premium of the premium have a chance of surviving significant random or small writes, it seems. "Ultra" isn't quite enough, we need "Extreme Pro"? (Needing to figure out the ranking of superlatives is silly.) There's "Max Endurance" or "Pro Endurance" now, sounds promising ... but those are barely any more expensive, so I'm not sure how much different they could be ... You can get Swissbit pSLC (pseudo-SLC) SD cards from component distributers like Mouser and Digikey, and they are much more expensive, but should be "industrial grade". I've recently purchased a couple of their 8GB microsd cards for about $30 each (but haven't put them through a lot of real usage yet ...)
- londons_explore 2y ago
  Reliability is mostly just better software that does better write aggregation, wear levelling and error correction. Develop it once, and you can make as many SD cards as you like reliable for no extra money...

actionfromafar 2y ago

Is there any Linux compatible file system which will be mounted redundantly without any special mount options? So that for instance a 64 gigabyte would have a filesystem with 32 gigabyte space but with a lot of redundancy? I know there are many ways to achieve similar outcomes but all I know of relies on knowing beforehand how to mount the thing.

londons_explore 2y ago
Unfortunately because of the way the SD card - computer interface works, storing everything twice on the same card doesn't actually get you much more reliability. The card is unaware of which bits of data are duplicates, and therefore can easily end up storing both copies of some data on areas of the physical flash that are weak - and this is even more likely considering they will probably be written around the same timestamp.
- kragen 2y ago
  i've heard some cards do duplicate data detection? that way they can avoid storing multiple copies of the same data, so for example copying a large file is fast
  - londons_explore 2y ago
    Possible, but I haven't heard the same. Sometimes copying a large file feels fast because the OS has a huge cache, and it'll just take a long time in the background later when you unmount the disk.
    - kragen 2y ago
      that is definitely a thing that happens
vbezhenar 2y ago
mdraid? Create two partitions and build a raid on top of them. Or any of advanced filesystems with built-in raid capabilities like bcachefs, btrfs or zfs.
- ssl-3 2y ago
  Using mdraid requires special considerations when mounting in the future, doesn't it?
ssl-3 2y ago
ZFS can do this. Just set copies=2 on a dataset, and it'll always keep 2 copies of everything written thenceforth (with ~twice the storage usage). And since it's a dataset setting instead of a pool setting, it can be applied selectively on a storage device. "Oh, those photos are also backed up to Google Photos. It's maybe not ideal, but it's a lot of data and not worth keeping 2 copies those locally, so I'll leave those at the default of copies=1. But the this collection of code and short stories and I've written? That's pretty important to me, and it's actually kind of small -- I'll set that as copies=2. Actually, maybe copies=3 would be even better..." The setting will stick with it between machines/sessions/whatevers without further user input. Things with copies=2 get mounted the same way as anything else is with ZFS in a given environment. ZFS already detects bitrot very well even with copies=1, but it can't fix bitrot with chechsums alone. With copies=2, it has a non-zero chance at actually fixing it (or at least being completely readable so the data can be transitioned to the next storage device, completely intact). https://docs.oracle.com/cd/E19253-01/819-5461/gevpg/index.html https://docs.oracle.com/cd/E19253-01/819-5461/gevpg/index.ht...
- account42 2y ago
  Is there even that much value to keeping multiple copies on the same device? I fail to see what this adds over backups which can be done with any filesystem.
  - HankB99 2y ago
    If you have a pool consisting of a single device (such as a drive in a laptop) the second copy could protect from loss of a file that includes storage that can not be read. In a multiple device pool (with some redundancy) it would IMO be a extra and unneeded layer of protection.
  - ssl-3 2y ago
    RAID is good (if one pays for it and makes it work). Backups are better (if they actually get done, and if they are actually usable for recovery). Backups + RAID is better yet. But OP asked for a particular set of things that do not have any direct relationship to the traditional concepts of either backups or RAID, and ZFS can provide that set of things using copies=2. It isn't for us to decide the value of this for them.

goodburb 2y ago

Maybe there are more lemons with SD cards. The fact that they're used in dash cams, exposed to high temperature thermal cycles, and recording tens of terabytes of data which can be more than the typical consumer HDD/SSD in it's lifetime is quite impressive.

weweweoo 2y ago
Recently I found a long lost go-pro style camera using metal detector, after it had fallen off an RC plane, which crashed into a tree in dense forest. Over nearly a decade the camera had been buried several centimeters into the ground, and the card inside the camera wasn't particularly well protected (just an open slot, not waterproof or anything), which made it exposed to lots of water and freezing winters for nearly a decade. After some cleaning the SD card worked perfectly, and all the data was intact. It even had the crash recorded in it. I was impressed by that. The camera didn't survive though.
- Scoutmaster 2y ago
  I recommend everyone take a picture of their contact information with their digital cameras, that way if it gets lost and found, people know who to at least send the pictures to if the camera is destroyed. I imagine the RC plane's owner would get a real kick if years later the camera was found and they got the footage back. (I used to work at Philmont Scout Ranch and would do a lot of detective work to get people their cameras back.)

deleted 2y ago

[deleted]

ploxiln 2y ago

SD Cards are just so damn cheap. Less than $20 USD for a 128GB microsd is unbelievable (cost and size). Only the most premium of the premium have a chance of surviving significant random or small writes, it seems. "Ultra" isn't quite enough, we need "Extreme Pro"? (Needing to figure out the ranking of superlatives is silly.) There's "Max Endurance" or "Pro Endurance" now, sounds promising ... but those are barely any more expensive, so I'm not sure how much different they could be ... You can get Swissbit pSLC (pseudo-SLC) SD cards from component distributers like Mouser and Digikey, and they are much more expensive, but should be "industrial grade". I've recently purchased a couple of their 8GB microsd cards for about $30 each (but haven't put them through a lot of real usage yet ...)

londons_explore 2y ago
Reliability is mostly just better software that does better write aggregation, wear levelling and error correction. Develop it once, and you can make as many SD cards as you like reliable for no extra money...