4 ms·

Is there any Linux compatible file system which will be mounted redundantly without any special mount options? So that for instance a 64 gigabyte would have a

Is there any Linux compatible file system which will be mounted redundantly without any special mount options? So that for instance a 64 gigabyte would have a filesystem with 32 gigabyte space but with a lot of redundancy? I know there are many ways to achieve similar outcomes but all I know of relies on knowing beforehand how to mount the thing.

londons_explore 2y ago
Unfortunately because of the way the SD card - computer interface works, storing everything twice on the same card doesn't actually get you much more reliability. The card is unaware of which bits of data are duplicates, and therefore can easily end up storing both copies of some data on areas of the physical flash that are weak - and this is even more likely considering they will probably be written around the same timestamp.
- kragen 2y ago
  i've heard some cards do duplicate data detection? that way they can avoid storing multiple copies of the same data, so for example copying a large file is fast
  - londons_explore 2y ago
    Possible, but I haven't heard the same. Sometimes copying a large file feels fast because the OS has a huge cache, and it'll just take a long time in the background later when you unmount the disk.
    - kragen 2y ago
      that is definitely a thing that happens
vbezhenar 2y ago
mdraid? Create two partitions and build a raid on top of them. Or any of advanced filesystems with built-in raid capabilities like bcachefs, btrfs or zfs.
- ssl-3 2y ago
  Using mdraid requires special considerations when mounting in the future, doesn't it?
ssl-3 2y ago
ZFS can do this. Just set copies=2 on a dataset, and it'll always keep 2 copies of everything written thenceforth (with ~twice the storage usage). And since it's a dataset setting instead of a pool setting, it can be applied selectively on a storage device. "Oh, those photos are also backed up to Google Photos. It's maybe not ideal, but it's a lot of data and not worth keeping 2 copies those locally, so I'll leave those at the default of copies=1. But the this collection of code and short stories and I've written? That's pretty important to me, and it's actually kind of small -- I'll set that as copies=2. Actually, maybe copies=3 would be even better..." The setting will stick with it between machines/sessions/whatevers without further user input. Things with copies=2 get mounted the same way as anything else is with ZFS in a given environment. ZFS already detects bitrot very well even with copies=1, but it can't fix bitrot with chechsums alone. With copies=2, it has a non-zero chance at actually fixing it (or at least being completely readable so the data can be transitioned to the next storage device, completely intact). https://docs.oracle.com/cd/E19253-01/819-5461/gevpg/index.html https://docs.oracle.com/cd/E19253-01/819-5461/gevpg/index.ht...
- account42 2y ago
  Is there even that much value to keeping multiple copies on the same device? I fail to see what this adds over backups which can be done with any filesystem.
  - HankB99 2y ago
    If you have a pool consisting of a single device (such as a drive in a laptop) the second copy could protect from loss of a file that includes storage that can not be read. In a multiple device pool (with some redundancy) it would IMO be a extra and unneeded layer of protection.
  - ssl-3 2y ago
    RAID is good (if one pays for it and makes it work). Backups are better (if they actually get done, and if they are actually usable for recovery). Backups + RAID is better yet. But OP asked for a particular set of things that do not have any direct relationship to the traditional concepts of either backups or RAID, and ZFS can provide that set of things using copies=2. It isn't for us to decide the value of this for them.

londons_explore 2y ago

Unfortunately because of the way the SD card - computer interface works, storing everything twice on the same card doesn't actually get you much more reliability. The card is unaware of which bits of data are duplicates, and therefore can easily end up storing both copies of some data on areas of the physical flash that are weak - and this is even more likely considering they will probably be written around the same timestamp.

kragen 2y ago
i've heard some cards do duplicate data detection? that way they can avoid storing multiple copies of the same data, so for example copying a large file is fast
- londons_explore 2y ago
  Possible, but I haven't heard the same. Sometimes copying a large file feels fast because the OS has a huge cache, and it'll just take a long time in the background later when you unmount the disk.
  - kragen 2y ago
    that is definitely a thing that happens

vbezhenar 2y ago

mdraid? Create two partitions and build a raid on top of them. Or any of advanced filesystems with built-in raid capabilities like bcachefs, btrfs or zfs.

ssl-3 2y ago
Using mdraid requires special considerations when mounting in the future, doesn't it?

ssl-3 2y ago

ZFS can do this. Just set copies=2 on a dataset, and it'll always keep 2 copies of everything written thenceforth (with ~twice the storage usage). And since it's a dataset setting instead of a pool setting, it can be applied selectively on a storage device. "Oh, those photos are also backed up to Google Photos. It's maybe not ideal, but it's a lot of data and not worth keeping 2 copies those locally, so I'll leave those at the default of copies=1. But the this collection of code and short stories and I've written? That's pretty important to me, and it's actually kind of small -- I'll set that as copies=2. Actually, maybe copies=3 would be even better..." The setting will stick with it between machines/sessions/whatevers without further user input. Things with copies=2 get mounted the same way as anything else is with ZFS in a given environment. ZFS already detects bitrot very well even with copies=1, but it can't fix bitrot with chechsums alone. With copies=2, it has a non-zero chance at actually fixing it (or at least being completely readable so the data can be transitioned to the next storage device, completely intact). https://docs.oracle.com/cd/E19253-01/819-5461/gevpg/index.html https://docs.oracle.com/cd/E19253-01/819-5461/gevpg/index.ht...

account42 2y ago
Is there even that much value to keeping multiple copies on the same device? I fail to see what this adds over backups which can be done with any filesystem.
- HankB99 2y ago
  If you have a pool consisting of a single device (such as a drive in a laptop) the second copy could protect from loss of a file that includes storage that can not be read. In a multiple device pool (with some redundancy) it would IMO be a extra and unneeded layer of protection.
- ssl-3 2y ago
  RAID is good (if one pays for it and makes it work). Backups are better (if they actually get done, and if they are actually usable for recovery). Backups + RAID is better yet. But OP asked for a particular set of things that do not have any direct relationship to the traditional concepts of either backups or RAID, and ZFS can provide that set of things using copies=2. It isn't for us to decide the value of this for them.