4 ms·

Use a filesystem that does end-to-end error detection and correction. I use ZFS on all of my USB drives (even "real" SATA and M.2 SSDs via adaptors) and it fre

by matja 4y ago

Use a filesystem that does end-to-end error detection and correction. I use ZFS on all of my USB drives (even "real" SATA and M.2 SSDs via adaptors) and it frequently finds checksum errors (and thanks to the design of ZFS, is able to completely recover without any risk of further corruption).

amelius 4y ago
Isn't any filesystem that runs on top of a SSD or flash drive at risk from the SSD/flash drive controller corrupting the disk or making it inaccessible, when the volume is not cleanly unmounted? I can remember losing a USB drive that way, even though it was using a journaling filesystem.
- paulmd 4y ago
  Potentially yes, realistically no. This is the old "if the storage controller is lying about flushing to disk, there's not a lot you can do about it" problem. ZFS does have defensive mechanisms, like doing a read after write to try and be sure that what is written is actually committed to disk, but if the storage controller chooses to serve that out of cache then that could be a lie too. It's the old "trusting trust" predicament, there's no way for hardware higher in the stack to prove that lower levels aren't simply a tower of lies, only instead of viruses it's flush. That said, in practice very little hardware is actually a giant tower of lies. Flash drives typically do not have enough of a controller to actually cache anything, thus no real capability to lie about writes/etc. SSDs do, but they also generally obey the expected behavior around flushing actually flushing and not just lying about it. RAID controllers and similar are the danger zone, because they may have cache and may lie to the CPU about it on the assumption that it actually will eventually be flushed, and that's the dangerous thing.
  - amelius 4y ago
    The question remains: how did I lose an entire ext4 volume after pulling the USB drive from the machine without unmounting it. Is anyone testing for these kinds of things?
  - floatboth 4y ago
    More importantly ZFS by design ensures that if you unplug early and lose the last few writes, you'll end up in an earlier consistent state.
    - amelius 4y ago
      I think the one you are replying to understands this, but the point is that "the last few writes" may be executed in different order by the underlying hardware and this might confuse the filesystem.
floatboth 4y ago
That's great if you only use these drives on your big computers. Sadly if you do want a "universally" portable drive – usable on an Android phone and an iPad and whatnot – you're stuck with horrible basic filesystems like exfat :(

by matja 4y ago

amelius 4y ago

Isn't any filesystem that runs on top of a SSD or flash drive at risk from the SSD/flash drive controller corrupting the disk or making it inaccessible, when the volume is not cleanly unmounted? I can remember losing a USB drive that way, even though it was using a journaling filesystem.

paulmd 4y ago
Potentially yes, realistically no. This is the old "if the storage controller is lying about flushing to disk, there's not a lot you can do about it" problem. ZFS does have defensive mechanisms, like doing a read after write to try and be sure that what is written is actually committed to disk, but if the storage controller chooses to serve that out of cache then that could be a lie too. It's the old "trusting trust" predicament, there's no way for hardware higher in the stack to prove that lower levels aren't simply a tower of lies, only instead of viruses it's flush. That said, in practice very little hardware is actually a giant tower of lies. Flash drives typically do not have enough of a controller to actually cache anything, thus no real capability to lie about writes/etc. SSDs do, but they also generally obey the expected behavior around flushing actually flushing and not just lying about it. RAID controllers and similar are the danger zone, because they may have cache and may lie to the CPU about it on the assumption that it actually will eventually be flushed, and that's the dangerous thing.
- amelius 4y ago
  The question remains: how did I lose an entire ext4 volume after pulling the USB drive from the machine without unmounting it. Is anyone testing for these kinds of things?
- floatboth 4y ago
  More importantly ZFS by design ensures that if you unplug early and lose the last few writes, you'll end up in an earlier consistent state.
  - amelius 4y ago
    I think the one you are replying to understands this, but the point is that "the last few writes" may be executed in different order by the underlying hardware and this might confuse the filesystem.

floatboth 4y ago

That's great if you only use these drives on your big computers. Sadly if you do want a "universally" portable drive – usable on an Android phone and an iPad and whatnot – you're stuck with horrible basic filesystems like exfat :(